I have absolutely no reason to ever use this but I'm absolutely going to throw money at the author because I'm a sucker for pixel art and this looks incredible. (I spent like five minutes staring at the zombie .gif, slackjawed.)
My girlfriend (who is responsible for most of the art in the examples) and I pretty much reacted the same way when we first got the algorithm working. It was one of those things that went from 'not working at all' to 'suddenly working perfectly when you fix one broken thing'. Good times.
Is my understanding of the technology correct that in the most basic sense what you are doing is creating normal maps, diffuse maps etc for a 3D plane?
This tool looks amazing and this will be my first time supporting a project via kickstater!
That is pretty much correct, except we are creating normal maps, depth maps, and ambient occlusion maps (and anisotropy maps but I haven't talked much about that yet). The diffuse map is something the artist will have to paint.
Cool, but wouldn't it be better/easier to have a tool where you actually draw a depth map by hand but the tool shows you illumination from various directions in real time? At least for the pixel art case. I'm not an artist but to me that seems more intuitive.
Whoa cool. This will work well. I implemented a similar dynamic lighting solution for sprites back in 1997. I used 3D models though that were rendered and then the pure colors and Z-buffer were captured for dynamic lighting effects. Mentioned here as "Software Prototype: Real-time 2D Sprite Shader":
Very cool. I think some MMOs are using that technique in conjunction with server-side rendering for displaying good quality distant objects without having to push as much geometry to the client. Also nice work on Clara, I'm particularly looking forward to V-Ray support.
Do you have any examples of actual games, even if you're not sure that's what they're doing? I don't know of any MMO that has the server side resources to render meshes and send them to clients. Of course, most MMOs aren't "pushing geometry" to clients either, with the exception of Second Life style virtual worlds.
I was thinking about the same thing myself. If the lamp can unpack the 3d data from 2d assets in a web browser you could seriously increase the art quality in webapps without hugely increasing download size.
I'm not really sure how. Most web pages don't have dynamic lighting. I suspect that the amount of information to encode the flat image plus the normal map is going to be more than the amount of information to encode the pre-rendered effect.
Now, if you were planning on doing some kind of dyanmic lighting in a web page, then sure, this would help. But I can't think of much use for dynamic lighting outside of games or other physical simulations, which is likely overkill for most webapps.
This is awesome and kind of makes me want to get back into game programming. Unfortunately, I have no artistic ability whatsoever, and don't really have the time if I did.
Yes, so then the hard part is actually getting a photo of a completely un-shaded object. Alternatively, you could draw the lighting profiles and then specify the true lighting direction for the given photograph -- this is potentially enough for a separate algorithm to the remove the lighting from the initial photo, but that doesn't exist yet in this software (and can get complicated -- for some recent related research see http://www.cs.berkeley.edu/~barron/BarronMalikCVPR2012.pdf).
My guess is that it works by blending the hard-light-case sprites trigonometrically, w.r.t. the vector between the position of the light and the origin/sprite.
Which part? For the main lighting technique, most of the magic comes from the artist providing the surface normal components. Then to produce the image under some specified lighting direction, you can do the normal 3D graphics thing: dot product the light vector with the normal vector and scale that by the diffuse color.
http://en.wikipedia.org/wiki/Lambertian_reflectance
In case the artist doesn't draw the X,Y, and Z surface normal components directly but instead chooses some other set of lighting profiles, you could use photometric stereo to recover the surface normals. (If this is the approach used, then applying such a technique to specially-crafted pixel art is indeed novel).
Yeah it's the second case I was interested in. It looks like the algorithm generates a best fit normal map based on the various lighting profiles - presumably it must be told which direction the light is coming from.
I do a lot of research work with stereo, so thanks for the link! Have to give that a go sometime :)
Super cool! Confederate Express[1] is using a similar workflow, with great results. It's a really cool idea, and I'm surprised that it hasn't been used in more games up to this point. Seeing projects like this -- and spriter[2] and spine[3], really does make me want to get back into game development.
So all the examples only use one light... how many lights can you have in a scene? What kinds of lights are supported (omni, spot, etc)? Can you use different colors of lights? If I wanted to use hundreds of lit particles in a scene, how performant would it be?
I'm no graphics guy (not even by a long shot), but my understanding of this is that it's generating the same maps that you'd otherwise have to generate through some other process. The images used to generate them have a single point source, but I have to assume that, once generated, the maps would work just as well for multiple or different types of light sources. As for performance, that's entirely up to whatever engine you're using to render the damn thing; once you have the maps, you're done with SpriteKit.
As far as lighting goes, I think it would be as performant as it gets. It's the same idea as deferred shading; you only have to shade the pixels that the light affects.
As for types of lights and colors, yes, I think that would all be possible.
It seems like the main use case is exporting the normal/depth/ambient occlusion maps, which you'd probably end up dropping into your engine of choice rather than using Sprite Lamp itself in your game (Sprite Lamp would generate the assets, but the game engine would display them).
If you have the maps, there's no hard limit to the number of lights you can have in your scene, or colors, etc; it just depends how many lights your engine can support with decent performance.
Think of Sprite Lamp more like Blender than like Unity — Sprite Lamp is a way to create assets.
This is a very cool technique. I have been thinking about how to generate normal maps for traditionally drawn 2D images ever since seeing the normal mapped canvas demo[1]. This seems to be an answer.
Drawing lighting profiles that are coherent however does not seem simple. One of the uses of this technique "Depth maps for stereoscopic 3D" appears much more complicated to me that drawing a depth map by hand in the first place. I drew a depth map for my drawing Starry Venice[2] as a step in making it into a wigglegif. Drawing multiple correct light profiles to generate the depth map for a scene such as Starry Venice seems almost impossible to me. This is far from the base use-case, but still.
It will be interesting to see how forgiving the creation of the normal map will be on imperfect light profile inputs. Also, it will be interesting to see if any artists who are masters of this technique will emerge.
As an artist myself, I actually think making these assets would be quite easy. I already use layers in Photoshop to first draw my "diffuse map" image, then draw the shading as layers on top of it. It would essentially be the same exact process I already go through to draw cartoons, just with the added steps of drawing a couple more shading layers.
On a second look, it may not be as hard as it first appeared. When shading, the artist will just have to be very conscious of the angle of incidence from the light source. We will see how difficult this is once alpha is released and we get a chance to compute some normal maps and debug their lighting profiles.
Shading isn't terribly hard and you could probably afford to be a little sloppy in this case. I would suspect that you'll end up spending about 2 times as long on each animation than you would have before.
BUT, with that 2x effort, you're getting a significant improvement in visual quality. The alternative would be the Donkey Kong Country option: model the character in 3D (easily 10x more effort than flat 2D animation, with a much more expensive work force and software), bake in the lighting, and generate gigantic animation sheets. Your asset library will explode in size. The games that have done this have tended to employ significant compression on the images, which can negatively impact visual quality.
Besides, the 3D rendering technique is apples compared with the oranges of pixel art. There's really nothing like artisanally crafted, locally sourced pixels made with love ;)
Yeah, as a technical guy I would tend to go with the full 3D route. It might be 10x the upfront work but having a fully automated pipeline might save you a lot of work down the line. For example just changing the color of a character could be as few as two clicks in the full 3D solution, but you might have to manually go through each sprite sheet with the other route.
And technically you could export the sprite sheet with however many frames you want (and be able to lower and increase the number easily) while still getting the exact same results as the Sprite Lamp solution. And of course artists could go in and manually make any changes they want.
It's interesting hearing the perspective of the artists. Thanks.
If you're hand-crafting sprites there's a good chance you're using a palletized paint program, which makes changing a character a matter of two or three clicks.
Also there are major stylistic advantages to drawing it by hand. Check out the baked-in motion blur on Sonic's feet in this sprite rip of Sonic 1: http://www.spriters-resource.com/genesis_32x_scd/sonicth1/sh... a while back there was a 2.5d Sonic game, and its motion had a lot less impact because no attempt was made to replicate the motion blur.
Plus of course if you're just drawing it you don't have to worry whether or not it actually makes sense - a lot of the more stylized cartoon characters are VERY hard to build spot-on 3d models of, because they're full of weird abstractions that only make sense in the 2d plane.
And finally, some people just don't like modeling stuff in 3d.
You could easily combine both worlds. You don't have to bake in the lighting. Just export the light maps, and then use them with a tool like Sprite Lamp to dynamically merge them in at runtime.
Thanks! And, that javascript canvas demo is relevant to my interest - thanks for the link!
You're right in your suspicion that depth map generation in Sprite Lamp is not a silver bullet for stuff like that. Images with big discontinuities in depth (especially open scenes, like the one you linked) will likely get you some pretty dubious results in Sprite Lamp. On the other hand, if you look closely at the self-shadowing on the brick gif from the website, I think you'll agree that the results are pretty accurate (note that the little notches and scrapes in the surface of some bricks get picked up accurately too) - while you could paint that map by hand, I suspect that getting the results that nice would take some time, and Sprite Lamp does it in a second or two (pre-optimisation). Stuff like character artwork (like the zombie or the plague doctor) fall somewhere in between - you get results that are good enough for self-shadowing, and with some tweaking you can generate a nice stereogram, but it's not necessarily 'physically accurate' (which in this case is another way of saying "I can't guarantee the results are what the artist pictured").
I'm reluctant to promise features that I haven't tried yet, but I'm planning on some experimentation with a combination of painting depth values and using Sprite Lamp - this will take the form of some tools for messing with the depth map from within Sprite Lamp after it's been generated, with an eye to intelligently detecting potential edges and letting you move whole bits of the scene around at once (and then an integrated means of actually looking at the depth map you've created - wigglegifs might be a good option there, actually).
For line art (like the zombie example) it may be possible to use one of several available semi-automated algorithms to generate a depth map. Here's a page that shows a few:
130 comments
[ 2.7 ms ] story [ 174 ms ] threadOh and thanks for the offer of thrown money!
This tool looks amazing and this will be my first time supporting a project via kickstater!
And, thanks!
http://www.exocortex.org/ben/#High_School-Era_and_Earlier_Pr...
Now, if you were planning on doing some kind of dyanmic lighting in a web page, then sure, this would help. But I can't think of much use for dynamic lighting outside of games or other physical simulations, which is likely overkill for most webapps.
My first thought was it is probably just a clever little convolution kernel trick but it seems harder after reading about it.
In case the artist doesn't draw the X,Y, and Z surface normal components directly but instead chooses some other set of lighting profiles, you could use photometric stereo to recover the surface normals. (If this is the approach used, then applying such a technique to specially-crafted pixel art is indeed novel).
Here's a factorization technique for photometric stereo that could be applied to the artist inputs: http://www.wisdom.weizmann.ac.il/mathusers/vision/courses/20...
I do a lot of research work with stereo, so thanks for the link! Have to give that a go sometime :)
[1]: http://www.kickstarter.com/projects/829559023/confederate-ex...
[2]: http://www.brashmonkey.com/spriter.htm
[3]: http://esotericsoftware.com/
Kudos to both, dynamic lighting on 2D art looks great.
This game totally needs to implement this. It has some similar ideas though. Take a look at this closely: https://s3.amazonaws.com/ksr/assets/000/945/686/c415709a876b...
notice how the wind blowing right to left affects the sprite: his cloak moves with the wind. Now that's attention to detail!
As for types of lights and colors, yes, I think that would all be possible.
If you have the maps, there's no hard limit to the number of lights you can have in your scene, or colors, etc; it just depends how many lights your engine can support with decent performance.
Think of Sprite Lamp more like Blender than like Unity — Sprite Lamp is a way to create assets.
This is a very cool technique. I have been thinking about how to generate normal maps for traditionally drawn 2D images ever since seeing the normal mapped canvas demo[1]. This seems to be an answer.
Drawing lighting profiles that are coherent however does not seem simple. One of the uses of this technique "Depth maps for stereoscopic 3D" appears much more complicated to me that drawing a depth map by hand in the first place. I drew a depth map for my drawing Starry Venice[2] as a step in making it into a wigglegif. Drawing multiple correct light profiles to generate the depth map for a scene such as Starry Venice seems almost impossible to me. This is far from the base use-case, but still.
It will be interesting to see how forgiving the creation of the normal map will be on imperfect light profile inputs. Also, it will be interesting to see if any artists who are masters of this technique will emerge.
[1] http://29a.ch/2010/3/24/normal-mapping-with-javascript-and-c... [2] http://fooladder.tumblr.com/post/61216111704/starry-venice
The artist should be doing this already!
I could see this getting prohibitive when creating animations for example.
BUT, with that 2x effort, you're getting a significant improvement in visual quality. The alternative would be the Donkey Kong Country option: model the character in 3D (easily 10x more effort than flat 2D animation, with a much more expensive work force and software), bake in the lighting, and generate gigantic animation sheets. Your asset library will explode in size. The games that have done this have tended to employ significant compression on the images, which can negatively impact visual quality.
And technically you could export the sprite sheet with however many frames you want (and be able to lower and increase the number easily) while still getting the exact same results as the Sprite Lamp solution. And of course artists could go in and manually make any changes they want.
It's interesting hearing the perspective of the artists. Thanks.
Also there are major stylistic advantages to drawing it by hand. Check out the baked-in motion blur on Sonic's feet in this sprite rip of Sonic 1: http://www.spriters-resource.com/genesis_32x_scd/sonicth1/sh... a while back there was a 2.5d Sonic game, and its motion had a lot less impact because no attempt was made to replicate the motion blur.
Plus of course if you're just drawing it you don't have to worry whether or not it actually makes sense - a lot of the more stylized cartoon characters are VERY hard to build spot-on 3d models of, because they're full of weird abstractions that only make sense in the 2d plane.
And finally, some people just don't like modeling stuff in 3d.
(I'm an artist and ex-animator.)
You're right in your suspicion that depth map generation in Sprite Lamp is not a silver bullet for stuff like that. Images with big discontinuities in depth (especially open scenes, like the one you linked) will likely get you some pretty dubious results in Sprite Lamp. On the other hand, if you look closely at the self-shadowing on the brick gif from the website, I think you'll agree that the results are pretty accurate (note that the little notches and scrapes in the surface of some bricks get picked up accurately too) - while you could paint that map by hand, I suspect that getting the results that nice would take some time, and Sprite Lamp does it in a second or two (pre-optimisation). Stuff like character artwork (like the zombie or the plague doctor) fall somewhere in between - you get results that are good enough for self-shadowing, and with some tweaking you can generate a nice stereogram, but it's not necessarily 'physically accurate' (which in this case is another way of saying "I can't guarantee the results are what the artist pictured").
I'm reluctant to promise features that I haven't tried yet, but I'm planning on some experimentation with a combination of painting depth values and using Sprite Lamp - this will take the form of some tools for messing with the depth map from within Sprite Lamp after it's been generated, with an eye to intelligently detecting potential edges and letting you move whole bits of the scene around at once (and then an integrated means of actually looking at the depth map you've created - wigglegifs might be a good option there, actually).
http://parter.kaist.ac.kr/jyhahn76/project13.html
http://forums.tigsource.com/index.php?topic=24094.msg958013#...
http://www-ui.is.s.u-tokyo.ac.jp/~takeo/teddy/teddy.htm