Show HN: Stable Diffusion powered level editor for a 2D game (generalrobots.substack.com)
Hey folks, I’ve been working on using control-net to take in a video game level (input as a depth image) and output a beautiful illustration of that level. Play with it here: dimensionhopper.com or read the blog post about what it took to get it to work. Been a super fun project.
91 comments
[ 2.9 ms ] story [ 154 ms ] threadWe have the technology to do this right now.
This guy just demonstrated what's required to generate a theme, and it's not far off from extrapolating further from that in using a LLM to generate Lore and just some random maze generator to create the base control image.
>> I would've expected at least a not grid-based zoning so that buildings on curves look more natural. All these empty pieces of land in between buildings look really bad and kind of force us to make grid cities. And that is not even an innovation, it was already present in the SimCity series. But some procedurally generated buildings for smooth corners and connecting buildings would be nice.
> Its hard to make assets that would work with every curve. When you see screenshots of nice cities like this, people are using mods to hand place assets with them clipping into each other to make a unified wall of buildings along the curve or corner.
https://news.ycombinator.com/item?id=36294742
ML generated building configurations for city builder games. Readily adaptable to any shape, and as a bonus can break up excessive repetition a bit. If you want to be ambitious, train a model on real-world aerial photos.
Great example of using AI as a tool to make something exceptional
As a game dev, I think at this stage AI can be a helpful utility, but it does not replace a designer's touch for professionally looking games.
Many of the quirks of our technology, like audio distortion for example, quickly become key components of certain styles. I remember as a child growing up in a funny valley after the acceptance of analogue distortion but before the widespread adaption of digital distortion.
Right now I'm thinking of someone like James Gerde, where the frame-to-frame shifts of AI imagination are part of the aesthetic. I think it's only a matter of time before this effect is matched up with something that makes emotional sense, and then it will blow up.
Personal use, a mod, or a free experiment is one thing, but a shipping game is a different can of worms.
On the other hand, I’ve never been an artist myself. So I’ve never been able to make my game ideas come true until now. The world is much more open to me in a creative side that my mechanical skills prevented.
Artists will continue to make art because it’s a compulsion. But I wish we had a world that was less oriented towards rewarding meaningless toil and would at least allow our born artists, writers, and creators the chance to do their obsessions to our benefit. Especially as we move post scarcity, I hope we can build a WPA like entity - perhaps, in a crazy twist, funded by AI?
The artists can now create on the tech side too, courtesy of inbound GPT-like LLMs. This isn't a one way street. The techies can craft art, the artists can craft tech.
It's opening up enormous pathways whether you're a programmer or artist. The artist has to be willing to expand and take on more responsibilities, just as the techie does if they want to craft quality AI art for a game.
With all the various game engines available now, we're not far away from being able to relatively easily have an LLM build nearly all the software side for you via prompting. From there you can bring whatever your strength is to customizing, implementing the game. Maybe you're good at ensuring high quality gameplay, maybe you're an artist that has an elite eye for how things should look, maybe you're a programmer and your game will be better optimized (and so on).
We already are with digital goods, society just hasn't caught up yet. I can make essentially infinite many copies of, say, Braid, and give every person in the world with an Internet connection a copy of it for a couple thousand dollars by using Cloudflare unlimited bandwidth R2 and bittorrent. A couple thousand dollars is basically a rounding error in the scheme of things. As I am not Jonathan Blow, distributing Braid would be a violation of copyright law, but copyright law is just a social contract that we entered into to incentivize the creation of work. If Jonathan Blow were compensated for every copy of Braid out there, I'm sure he would be quite happy to be (even more) rich.
So even in a post-digital-scarcity world, artists and programmers need to get paid, and so we have various DRM schemes, the first of which is the copyright system in the first place, but that works about as well as trying to make water not wet. Movies are leaked onto torrent sites like Rarbg (RIP) and people make copies all day long. libgen mirrors are still around despite the best efforts of the copyright regime. But let's be honest with ourselves, digital goods themselves are already post scarcity, we just haven't figured out how to incentivize the creation of works in our half-post-scarcity world and have no idea on how to move forwards.
Alternate solutions are out there, but we have no experience as a society in upending large social contracts (like copyright). You can easily imagine a system where what's popular gets tracked, and money flow to the creators of the media that people are actually watching and consuming. It would be a more draconian system than the DRM we have right now, but on the other hand, if it promotes the arts, then maybe it's worth it.
The artist will make art without reward. They always have. But few can do it for a living. Fewer will in this next phase.
To be clear, that is according to the definition you've made up. The commonly understood definition includes many or most of peoples' desires. Travel, housing, vacations, iphones, advanced health care, entertainment, higher education, etc.
> I think we are actually there, but have induced a scarcity economy with extreme imbalances and fake incentives to toil for toils sake.
Even in western countries, millions of people are struggling with basic costs of housing, energy, and even food. I mean people who actually work many hours every day and earn money are struggling with these things. We seem to be a long way off it even for your "basic necessities" definition.
https://en.m.wikipedia.org/wiki/Post-scarcity_economy
To your point, they do. However I’d assert that’s mostly due to income inequality and false scarcity to induce labor for labors sake.
And I'd assert it's not mostly due to income inequality.
If Karma is real, games made with uninteresting AI art will fail as we generate a sea of low-effort sludge.
I'm sure in many AI proponents' minds, this is the exact definition of "wasted": a human does something that machines can do.
(I'm still on the fence over this topic)
Why should we care about your mum, and not the guy who creates buggy whips?? Or the guy who runs the ferry that crosses the river who will be out of their job because of the new bridge?
Where do you draw the line at caring?
Everybody is an artist, and everybody can create and share art for themselves and the people around them. That is art at is best, in my opinion. Commercialization and mass production and display and distribution of "art" is what degrades it for me.
> Especially as we move post scarcity, I hope we can build a WPA like entity - perhaps, in a crazy twist, funded by AI?
Post-scarcity will never happen because even the people with mansions in many countries, billion-dollar yachts, and fleets of private airplanes do not have enough. And 10 billion people can't even live like them. Why would the people who own most of the capital suddenly collectively decide they would like to share the production with the rest of the world?
But if it did happen, we should not pay artists for art. We should give everybody enough so that if anybody wanted to paint a picture or hammer some metal into a horseshoe or knit a sweater or write some assembly code, they could do so.
(see also https://github.com/midzer/jumpnbump)
Now need someone to do other games of that time, place and genre: Tremor 3, C-Dogs, etc.
HN's submission title ("Show HN: Stable Diffusion powered level editor for a 2D game") made me think of the former. Article title ("2D Platformer using Stable Diffusion for live level art creation") was more accurate to me.
Edit: In case my sarcasm isn't clear, I hate this mentality and I am just bitterly griping about AI into the void. You should definitely ask permission to use data before training AI on it, but that will put you behind other AI people who aren't asking permission
[1] https://youtu.be/wAbLsRymXe4
[2] https://github.com/sebastianstarke/AI4Animation
A few people have tried training on sprite sheets and emitting them directly, and it did not work.
A few people have been working specifically on walking cycles, and it has a lot of limitations.
In my specific experience with other bespoke pixel art models, if you ask for a "knight," you're going to get a lot of the same looking knight. Fine-tuning will unlearn other concepts that are not represented in your dataset. LORAs have not been observed to work well for pixel art. You can try the Astropixel model, the highest quality in my opinion, for prototyping.
Part of this is you're really observing how powerful ControlNet, T2I-Adapters and LORAs are and you may have the expectation that something else you, a layperson, can do will be similarly powerful. Your thing is really cool. But is there some easy trick without doing all this science, for animation? No. Those are really big scientific breakthroughs, and with all the attention on video - maybe 100-1,000 academic and industry teams working on it - there still hasn't been something super robust for animation that uses LDMs. The most coherent video is happening in with NeRF, and a layperson isn't going to make that coherent with pixel art. Your best bet is to wait. That said, I'm sure people are going to link here to some great hand-processed LDM videos, and maybe there's a pipeline with hand artwork a layperson can do today that would work well.
Something like how this posing works: https://www.youtube.com/watch?v=CiG_v61cLxI
To get it down to small pixeled 'sprite' scale, the right thing may be to actually output 'realistic' character animation frames this way, and then 'de-res' them via img2img into pixel art. The whole pipeline could be automated so that your only inputs are a single set of varied walking/posing/jumping control net poses and the prompts describing the characters.