HN Explorer

Show HN: New AI edits images based on text instructions (github.com)

1098 points by bryced ↗ HN
This works suprisingly well. Just give it instructions like "make it winter" or "remove the cars" and the photo is altered.

Here are some examples of transformations it can make: Golden gate bridge: https://raw.githubusercontent.com/brycedrennan/imaginAIry/ma... Girl with a pearl earring: https://raw.githubusercontent.com/brycedrennan/imaginAIry/ma...

I integrated this new InstructPix2Pix model into imaginAIry (python library) so it's easy to use for python developers.

240 comments

[ 4.2 ms ] story [ 256 ms ] thread
[deleted]
(comment deleted)
nullish_signal
>11GB VRAM

Aaarrrgghh let me know when it's down to 4GB like Stable Diffusion

The prompt-based masking sounds incredible, with either pixel +/- or Prompt Relevance +/-

VERY impressive img2img capabilities!

bryced
It is stable diffusion but yes my fork does not have the memory optimizations needed to run it on only 4gb
[deleted]
(comment deleted)
NavinF
You can get a used 2080Ti for under $300 on eBay
smcleod
That's a lot of money for most people. It also means they have to have a PC to put it in.
nullish_signal
Thank you for your sensical response... Very happy with my 6GB VRAM card and don't have $300 lying around to use on a git repo that will probably be slimmed down in a month or two
[deleted]
(comment deleted)
ilaksh
Does anyone know if there is something like Google Cloud for GPUs but with an easy way to suspend the VM or container when not in use? Maybe I am just looking for container hosting with GPUs.

I am just trying to avoid some of the basic VM admin stuff like creating, starting, stopping for SaaS if someone already has a way to do. Maybe this is something like what Elastic Beanstalk does.

midlightdenight
Maybe not quite what you’re looking for, but I’ve seen some people mention banana.dev

https://www.banana.dev/

Never used it myself, but looks like AWS Lambda/GCP Cloud Functions tailored to ML models.

TekMol
"Log in with Github". No thanks.
punkspider
I think there's https://brev.dev and https://banana.dev
naderkhalil
Hey, founder of Brev.dev here. Brev lets you suspend the instances when not in use, and also auto-stops it after 3 hours of inactivity to avoid expensive surprises. Would love for you to give it a shot
nicd
vast.ai, paperspace.com
PaulMest
I've played with several of these Stable Diffusion frameworks and followed many tutorials and imaginAIry fit my workflow the best. I actually wrote Bryce a thank you email in December after I made an advent calendar for my wife. Super excited to see continued development here to make this approachable to people who are familiar with Python, but don't want to deal with a lot of the overhead of building and configuring SD pipelines.
TekMol
How can I try this?

Can this be run on a Digitalocean VM?

I looked around on DO's products, but none seems to advertise that it has a GPU. So maybe it is not possible?

malux85
Try paperspace, they have GPUs and you can set billing limits to stop accidental overusage

(no affiliation other than being a happy customer)

kumarm
Doesn't work if any people are in the photos: https://twitter.com/kumardexati/status/1616972740728356867/p...
bryced
Works fine for me, you just need to adjust the strength of the edit.
kumarm
You mean steps?
bryced
no. in imaginairy it's called `--prompt-strength`. In other libraries it's called CFG or "classifier-free guidance". For the image edits I vary the strength of the effect from between 3-25
bryced
For the specific example you provide you could also use a prompt-based mask to prevent it from editing the person.
sam1r
Did you have to tweet it wasn’t working versus just not making it a public “omg it’s not working it’s no good”
lgas
It does work on some things with people. I colorized a black and white photo of myself and then turned the colorized version into me as a Dwarven king.
0x4164
I hope there is a James Fridman version of this kind of AI.
yieldcrv
“Add a dog in my arms”

I’ll keep you posted how well this works for dating apps

Uehreka
It's a little premature, fine, but I want to start liquidating my rhetorical swaps here: I've been saying since last summer (sometimes on HN, sometimes elsewhere) that "prompt engineering" is BS and that in a world where AI gets better and better, expecting to develop lasting competency in an area of AI-adjacent performance (a.k.a. telling an AI what to do in exactly the right way to get the right result) is akin to expecting to develop a long-lasting business around hand-cranking people's cars for them when they fail to start.

Like, come on. We're now seeing AIs take on tasks many people thought would never be doable by machine. And granted, many people (myself included to some extent) have adjusted their priors properly. And yet so many people act like AI is going to stall in its current lane and leave room for human work as opposed to developing orders of magnitudes better intelligence and obliterating all of its current flaws.

chadcmulligan
Haven't we been here before? - see self driving cars.
c7b
LLMs and Image AIs are the opposite of self-driving cars. "Everybody" had concrete expectations for at least half a decade now that the moment where self-driving cars would surpass human ability was imminent, yet the tech hasn't lived up to it (yet). While practically nobody was expecting AI to be able to do the jobs of artists, programmers or poets anywhere near human level anytime soon, yet here we are.
Der_Einzige
Still bad at poetry due to the tokenizer though. I wrote a whole paper on how to fix it: https://paperswithcode.com/paper/most-language-models-can-be...
c7b
Great work, congratulations! One question, if I understood it right you based your demo on GPT-2 - what is your experience working with those open-source language AIs. In terms of computational requirements and performance?

I'm really fascinated by all the tools the OS community is building based on StableDiffusion (like OP's), which compares favourably with the latest closed-source models like Dall-E or Midjourney, and can run reasonably well on a high-end home computer or a very reasonably-sized cloud instance. For language models, it seems the requirements are substantially higher, and it's hard to match the latest GPT versions in terms of quality.

yyyk
If LLMs (etc.) had the same requirements and business models as AV cars they'd still be considered a failure. Nobody expects Stable Diffusion to have a 6-sigma accuracy rate*, nor do we expect ChatGPT to seamlessly integrate into a human community. The AV business model discourages individual or small scale participation, so we wouldn't even have SD (would anyone allow a single OSS developer to drive or develop an AV car? Ok, there's Comma, that's all there is on the OSS side).

* The amount of times that I've seem an 'impressive' selections of AI images that I consider a critical failure deserves it's own word. The AIs are impressive for even getting that far, it's just that some people have bad taste and pick the bad outputs.

kristiandupont
I've certainly seen this argument before..

Yes, it's true that not all technology evolves as fast as predicted (by some, at some point), but first of all I still believe we will see self driving cars in the future and secondly, it's one anti-example in a forest of examples of tech that evolves beyond anyone's expectations. I don't find it very convincing.

trompetenaccoun
Autonomous driving as it currently exists came unexpected for most people. Now many look at it with the power of hindsight but back in the day the majority never thought we'd have cars (partly) driving on their own within a few years. The case of AI art seems the exact same to me, now that many are working on it there's lots of progress but it's still nowhere near what an experienced human can do. And that seems to be the general rule, not an exception.

We might need to create real intelligence for that to become true. A machine that can think and is aware of its purpose.

[deleted]
(comment deleted)
xdfgh1112
I think that "prompt engineering" stuff went away when ChatGPT came out.
c7b
Has it? I mean, maybe the idea of people doing this as a long-time career has, but practically, I still find it a challenge to get those AIs to do exactly what I want. I've played around with Dreambooth-style extensions now, and that goes some way for some applications, and I'm excited to try OP's solution, but in my experience, it is still a bit of a limitation for working with those AIs right now.
rjh29
Oh yeah it's definitely still an issue right now! But I think the power of ChatGPT's ability to understand and execute instructions has convinced most people that "prompt engineering" isn't going to be a career path in the future.
c7b
Absolutely. I briefly thought about asking ChatGPT to write a prompt, but then I remembered that the training corpus is probably older than those tools (I heard that if you ask it the right way, it will tell you that its corpus ended in 21 - whether it's true or not, it sounds plausible). But that's a truly temporary issue, the respective subreddits probably have enough information to train an AI for prompt engineering already (if you start from a strong foundation like the latest GPT versions).

Plus, who knows whether future models won't be able to integrate those different modes much better (along those lines https://www.deepmind.com/publications/a-generalist-agent).

rjh29
In the near future you can totally imagine a dialogue like that you'd have with a real designer, "can you make it pop a bit more?" or "can you move that logo to the right side?". It might some trial and error but it's only going to improve.

Making the AI truly creative (which means going beyond what the client asks for, towards things the client doesn't even know they want) would be a much larger leap and potentially take a lot longer.

TeMPOraL
I don't get it. Pre-ChatGPT prompt engineering was a BS exercise in guessing how a given model's front-end tokekizes and processes the prompt. ChatGPT made it only more BS. But I've seen a paper the other day, implementing more structured, formal prompt language, with working logic operators implemented one layer below - instead of adding more cleverly structured English, they were stepping the language model with variations of the prompt (as determined by the operators), and did math on probability distributions of next tokens the model returned. That, to me, sounds like valid, non-BS approach, and strictly better than doubling down on natural language.
c7b
Think about the problem in an end-to-end fashion: the user has an idea of what sort of image they want, they just need an interface to tell the machine. A combination of natural language plus optional image/video input is probably the most intuitive interface we can provide (at least until we've made far more progress on reading brain signals more directly).

How exactly we get there, by adding layers like on top like language models, or adding layers below like what you described, doesn't seem like such a fundamental difference. It's engineering, you try different approaches, vary your parameters and see what works best. And from the onset, natural language does seem like a good candidate for encoding nuances like "make it pink, but not cheesy" or "has the vibes of a 50's Soviet propaganda poster, but with friendlier colors".

bryced
Been doing a lot with prompts lately. What people are calling "prompt engineering" I'd call "knowing what to even ask for and also asking for it in a clear manner". That was a valuable skill before computers and will continue to be one as AI progresses.

I've been pretty disappointed to introduce ChatGPT to people in jobs where it would be a game changer and they just don't know what to do with it. They ask it for not-useful things or useful things in a non-productive way. "here is some ad copy I wrote, write it better". Whether you're instructing a human, chatgpt, or AI god... that's just too vague of instructions.

Semaphor
> I'd call "knowing what to even ask for and also asking for it in a clear manner".

It was a very important skill for searching. Nowadays, with Google "I know what you want better than you" search, it’s not that useful anymore (not useless, I get better search results by not using google and knowing what I want, just less required).

lobocinza
Most people struggle with deliberate logical thought.
gpderetta
If Asimov had robopsychologists in his stories, why can't we in real life? Who wants to be the first Susan Calvin?
felipeerias
IMHO it stems from lack of imagination. Impressive as the results may sometimes be, the user interfaces for AI are still extremely crude.

Soon we will see AI being used to define semantic operations on images that are hard to define exactly (imagine a knob to make an image more or less "cyberpunk", for example).

I also expect AI-powered inpainting to become a ubiquitous piece of functionality in drawing and editing tools (there are already Photoshop plugins).

Furthermore, my hunch is that many of the use cases around image creation will gradually move towards direct manipulation. Somewhat like painting, but without a physical model. AI components will be probably applied to interpreting the user's touch input in a similar way to how they are currently deployed to understand text input.

js8
Prompt engineering already exists, it's called management.
fassssst
Large language models are stateless. The apps and fine-tuned models are doing prompt engineering on users' behalf. It's very much a thing for developers, with the goal of making it invisible for end users.
[deleted]
(comment deleted)
[deleted]
(comment deleted)
[deleted]
(comment deleted)
Der_Einzige
Hoping that this is quickly implemented into the automatic1111 webUI.
kewp
anyone know how to use this? kind of confusing install instructions in the readme
bryced
If you're used to installing python packages it should be relatively easy. There are other projects with nice UIs but that's not what this library is for.
nicbou
Can it make it pop? Because that was the #1 request I remember dealing with.
TekMol
#1 request of what, for what, requested by whom?
stefanvdw1
That is a common request when working with clients. They have a hard time describing what they want so end up asking to “make it pop”
tamrix
Maybe but it could put their business logo anywhere!
prox
I don’t know why people use this “AI” thing, I have been using make my logo bigger cream (tm) for ages with success.

https://www.youtube.com/watch?v=GOwi3x92teo

;)

awestroke
Try these prompts:

"Add lens flare"

"Increase saturation"

"Add sparkles and gleam"

perfrom1
this should do it:

>> aimg edit input.jpg "make it pop" --prompt-strength 25

c7b
Many thanks to the OP, can't wait to try this out! I have a question I'm hoping to slide in here: I remember there were also solutions for doing things like "take this character and now make it do various things". Does anyone remember what the general term for that was, and some solutions (pretty sure I've seen this on here, apparently forgot to bookmark).

PS: I'm not trying to make a comic book, I'm trying to help a friend solve a far more basic business problem (trying to get clients to pay their bills on time).

bryced
dreambooth perhaps?
c7b
Dreambooth is what I'm using now, but I think I remember the concept had a specific name, something like 'context transfer' or so (pretty sure that was not the term) and tools that were pretty good at it that came out before Dreambooth. If I could at least remember the term it might be easier to search for them.

Dreambooth is ok at it, but it requires multiple images (you often read 30, but I've actually had decent results with as few as five) to recognize what it's supposed to replicate. I remember there were tools that were more adapted to the workflow "create a humanoid cartoon character with a bunny face", pick one image that you like, and then "now show that same character in scene X, eg teaching in a classroom" or "wearing a cowboy outfit".

cma
"textual inversion"
c7b
I think that's it, thank you!
airbreather
I'm getting mixed results, and for a given topic it seems to invariably give a better result first time you ask, then not so good if you ask again.

It could be random and my imagination, but seems that way.

natch
The headline and the heavy promotional verbiage on the site seems to be claiming this is some new functionality we didn’t have before. Image2image with text instructions isn’t new as the headline implies.

InvokeAI (and a few other projects as well) already does all this stuff much better unless I’m missing something. There are plenty of stable diffusion wrappers. Why not help improve them instead of copying them?

I’m not against having enthusiasm for one’s project, but tell us why this is different and please don’t pretend the other projects don’t have this stuff.

bryced
I'm not aware of any pre-existing open-source model that selectively edits images (leaving some parts untouched) based on instructions. This new method is much better than the image2image that shipped with the original stable diffusion. I'm looking at the InvokeAI docs right now and don't see anything like this feature. We previously had smart-masks, but InstructPix2Pix mostly does away with the need for those as well.

If I am mistaken please provide links to these prior features.

sebastiennight
It's very interesting, thanks! I've noticed (on the Spock example) that "make him smile" didn't produce a very... "comely" result (he basically becomes a vampire).

I was thinking of deploying something like that in one of our app features, but I'm scared of making our Users look like vampires :-)

Is it your experience that the model struggles more with faces than with other changes?

bryced
Yes if you're not careful it can ruin the face. You can play with the strength factor to see if something can be worked out. Bigger faces are safer.