Good effort, somewhat marred by poor prompting. Passing in “the tower in the image is leaning to the right,” for example, is a big mistake. That context is already in the image, and passing that as a prompt will only make the model apt to lean the tower in the result.
I should have been more clear. Those are NOT the direct prompts. They are the starter prompts. In fact that's why the attempt numbers change, we adapt the exact prompts depending on the model.
Everyone is sleeping on Gemini 2.5 Flash Image / Nano Banana. As shown in the OP, it's substantially more powerful than most other models while at the same price-per-image, and due to its text encoder it can handle significantly larger and more nuanced prompts to get exactly what you want. I open-sourced a Python package for generating from it with examples (https://github.com/minimaxir/gemimg) and am currently working on a blog post with even more representative examples. Google also allows generations for free with aspect ratio control in AI Studio: https://aistudio.google.com/prompts/new_chat
That said, I am surprised Seedream 4.0 beat it in these tests.
No one is sleeping on nano-banana/Gemini Flash, it's highly over-tuned for editing vs novel generation and maxes out at a pretty low resolution.
Seedream 4.0 is somewhat slept on for being 4k at the same cost as nano-banana. It's not as great at perfect 1:1 edits, but it's aesthetics are much better and it's significantly more reliable in production for me.
Models with LLM backbones/omni-modal models are not rare anymore, even Qwen Image Edit is out there for open-weights.
> That said, I am surprised Seedream 4.0 beat it in these tests.
OP here. While Seedream did have the edge in adherence it also tends to introduce slight (but noticeable) color gradation changes. It's not a huge deal for me, but it might be for other people depending on their goals in which case NanoBanana would be the better choice.
Gemini likely has a more powerful text encoder, which is why it's better at parsing complex, nuanced prompts. Seedream, on the other hand, might have a more advanced diffusion U-Net architecture that's better at preserving textures and handling local edits. One model understands better, the other draws better
I do not use ai image generating much lately. It seemed like there was a burst of activity a year and half ago with self hosted models and using some localhost web guis. But now it seems like it is moving more and more to online hosted models.
Still, to my eye, ai generated images still feel a bit off when doing with real world photographs.
George's hair, for example, looks over the top, or brushed on.
The tree added to the sleeping person on the ground photo... the tree looks plastic or too homogenized.
If you take a base model and train it on a hundred Seinfeld frames, it would pick up the specific style - the color grading, grain, lighting - and it would add the hair way more naturally
Thank you for the pointer. I was struggling with Nanobanana for editing an image which it had created earlier, but Reve gave me the edit result exactly the way I wanted in the first pass.
My usecase: An image of a cartoon character, holding an object and looking at it. Wanted to edit so that the character no longer has the object in her hand and now looking towards the camera.
Result Nanobanana: At first pass it only removed the object that the character was holding, however there was no change in her eyeline, she was still looking down at her now empty hand. Second prompt explicitly asked to change the eyeline to look at camera. Unsuccessful. Third attempt asked the character to look towards ceiling. Success but unusable edit as I wanted the character to look at the camera.
Result Reve: At first attempt it gave me 4 options and all 4 are usable. It not only removed the object and changed the eyeline of the character to look at the camera, but it also made posture changes so that the empty hands were appropriately positioned, and now since the character is in a different situation (sans the object that was holding her attention) Reve posed the character in different ways which were very appropriate - which I didn't think of prompting for earlier (maybe because my focus was on immediate need - object removal and change in eyeline).
On a little more digging found this writeup which will make me to signup for their product.
Some might critique the prompts and say this or that would have done better, but they were the kind of prompt your dad would type in not knowing how to push the right buttons.
OP here. You're the second person to say this. I cut my teeth on SD 1.5 - so I'm rather intimately familiar (for better or worse) with the level of prompt craft necessary depending on the model.
I feel like the FAQ section isn't displayed prominently enough:
How are the prompts written?
In addition to giving models several attempts to generate an image, we also write several variations of the prompt to ensure that models don't get stuck on certain keywords or phrases depending on their training data. For example, while hippity hop is a relatively common name for the ball riding toy, it is also known as a space hopper. We try to use both terms in the prompts to ensure that models are not biased towards one or the other.
Prompts for Hunyuan were attempted in both Chinese and English with and without Image Optimization.
Additionally when you see a prompt like "Turn on the lights" - the idea is to actually go beyond direct prompting commands - we're actually probing the capabilities of a truly multimodal LLM. It's a prompt that would spectacularly fail in more traditional models (such as SDXL).
I've been using Nano Banana quite a lot, and I know that it absolutely struggles at exterior architecture and landscaping. Getting it to add or remove things like curbs, walkways, gutters, etc, or to ask to match colors is almost futile.
Actually 5. The gemini result is pretty correct. And for that test, IMO only gemini properly preserved the original aesthetic. All others don't have the dark/scary mood.
I wonder how much longer those annoying stock photo database will continue. They are great for press photography and such. But stock pics of people in offices for a website are nothing, I would buy a min 3 month subscription for anymore
It is fun being one of the elderly who set their standards back in distant 2022. All these demos look incredible compared to SD1, 2 & 3. We've entered a very different era where the models seem to actually understand both the prompt and the image instead of throwing paint at the wall in a statistically interesting manner.
I think this was fairly predictable, but as engineering improvements keep happening and the prompt adherence rate tightens up we're enjoying a wild era of unleashed creativity.
I'm pretty sure that "replace the homeless man with a park bench" image was a reference to some TV show making a gentrification joke, but I can't put my finger on it. Anyone recall?
Kontext is very good. Get yourself a 5060 ti 16GB and never have to pay for API calls again for this purpose, at least not when you have the time spare. If you need this sort of editing at the speed of gui-clicking + 10s, then you'll need to pay API tolls, or capex for > 5070/80.
This is not the point of this post, but is anyone else getting tired of this front end style that Claude creates? I see it on web apps everywhere and (just like with AI writing and images) I get that funny "is this slop?" feeling
Here's a post I wrote on the Replicate blog putting these image editing models head-to-head. Generally, I found Qwen Image Edit to be the cheapest and fastest model that was also quite capable of most image editing tasks.
If I were to make an image editing app, this would be the model I'd choose.
I still feel varying the prompt text, number of tries, and varying strictness combined with only showing the result most liked dilute most of the value in these test. It would be better if there was one prompt 8/10 human editors understood and implemented correctly and then every model got 5 generation attempts with that exact prompt on different seeds or something. If it were about "who can create the best image with a given model" then I'd see it more, but most of it seems aimed at preventing that sort of thing and it ends up in an awkward middle zone.
E.g. Gemini 2.5 Flash is given extreme leeway with how much it edits the image and changes the style in "Girl with Pearl Earring" only to have OpenAI gpt-image-1 do a (comparatively) much better job yet still be declared failed after 8 attempts, while having been given fewer attempts than Seedream 4 (passed) and less than half the attempts of OmniGen2 (which still looks way farther off in comparison).
Is there anything like this comparison for nsfw images? I'm married to a boudoir photographer who sometimes wants to use ai tools for things, and they are all _awfull_ if there is nudity on photos. It's like some sort of neo puritanism has taken over.
Neat comparison. The only qualm I have is giving a pass on that last giraffe... it's not visibly any shorter, just bent awkwardly.
Even so, Gemini would lose by 1, but I found that I would often choose it as the winner(especially say, The Wave surfer). Would love to see a x/10 instead of pass/fail.
Yeah, it’s kinda crazy how fast this stuff leveled up.
A year ago we were happy if hands looked normal — now we’re nitpicking shadows and curb textures. Wild times.
This is so much more useful than synthetic benchmarks. The most important column here isn't pass/fail, it's attempts. In production a model that gets it right in 2 attempts is 10x more valuable than one that needs 20 iterations of prompt engineering. It's a direct measure of cost and predictability.
Seedream 4 won on points, but Gemini seems more steerable and required less fighting on many of the tasks
39 comments
[ 2.7 ms ] story [ 65.3 ms ] threadThat said, I am surprised Seedream 4.0 beat it in these tests.
Seedream 4.0 is somewhat slept on for being 4k at the same cost as nano-banana. It's not as great at perfect 1:1 edits, but it's aesthetics are much better and it's significantly more reliable in production for me.
Models with LLM backbones/omni-modal models are not rare anymore, even Qwen Image Edit is out there for open-weights.
OP here. While Seedream did have the edge in adherence it also tends to introduce slight (but noticeable) color gradation changes. It's not a huge deal for me, but it might be for other people depending on their goals in which case NanoBanana would be the better choice.
these aren't cases where I'm trying to do something that skirts the edge of copyright, either (like "Ghiblifying" images, for example).
that said, when it does work, it is super impressive.
Still, to my eye, ai generated images still feel a bit off when doing with real world photographs.
George's hair, for example, looks over the top, or brushed on.
The tree added to the sleeping person on the ground photo... the tree looks plastic or too homogenized.
If you take a base model and train it on a hundred Seinfeld frames, it would pick up the specific style - the color grading, grain, lighting - and it would add the hair way more naturally
My usecase: An image of a cartoon character, holding an object and looking at it. Wanted to edit so that the character no longer has the object in her hand and now looking towards the camera.
Result Nanobanana: At first pass it only removed the object that the character was holding, however there was no change in her eyeline, she was still looking down at her now empty hand. Second prompt explicitly asked to change the eyeline to look at camera. Unsuccessful. Third attempt asked the character to look towards ceiling. Success but unusable edit as I wanted the character to look at the camera.
Result Reve: At first attempt it gave me 4 options and all 4 are usable. It not only removed the object and changed the eyeline of the character to look at the camera, but it also made posture changes so that the empty hands were appropriately positioned, and now since the character is in a different situation (sans the object that was holding her attention) Reve posed the character in different ways which were very appropriate - which I didn't think of prompting for earlier (maybe because my focus was on immediate need - object removal and change in eyeline).
On a little more digging found this writeup which will make me to signup for their product.
https://blog.reve.com/posts/reve-editing-model/
Some might critique the prompts and say this or that would have done better, but they were the kind of prompt your dad would type in not knowing how to push the right buttons.
I feel like the FAQ section isn't displayed prominently enough:
How are the prompts written?
Additionally when you see a prompt like "Turn on the lights" - the idea is to actually go beyond direct prompting commands - we're actually probing the capabilities of a truly multimodal LLM. It's a prompt that would spectacularly fail in more traditional models (such as SDXL).I've been using Nano Banana quite a lot, and I know that it absolutely struggles at exterior architecture and landscaping. Getting it to add or remove things like curbs, walkways, gutters, etc, or to ask to match colors is almost futile.
Still useful comments, as the models mostly overlap
I think this was fairly predictable, but as engineering improvements keep happening and the prompt adherence rate tightens up we're enjoying a wild era of unleashed creativity.
If I were to make an image editing app, this would be the model I'd choose.
https://replicate.com/blog/compare-image-editing-models
E.g. Gemini 2.5 Flash is given extreme leeway with how much it edits the image and changes the style in "Girl with Pearl Earring" only to have OpenAI gpt-image-1 do a (comparatively) much better job yet still be declared failed after 8 attempts, while having been given fewer attempts than Seedream 4 (passed) and less than half the attempts of OmniGen2 (which still looks way farther off in comparison).
Even so, Gemini would lose by 1, but I found that I would often choose it as the winner(especially say, The Wave surfer). Would love to see a x/10 instead of pass/fail.
Seedream 4 won on points, but Gemini seems more steerable and required less fighting on many of the tasks