I feels like it's a bit hard to take much from this without running this trial many times for each model. Then it would be possible to see if there are consistent themes among each model's solutions. Otherwise, it feels like the specific style of each result could be somewhat random. I didn't see any mention of running multiple trials for each model.
What I take from this is that LLMs are somewhat miraculous in generation but terrible at revision. Especially with images, they are very resistant to adjusting initial approaches.
I wonder if there is a consistent way to force structural revisions. I have found Nano Banana particularly terrible at revisions, even something like "change the image dimensions to..." it will confidently claim success but do nothing.
It would be interesting to see if they would get better results if they didn't grade their own work. Feed the output to a different model and let that suggest improvements, almost like a GAN.
This wasn’t just “add more details”—it was “make this mechanically coherent.”
The overall text doesn’t appear to be AI written, making this all the more confusing. Is AI making people write this way now on their own? Or is it actually written by an LLM and just doesn’t look like it?
I tried an experiment like this a while back (for the GPT-5 launch) and was surprised at how ineffective it was.
This is a better version of what I tried but suffers from the same problem - the models seem to stick close to their original shapes and add new details rather than creating an image from scratch that's a significantly better variant of what they tried originally.
Iterating a Markov chain does not make it any more or less "agentic". This is yet another instance of corporate marketing departments redefining words b/c they are confused about what exactly they're trying to build & sell.
I don't have all the most recent models but I've found image generation to be terribly disappointing in most that I've tried in that it doesn't seem capable of understanding directions or fixing mistakes.
"Create a drawing of a cliff in the desert."
Get something passing.
"Add a waterfall."
Get a waterfall that has no connection or outlet.
"Make the waterfall connect to a creek that runs off to the left."
Get a waterway that goes by the waterfall without connecting and goes straight through the center of the image.
Give up on that and notice that the shadows go to the left but the sun is straight behind.
"Move the sun to the right so that it matches the shadows more accurately."
Sun stays in the same spot, but grows while exaggerated and inaccurate shadows show up that seem to imply the backside of the cliff doesn't block light.
I love this experiment and am surprised that the Claude models performed that much better than the competition. Opus was particularly impressive both in the quality itself and the ability to iterate meaningfully.
Now... Was this article LLM written?
This part triggered all my LLM flags:
```
Adding a bicycle chain isn’t just decoration—it shows understanding of mechanical relationships. The wheel spokes, the adjusted proportions—these are signs of vision-driven refinement working as intended.
```
What prevents LLM designers from cheating and including a human handcrafted SVG into the model for this specific request (allowing for variations between calls)?
What's troubling to me is that it doesn't seem to have much account for "drift" -- it sort-of just goes down a single path and tries to improve as it goes.
What about structuring the agentic loop to do a simple genetic algorithm -- generate N children (probably 2 or 3), choose the best of the N+1 options (original vs. child A vs. child B vs. child C, and so-on) and then iterate again?
18 comments
[ 2.6 ms ] story [ 44.0 ms ] threadThat's what working with GPT-5-Codex on actual code also feels like.
I wonder if there is a consistent way to force structural revisions. I have found Nano Banana particularly terrible at revisions, even something like "change the image dimensions to..." it will confidently claim success but do nothing.
This is a better version of what I tried but suffers from the same problem - the models seem to stick close to their original shapes and add new details rather than creating an image from scratch that's a significantly better variant of what they tried originally.
If first output is crappy, the next 3 iterations will improve the same crap.
This was not a good test.
Poor feet.
it uses AGENTS
ITS AGENTIC
AGENTIC.
"Create a drawing of a cliff in the desert."
Get something passing.
"Add a waterfall."
Get a waterfall that has no connection or outlet.
"Make the waterfall connect to a creek that runs off to the left."
Get a waterway that goes by the waterfall without connecting and goes straight through the center of the image.
Give up on that and notice that the shadows go to the left but the sun is straight behind.
"Move the sun to the right so that it matches the shadows more accurately."
Sun stays in the same spot, but grows while exaggerated and inaccurate shadows show up that seem to imply the backside of the cliff doesn't block light.
...
Now... Was this article LLM written?
This part triggered all my LLM flags: ``` Adding a bicycle chain isn’t just decoration—it shows understanding of mechanical relationships. The wheel spokes, the adjusted proportions—these are signs of vision-driven refinement working as intended. ```
What about structuring the agentic loop to do a simple genetic algorithm -- generate N children (probably 2 or 3), choose the best of the N+1 options (original vs. child A vs. child B vs. child C, and so-on) and then iterate again?