So strange to write a whole post with Claude giving the best results and Grok consistently the worst, but awarding Grok the winner because at least it did the worst fastest?
I tried to one-shot the first test (the Rubik's Cube test) with LucidQuery's Swift model, to test it, as there are not much benchmarks about it and that they brag a lot about it, and I was pleasantly surprised to see it achieving a result similar to Grok 4.5 but in one shot (there is the same issue that if you scramble twice the solve button does not work anymore, but it got it in one shot).
Though it crunched most of the free quota, 47111 tokens, so I couldn't make multiple attempts.
Love the idea, I think more complex games would show the gap in ability better.
Do it again but this time get them to make a multiplayer online Jetmen REVIVAL game. Online play is key, because it's very complex. Jetmen is a good game for this since it has physics and customization that's complex enough but still simple.
Too nice to Grok, if there are really cost savings it should say how much each of the three demos cost so we can judge if it's worth the lower quality (probably not). The time to complete each would also be interesting.
> This feels like a kid trying to do science. The will is there, but lacks experience.
It's funny, when I saw the title I was hoping the article would include some sort of blind ranking, where you could see the outputs (without knowing which model they came from) and score them on some criteria. Could have been a fun way to get a better ranking of the results.
In this case, the requirement is clear enough, and the result is similar enough to judge it subjectively.
Even code-wise, if tests pass and the features are there in all cases, the rest of what matters (architecture, code quality, style, readability) is all subjective too.
You broke the HN guidelines badly with this, first because of the name-calling and personal attack, and second because it is a shallow dismissal in exactly the sense we ask people not to post here:
On the plus side, your comment does contain some specific observations which could have made for a good comment—one that communicates interesting information respectfully. If you'd please post that way in the future, we'd appreciate it.
I have not used grok 4.5 yet, but the other pictures match my experience doing anything graphical with the other models that it cracks me up. gpt 5.5 has no design sense whatsoever. It cannot even make terminal output not look terrible. I've asked it to use colors and formatting in various ways and got goofy randomly colored output. opus 4.7 and later seemed to have an inuitive design sense by comparison - 2d or 3d. Fabel 5 is just rock solid.
Yes, subjective. But it matches my repeated experiences with these models for what it is worth.
I get better results with Opus than Fable 5 on various oneshots (including our old friend of "generate an SVG of a pelican riding a bicycle"). (Opus is also far and away better than pretty much any other SOTA or near-SOTA model.)
I get the point of this demo but if instructions are clear, tech stack related resources are available, then the models do not differ as much.
I use different models all the time. And mostly lower cost ones. I do not know how people write software these days, but I have clean instructions, usually in Epics and they have Tasks.
I have been using DeepSeek V4 Flash for much of my coding in https://github.com/brainless/akar for example. Planning is mostly done by Qwen latest (in opencode) or Sonnet.
For my commercial, client work I use Claude but barely use Opus. Sonnet does most of the work. For a recent project, I went through a 35 page PRD in about 4 weeks, that includes client calls, changes, Ecpi/Task generation, a massive test suite, deployment.
I’m spending a significant portion of my day waiting for agents to execute.
What’s more interesting to me than time-to-first token or latency is the time it takes for the agent to execute, from starts to finish, excluding when it’s waiting on a human.
Half year ago I tried to use Codex, Claude and Gemini build the same scripts to automate various things on my machine. Claude was the clear winner back then, making the most reasonable assumptions, presenting results in the easiest-to-read format, writing runnable script with minimum dependency. Half year later I think Codex and Claude models have both advanced a lot, but Gemini is still lackluster. Gemini could catch problems when reviewing Claude/Codex's design plans and code, but it's hard to make Gemini make complex plans or implement complex code by itself.
43 comments
[ 3.2 ms ] story [ 42.2 ms ] threadFor hard tasks , that needs precision I will wait and pay expensive tokens
For everything else , query data , logs, rolling out releases , I’m using grok and it’s much better vs other tools and much cheaper too .
> “snappy stylist”
Funny you can tell its slop just by this
Variance in quality on these things is so, so high.
Though it crunched most of the free quota, 47111 tokens, so I couldn't make multiple attempts.
Do it again but this time get them to make a multiplayer online Jetmen REVIVAL game. Online play is key, because it's very complex. Jetmen is a good game for this since it has physics and customization that's complex enough but still simple.
It's funny, when I saw the title I was hoping the article would include some sort of blind ranking, where you could see the outputs (without knowing which model they came from) and score them on some criteria. Could have been a fun way to get a better ranking of the results.
Nothing wrong with that.
In this case, the requirement is clear enough, and the result is similar enough to judge it subjectively.
Even code-wise, if tests pass and the features are there in all cases, the rest of what matters (architecture, code quality, style, readability) is all subjective too.
lol. These are the comments that keep bringing me back here. Great feedback delivered with a dash (or more) of salt.
https://upload.wikimedia.org/wikipedia/en/c/cd/Breakout_game...
https://upload.wikimedia.org/wikipedia/commons/1/1a/Screensh...
That the tiles are much to thick makes Grok the most reasonable result.
"Please don't post shallow dismissals, especially of other people's work. A good critical comment teaches us something." - https://news.ycombinator.com/newsguidelines.html
On the plus side, your comment does contain some specific observations which could have made for a good comment—one that communicates interesting information respectfully. If you'd please post that way in the future, we'd appreciate it.
Yes, subjective. But it matches my repeated experiences with these models for what it is worth.
Written by Claude. Ugh. If it’s worth publishing, it’s worth proofreading, folks.
I don't get why cost per reply is at all relevant here?
Why do so few who attempt comparisons actually compare dollars per task.
I use different models all the time. And mostly lower cost ones. I do not know how people write software these days, but I have clean instructions, usually in Epics and they have Tasks.
I have been using DeepSeek V4 Flash for much of my coding in https://github.com/brainless/akar for example. Planning is mostly done by Qwen latest (in opencode) or Sonnet.
For my commercial, client work I use Claude but barely use Opus. Sonnet does most of the work. For a recent project, I went through a 35 page PRD in about 4 weeks, that includes client calls, changes, Ecpi/Task generation, a massive test suite, deployment.
I keep it pretty up to date (tomorrow Grok 4.5 and Sonnet 5 should be pushed).
What’s more interesting to me than time-to-first token or latency is the time it takes for the agent to execute, from starts to finish, excluding when it’s waiting on a human.