Tinfoil hat time: but the way they waited a couple weeks to evaluate Mimo 2.6 Flash, allowing GPT-6 Luna to sit on the Pareto frontier in the meantime, really did make me think they might have some bias going on... (Luna was displaced when they finally evaluated Mimo 2.6 Flash, which had been released the day before Luna)
I’ve very mixed feelings on Mimo. The new models do surprisingly well on some tasks, but they don’t allow you to set the thinking level and the model often times spends next to zero time thinking about a problem. If you coax it just a little bit, you can get it to solve it, but this isn’t an issue with other models. Also it’s incredibly easy to get it to get caught in a thinking loop.
I've personally experienced overthinking and the thinking loop issue more often than seeing it not think enough, I know what you mean though. I tried Haiku 5.5 and the most impressive thing about it to me was that it seemed not to overthink when it wasn't necessary, yet it somehow knew when it should spend some more time reasoning about a task.
American models are too rich for my blood though. I use GLM 5.3 Flash more than any other model and it has the same issues as Mimo. Deepseek 4.1 Flash is turning out to be a sweet spot for me; it's cheap and blazing fast on CoralBricks where I've been using it, so I don't mind when it overthinks (I try to pay attention to what my agents are doing, and it tends to finish tasks before I can skim half of its reasoning)
Neither is the primary cause, I'd argue. If you look at the coding index specifically, you're getting overall more plausible results (in particular, Terra no longer outperforming Fable). [1, 2]
I don't find it surprising that especially smaller Flash models that have been optimized primarily for coding fall behind on non-coding tasks and that their overall score suffers in comparison.
It does give rise to the question of how useful a single aggregate numeric index of very heterogeneous capabilities is, though.
Look also at the cyber index; while somewhat difficult to extrapolate because of refusals, even if for Opus 5.5 and Fable 5.1 you optimistically count all refusals as successes, the results are ... interesting.
It is hard to tell what people are trying to say when they use undefined words with no particular meaning. This doesn't seem to change the relative position of the Chinese models much. The Chinese are a month or few behind and the state of intelligence is so volatile that the top 10 will be completely different in 6 months. I'd expect it to still be a mix of US and Chinese models with the US models leading.
Benchmaxing is defined as targeted post-training attempts to optimise for higher benchmarks. This doesn’t change the aggregate intelligence or capability of the model, so it can make a model appear more capable than it is.
I’m pretty sure “benchmaxxing” as defined is a standard post-training procedure for ALL models.
One can argue though some models are post-trained heavily for specific benchmarks and less for other datasets. So they are weaker on out of distribution tasks.
- Broad class of problems or techniques that might apply to benchmark
Each has very different ramifications for how the model performs. The current evidence is that Anthropic and OpenAI is generally operating lower in the list.
Neither is the primary cause, I suspect. If you look at the coding index alone, you see more plausible results from a software engineering perspective (in particular, Terra no longer outperforming Fable) [1].
That especially the small flash models that are primarily optimized for coding tasks would perform worse at non-coding tasks is not per se surprising to me. The introduction of Terminal-Bench Science 0.1 in particular hurts their aggregate scores [2].
The general question remains of how meaningful aggregating the results of very heterogeneous tasks into a single numeric score is. (Aside from general questions about individual benchmark validity.)
Note: the cyber index is also interesting. While refusals make it somewhat opaque, even if you optimistically count all refusals for Opus and Fable as successes, the results are ... fascinating [3].
I'll add that I really want to know how Opus 4.x and Codex 5.x perform on the new index. Once we have that information, this is going to give us a better understanding of how meaningful the scores are.
One of the new benchmarks is FrontierCode 2.0 by Cognition. AA seems to have accepted Cognition's benchmark scores without questioning, but they should have looked more closely.
On this benchmark, Anthropic's models are run using Claude Code; OpenAI's using Codex, and Gemini is run using... Cognition's own Devin (not Antigravity). Thus its score is much lower than it should be, knocking ~3 points off its AA score. All open-source models are also inexplicably run using Devin, damaging their scores as well.
This may even be intentional, if Cognition has beef with Google, since Antigravity was built using Windsurf's technology. But aside from that, it is to their advantage to make Anthropic/OAI look like leaders in the AI market. They are a company known to lie and exaggerate again and again, so I have trouble giving them benefit of the doubt.
It's interesting that the Stirrup scores are somewhat close to the specialized harness scores. Either Stirrup is quite excellent or is it that harness doesn't matter so much after a point?
This seems to be a hit and a miss, there's no visual indication that opus 5.5 is better even though in 9 tests out of 10 it will outperform astra especially when it comes to niche domains and will absolutely destroy them in anything requiring raw intelligence rather than a naive search, naive search is what makes openai models extremely capable in scientific fields though.
The score hit on smaller models is a welcome change especially when compared to models that nearly 10 times larger.
Grok being punished quite a lot is also a very welcome change as the model is frankly not that good.
I would love to see few more points knocked off haiku by introducing steerability/creativity benchmarks.
25 comments
[ 0.28 ms ] story [ 7.1 ms ] threadAmerican models are too rich for my blood though. I use GLM 5.3 Flash more than any other model and it has the same issues as Mimo. Deepseek 4.1 Flash is turning out to be a sweet spot for me; it's cheap and blazing fast on CoralBricks where I've been using it, so I don't mind when it overthinks (I try to pay attention to what my agents are doing, and it tends to finish tasks before I can skim half of its reasoning)
I don't find it surprising that especially smaller Flash models that have been optimized primarily for coding fall behind on non-coding tasks and that their overall score suffers in comparison.
It does give rise to the question of how useful a single aggregate numeric index of very heterogeneous capabilities is, though.
Look also at the cyber index; while somewhat difficult to extrapolate because of refusals, even if for Opus 5.5 and Fable 5.1 you optimistically count all refusals as successes, the results are ... interesting.
[1] https://artificialanalysis.ai/#coding-agents-tabs
[2] To what extent Terminal-Bench 4.0 can be trusted, is a separate question, but unlikely to be biased towards one particular group of models: https://preferencemodel.com/blog/terminal-bench-4-is-saturat...
It is hard to tell what people are trying to say when they use undefined words with no particular meaning. This doesn't seem to change the relative position of the Chinese models much. The Chinese are a month or few behind and the state of intelligence is so volatile that the top 10 will be completely different in 6 months. I'd expect it to still be a mix of US and Chinese models with the US models leading.
One can argue though some models are post-trained heavily for specific benchmarks and less for other datasets. So they are weaker on out of distribution tasks.
- Exact problems on benchmark.
- Similar problems to benchmark.
- Broad class of problems or techniques that might apply to benchmark
Each has very different ramifications for how the model performs. The current evidence is that Anthropic and OpenAI is generally operating lower in the list.
That especially the small flash models that are primarily optimized for coding tasks would perform worse at non-coding tasks is not per se surprising to me. The introduction of Terminal-Bench Science 0.1 in particular hurts their aggregate scores [2].
The general question remains of how meaningful aggregating the results of very heterogeneous tasks into a single numeric score is. (Aside from general questions about individual benchmark validity.)
Note: the cyber index is also interesting. While refusals make it somewhat opaque, even if you optimistically count all refusals for Opus and Fable as successes, the results are ... fascinating [3].
I'll add that I really want to know how Opus 4.x and Codex 5.x perform on the new index. Once we have that information, this is going to give us a better understanding of how meaningful the scores are.
[1] https://artificialanalysis.ai/#coding-agents-tabs
[2] https://artificialanalysis.ai/evaluations/terminal-bench-sci...
[3] https://artificialanalysis.ai/#cyber-index-tabs
On this benchmark, Anthropic's models are run using Claude Code; OpenAI's using Codex, and Gemini is run using... Cognition's own Devin (not Antigravity). Thus its score is much lower than it should be, knocking ~3 points off its AA score. All open-source models are also inexplicably run using Devin, damaging their scores as well.
This may even be intentional, if Cognition has beef with Google, since Antigravity was built using Windsurf's technology. But aside from that, it is to their advantage to make Anthropic/OAI look like leaders in the AI market. They are a company known to lie and exaggerate again and again, so I have trouble giving them benefit of the doubt.
[1] https://artificialanalysis.ai/evaluations/frontiercode-2-0
It's interesting that the Stirrup scores are somewhat close to the specialized harness scores. Either Stirrup is quite excellent or is it that harness doesn't matter so much after a point?
The score hit on smaller models is a welcome change especially when compared to models that nearly 10 times larger.
Grok being punished quite a lot is also a very welcome change as the model is frankly not that good.
I would love to see few more points knocked off haiku by introducing steerability/creativity benchmarks.
https://x.com/zhengyaojiang/status/2100316960772800742
https://news.ycombinator.com/item?id=50268868