I find it fascinating that some of the main areas people are interested in are using benchmarks with quite few questions: ~200 for coding and ~150 for science.
As long as that's the case, it's no surprise that there is often a perceived difference of quality between benchmarks and real-world use.
I know that running the full composite benchmark set is probably already quite costly, and also that the benchmarks have to be rotated regularly, but it feels like a significant untackled problem.
I think it's pretty evident that Artificial Analysis loves to change the benchmark to place US models on the top, it's no the first time either.
The current placement of many models simply doesn't makes sense, GLM 5.3 is so much better than Astra at backend tasks, Astra does the most weird nonsensical bugs.
This is what happens when the benchmark team is funded by VC capital.
Even and especially for frontend UI, GLM 5.3 is a whole world better than Astra.
I can see (and believe) Astra beating GLM 5.3 on verifiable hard domains like maths, etc; but we’re at this stage where intelligence scores and even ‘frontier real world unsaturated benchmarks’ really doesn’t reflect real world use anymore.
The thing that makes the least sense is that Haiku 5.5 is apparently better than every single open weight model in existence except Kimi K3.1. Which is just... not true. I feel like I don't even have to explain it to anyone who has touched both sides of that equation for more than three minutes.
chinese labs are probably benchmaxxing but i'd like to see when the index changes relative to the position of the best chinese models (last suspiciously-timed update was when glm-5.3-flash was looking really strong)
The natural language choice in multilingual index seems pretty strange and biased to me, they tried to encompass a broad swath of Asian languages, but then there's like four European romance languages. They didn't include any Slavic one i.e Russian, or Nordic one like Swedish, or Finno-Ugric. And these romance ones are probably overrepresented in the training data anyways
Glancing over the methodology[1] made me curious about three things:
1. Is there a benchmark measuring how much better an LLM performs when you tell it "you are an X"?
2. Does telling an AI agent "you are an AI agent" actually help? Is it the LLM equivalent of reminding a senior developer they're a senior developer, making them feel guilty about being lazy and finally getting them to do their job?
3. What happens when the persona conflicts with the task? If you tell it "you are a PM" and then ask it to write code, does it struggle? Produce worse code? Spawn a sub-agent and tell it they're a developer?
Honestly it does somewhat match what I’ve felt playing a lot with all OpenAI / Claude / deepseek models
One thing I’m still surprised / unsure on is haiku 5.5 max besting latest deepseek - but also I think the last haiku was so incredibly bad I still hold some bias whilst using the new one.
It is embarrassing how much they suck up to american labs while those models only perform the first few days after release. With open weights you can be sure the quality stays the same unlike the closed source crap. How about a retest after some time?
Performance depends so much on configuration parameters like KV cache that you have essentially no idea what you're getting unless you run and benchmark everything yourself, which I'm sure you do not do. If anything, hosters of open weight models are more likely to gimp it either due to incompetence or malice because they have no reputation to lose.
Yeah, this smells. Ain't no way Terra went up and Deepseek so far low. I have had the later beat the former on multiple criteria by simply being less distracted.
There was a thread here about families of models responding in the same patterns to specific queries like "say a random word", and it takes less than two minutes to verify it is true. It'll take a bit of effort to not convince me this change wasn't reverse-benchmaxxed to screw everyone other than OpenAI. I'll be less distrustful after everyone is done IPO-ing.
The eloquence of numbers is pitiless; thanks for summing them up. There are players filing for IPOs who don't want to lose appeal while contrasted to open-weight counterparts, which can be run on prems, or accessed as a service offered for a bargain.
More to the point, I found fable to be actually more "intelligent" than opus once you step completly outside of the technical side of things (and I don't only mean code here), so for me it's hard to judge.
Eg asking "here are all the contract I'm subscribed to, here is what happened, is there any insurance that covers me and if yes which one is the best to make the claim to and why, and provide all the article to prove why and how i'm not in the exclusion list". Try it with random stuff of your every day life. Opus is both great and not great.
This might be the first time I’ve favourited a post without upvoting it.
This is really sus. I’d love a more detailed explanation and “take” from AA on what problem they felt needed correcting and why things have shuffled the way they have. And how they can claim their index has relevance if positions can be switched around so easily.
small (700b) open-weights models are great at coding and tool calling but lack world knowledge. because the parameter difference is so huge T to B, of course they also are not as intelligent.
this is i think the difference in this new analysis.
Their problem is really that individual benchmarks are not that informative. Bad enough models all perform at chance level, good enough models all saturate the benchmark. And the number of tasks is often low enough that differences of several percentage points might just be noise. Artificial Analysis' approach of aggregating scores in a weighted sum is largely just choosing which noise to amplify more.
I prefer Epoch's https://epoch.ai/eci methodology as they explicitly model benchmark saturation and also provide error bars for their estimates.
I noticed last night that openai had fallen way down the index, and I allowed myself to think for a minute that maybe AA aren't as biased as I thought. Then a few hours later they push this fix which sees openai rise again.
This update is even worse than usual because it now includes two private sources, data which we can't reproduce if we wanted to do.
Also note that Anthropic is the only AI company actually caught red handed cheating at evals. Teaching their models to memorize test answers and lie about it, using their hidden reasoning tokens to hide the deception [0]. Yet we see no comment from AA about this. No desire to figure out which tests are tainted. Hopefully Trump bans this fake intelligence index.
I'm still using DS-flash - it beats frontier model on speed and cost keeping good enough code quality.
It depends on the harness and constraints you give to the model but for most of the tasks its a great fit, I have not switched from DS since couple of weeks already (previously I would switch to Opus for some heavier work but I stopped recently).
My harness consists of multi-level AGENTS.md and SOUL.md containing "grug brain developer" and "asd-ste100" type of speech for clearer communication.
For me, DS Flash quality is still insufficient for bigger / more complex tasks. DS has gotten better by a lot, but I prefer Opus 5.5 because it gets things right with much fewer tokens. DS also sometimes gets stuck in a thinking loop "hmm... hmmm... hmmmm hmm hmmmm..." which is hilarious when you check on it after 30 minutes
Totally makes sense, for me it was just that I observed its doing good work so that I never really need anything stronger.
btw. I'm not reviewing code in detail anymore. I only give direction and provide architecture design vision, its the model and couple of async agents working in the background and consuming work resulting in PRs which I manually approve after an eye-glance.
The writeup fails to disclose that they quietly redacted several open-weight models from the new benchmark. Popular ones. They're not on the charts; they're not available to manually add to the charts; and their individual model pages now display scores of "N/A".
If they had an excuse, they'd have offered it already. Deleting one of their benchmark's top performers (GLM 5.3-flash was on their cost/intelligence Pareto frontier) is a highly visible change; if they chose to be silent about why did that, that silence is loud.
Their previous updates have sometimes been really bad for Chinese models. Either by exclusion, delays or simply weighting things differently so OpenAI looks better.
This feels like the opposite of benchmaxing. Maybe "minbenching" can become a word now?
53 comments
[ 0.18 ms ] story [ 4.2 ms ] threadThey perform pretty badly on the new benchmarks introduced.
As long as that's the case, it's no surprise that there is often a perceived difference of quality between benchmarks and real-world use.
I know that running the full composite benchmark set is probably already quite costly, and also that the benchmarks have to be rotated regularly, but it feels like a significant untackled problem.
The current placement of many models simply doesn't makes sense, GLM 5.3 is so much better than Astra at backend tasks, Astra does the most weird nonsensical bugs.
This is what happens when the benchmark team is funded by VC capital.
I can see (and believe) Astra beating GLM 5.3 on verifiable hard domains like maths, etc; but we’re at this stage where intelligence scores and even ‘frontier real world unsaturated benchmarks’ really doesn’t reflect real world use anymore.
Why are they using a closed source LLM as a judge? Closed source providers can see your traffic and adjust as they see fit.
AAv5.0: 25% Knowledge work: AA-Briefcase v1.1 - 15% AutomationBench-AA - 5% GDP.pdf - 5% (-5%) 25% Coding: FrontierCode 2.0 - 15% (new) Terminal-Bench4.0 - 10% 25% Science: Terminal-Bench Science .1- 15% (new) CritPt v2 (hard subset) - 10% (new version) 25% General: AA-Omniscience v1.1 - 15% HLE-Diamond - 10%
AAv4.3: 30% Agents: AA-Briefcase v1.1 - 15% GDPval-AA v2 - 10% AutomationBench-AA - 5% 20% Coding: Terminal-Bench4.0 - 10% SciCode - 10% 30% General: AA-Omniscience (Accuracy) - 10% AA-Omniscience (Non-hallucination) - 5% GDP.pdf - 10% AA-LCR v1.1 - 5% 20% Scientific Reasoning: HLE - 10% CritPt 10%
tldr: gained Terminal-Bench Science and FrontierCode, dropped SciCode, GDPval-AA and AA-LCR. Lowered weight of GDP.pdf.
1. Is there a benchmark measuring how much better an LLM performs when you tell it "you are an X"?
2. Does telling an AI agent "you are an AI agent" actually help? Is it the LLM equivalent of reminding a senior developer they're a senior developer, making them feel guilty about being lazy and finally getting them to do their job?
3. What happens when the persona conflicts with the task? If you tell it "you are a PM" and then ask it to write code, does it struggle? Produce worse code? Spawn a sub-agent and tell it they're a developer?
[1] https://artificialanalysis.ai/methodology/intelligence-bench...
One thing I’m still surprised / unsure on is haiku 5.5 max besting latest deepseek - but also I think the last haiku was so incredibly bad I still hold some bias whilst using the new one.
Or improves. With third-party fine-tuning.
Kimi K3.1 Pro: −9
DeepSeek V4.1 Flash 1017: −13
GLM-5.3: −13
MiMo-V2.6-Pro: −16
Proprietary models
GPT-6 Astra: +4
GPT-6.1 Sol: +3
GPT-6.1 Terra: +3
Claude Opus 5.5: 0
Claude Sonnet 5.5: −2
Claude Fable 5.1: −3
Gemini 4 Argon (high): −3
Muse Wonder Full: −6
Claude Haiku 5.5: −7
Muse Wonder: −9
Gemini 3.8 Flash (high): −10
Grok 4.7 (high): −11
There was a thread here about families of models responding in the same patterns to specific queries like "say a random word", and it takes less than two minutes to verify it is true. It'll take a bit of effort to not convince me this change wasn't reverse-benchmaxxed to screw everyone other than OpenAI. I'll be less distrustful after everyone is done IPO-ing.
Eg asking "here are all the contract I'm subscribed to, here is what happened, is there any insurance that covers me and if yes which one is the best to make the claim to and why, and provide all the article to prove why and how i'm not in the exclusion list". Try it with random stuff of your every day life. Opus is both great and not great.
Will an LLM topping this bench be the end of humanity? The doomer rhetoric has no place in what should be an objective measure.
This is really sus. I’d love a more detailed explanation and “take” from AA on what problem they felt needed correcting and why things have shuffled the way they have. And how they can claim their index has relevance if positions can be switched around so easily.
this is i think the difference in this new analysis.
doesnt mean deepseek is bad or useless.
I prefer Epoch's https://epoch.ai/eci methodology as they explicitly model benchmark saturation and also provide error bars for their estimates.
What happens when you release a new benchmark? Models magically start scoring better against it.
How do you overcome this? Release a new benchmark and start the process over again!
All other things being equal, they should retarget their prices if models start gaming the process to top their index.
Has this happened? Maybe. Is their new scoring system biased towards the American frontier models? Quite possibly.
I have my suspicions, but they have led me astray before. I will continue to be cautiously cynical, though.
This update is even worse than usual because it now includes two private sources, data which we can't reproduce if we wanted to do.
Also note that Anthropic is the only AI company actually caught red handed cheating at evals. Teaching their models to memorize test answers and lie about it, using their hidden reasoning tokens to hide the deception [0]. Yet we see no comment from AA about this. No desire to figure out which tests are tainted. Hopefully Trump bans this fake intelligence index.
[0] https://arxiv.org/abs/2608.09867
That seems improbable
My harness consists of multi-level AGENTS.md and SOUL.md containing "grug brain developer" and "asd-ste100" type of speech for clearer communication.
btw. I'm not reviewing code in detail anymore. I only give direction and provide architecture design vision, its the model and couple of async agents working in the background and consuming work resulting in PRs which I manually approve after an eye-glance.
Hard to trust that a for-profit benchmarking company isn't biased due to profit-maxxing.
GLM 5.3-flash: N/A
https://artificialanalysis.ai/models/glm-5-3-flash/
Qwen 3.8-flash-next: N/A
https://artificialanalysis.ai/models/qwen3-8-flash-next
If they had an excuse, they'd have offered it already. Deleting one of their benchmark's top performers (GLM 5.3-flash was on their cost/intelligence Pareto frontier) is a highly visible change; if they chose to be silent about why did that, that silence is loud.
This feels like the opposite of benchmaxing. Maybe "minbenching" can become a word now?