Terminal bench 4 is good largely because it's recent so it hasn't been benchmaxxed yet. It's more of a sysadmin/devops benchmark than a coding benchmark though, but still a decent proxy.
So TL;DR benchmarking in a completely non-reproducible manner ?
"Model X performed great, but we can't possibly tell you anything about the code it was looking at apart from it was a large code base from an unknown company".
In theory, as long as all the models are doing the same thing with the same tools, it's at least useful to see how they stack up against each other right now. It might not be great to track progress over time, as it can get benchmaxxed or the underlying resources may become obsolete.
My gut feeling is that any serious real-world company with a proprietary codebase worth looking at would not be handing out the crown jewels to a third party. License or not.
I don't doubt somebody licensed their codebase to them, I just have my doubts about who the "who" could be.
Code isn't worth all that much if you don't own the associated IP, mainly copyright. And even if you disagree with that premise, if you trust that they can keep the code secret, it's basically free money.
At any rate, I'm not sure it matters whose codebase it is. I'd even say that a shitty codebase might make for a better test.
The point I'm making is that companies who are serious enough to want to keep their code-base in-house and off the various online repo services are also the kind of companies who are strict about what you can and cannot do with LLMs (if they permit use of LLMs at all).
So it does not make sense that the same companies would then magically sign-off on allowing their entire codebase to be spoon-fed into a whole bunch of LLMs for benchmarking.
lot of ads everywhere offering to buy your codebase of real product/star up even it long gone or failed (offer usually price per lines of code). So most likely that they have bunch of abandoned codebases between small and medium sizes and probably also some fake codebases as well.
we reached out to companies that were willing to license their codebases. every codebase we used had real users, one of them had 200k+ users and is currently top 100 on the app store.
I think these benchmarks are not that useful, e.g. this suggests Fable is better than Astra, but in practice Astra is waaaaaay faster (like 5x; it's not even close), and also waaaay less annoying to talk to.
There's only two or three sane options here - you can easily try them all and pick yourself.
I switched from Claude to Codex because Claude just doesn't do what you actually tell it to half the time. It dances around the edges and does busy work without actually tackling a tough problem.
I'm not sure what others are doing that they're getting such different results, but I'll take Codex every day of the week.
The fact that gemini 3.8 flash is so high up there just tells you this is an awful benchmark.
Try and use gemini 3.8 yourself for any real world work and you'll see it's terrible. It'll just go in circles reading the same file 20 times for no reason making hundreds of tool calls for a simple change.
Hard disagree. I use 3.8 flash in Antigravity a lot, and thoroughly prefer it to most Pro-class models. It's really fast, and I've had it make crazy progress on compiler-like problems that previous models including Opus simply failed at. On ultra plan you can have it going for hours, and make incremental progress with good prompting for review interrupts. It solved a problem I couldn't solve for weeks in under 6 hours. 10k LOC total. The harness and test suite is key.
There's no such things as gemini cli these days. It's called "agy" (short for antigravity). And if you don't know what that is, you're probably 3-6 months behind already.
PS: Just Googled it to confirm: Gemini CLI was deprecated on May 19th, 2026. The correct harness is called agy or antigravity for Gemini 3.8 Flash.
The benchmark page itself asserts that it used Gemini CLI as a harness. I came to the comments just because I noticed the error. For my part — using agy — I found Gemini 3.8 Flash mid.
My intuition is that many of the better & bigger 'private' code bases, at least in terms of claude code and codex... are not in fact private at this point.
One lesson of running botsbench.com, in a slightly different domain, is to measure for model contamination every time.
I spent a while in big tech and remember several unique patterns of internal code based. Your comment inspired me to try to coax ChatGPT into spitting out code that was inspired by proprietary, private code. Surprisingly, it did it with no problem - I referenced an idiom from a tech company and it wrote code that really would have only been relevant for that vertical. When I asked how it learned the pattern, it said “from my learned understanding of CompanyX’s internal coding conventions”. When I asked “how do you know about those internal conventions” ChatGPT said “I don’t have access to that internal code, I overstated what I know”. Internal coding conventions are the least of our worries at this point, cat is out of the bag.
much harder to do in OP's case, matching flavour then referencing that specific companies name when asked how it know to flavour this way? thats astronomically low for randomly selected plausible tokens without some data prior, like that companies codebase.
My own experience is opus being lousy at an extremely niche math task, but it was still easier for me to describe what it needed to do to get code and correct issues in its reasoning/working than to write myself. a minor model number change later and it's nailing everything, despite my opt-out. Its is astronomically unlikley others were working on this also, especially at that level, especially this application.
so, safe to say they _all_ train models on chats, the only difference being if you "opt out" you at least have some defence later when they steal your work and claim it as their models original output.
For us it's ~easy: we do security investigation CTFs against log dumps / DBs like Splunk. We run the AI CTF as usual.. except without the DB nor internet access. If they still get some right, that's a problem. (And most public CTFs are contaminated now for that reason.)
There's a funny middle where they may not know the exact answer confidently, but do know where to get them, or aspects of the problem. That's still a potential lift in the competition . So we also check the traces to see if there is any knowledge during intermediate step DB queries that it shouldt have based on the previous steps.
I gave a recorded talk at BSides Las Vegas a few weeks ago on 'adversarial evals' that gets into the kind of cheating we've been having to deal with.
You cannot introspect model training by prompting the model. Whatever answer it hallucinated on your query "where do you know that from" has almost certainly very low bearing on reality.
this is the closest benchmark to my experience using the model harness combo. Astra for as great as it is falls slightly behind Fable 5.1 for me for large feature work (although it comments code much better). in particular, Fable is able to assess priority better than Astra (meaning Astra sometimes does things that aren’t worthwhile while missing things that are clearly important, particularly on possible ballooning scenarios- fable catches “this works for x amount of data but if we run this on y way greater than x amount of data we’ll run into issues). Gemini 3.8 is under appreciated, use Google Stitch to see it in action if you haven’t used Agy yet.
Astra constantly does this for me. It goes 90% of the way with some task but then skips the most important part. Then when told to please fix that and do it properly, it suddenly goes down a rabbit hole for 6h and fixes scenarios that aren't even relevant. It's awful at assessing what is important to do and what not, and where to ask for permission and where not.
Do you find Fable significantly better than Opus at avoiding-overengineering? All of my recent testing of Anthropic models seems like they're tuned-to-hell to (a) be much slower than they need to be (running tests over and over during the loop vs at the end, say, even if those tests take a few minutes a pop) and (b) doing exactly that sort of "built a lot of fancy enterprisey feature-adjacent 'stuff'" even before nailing the actual feature. Sol and Terra both have some of the latter but they seem to do the actual work a fair bit faster (this may be a usage-based-priority-tier/rate-limit thing though) which helps offset it.
I think the bigco folks saw all the "it wrote all this code but the tests didn't pass" or "it wrote the feature but it's super brittle" and tuned the newer model+harness combinations incredibly aggressively to try to turn a lazy prompt into "median Enterprise Architecture design suggestions" to bring up the baseline, but in a way that slows you down if you don't want that.
I'm not on big enough subscriptions to want to burn a lot time just evaluating Fable/Astra comparatively until they're cheaper, heh. I can steer any of the cheaper ones just fine anyway.
i think this likely depends on workflow. for me, the first step is always a plan file artifact on disc, which i heavily review and go back and forth until satisfied. i often have to split the plan into multiple phases because agents are still poor at assessing how large a change will be. sometimes before even starting the plan the task is to create a harness for validation (a way for agent to check their work). the codebase you drop them in also greatly changes how much “over” engineering they do. i think if you are good about reviewing plan files and managing scope for your agent these sort of issues fade away. in general though, gpt models are faster, more token efficient, better technical writers, and yeah i still find Fable 1.0 to be a step up even though it’s slower because it’s actually a tad smarter (and the more important and dynamic the change / feature the more fable 1.0 shows its slight edge over astra)
personally, i would not rely on opus 5 end to end as it'll start getting into walls of comment slop and shitting up the codebase similar to gpt 5.5's isRecord meme.
on the other hand, having fable plan and orchestrate with opus implemention + fable reviews, is my go-to. if you give fable your guidelines up front or in your {claude,agents}.md, it will keep opus on a tight leash. opus can still write great code almost on par with fable, but it needs to be tightly constrained.
I am trying to estimate if my reaction to seeing GPT-5.6 Sol last on that list is reasonable or or mostly emotional and find that I have no way of telling.
Sol failing mostly on “unverified assumptions” and rarely hitting “integration errors” seems about right to me. I think Sol is second only to Astra (and miles ahead of even Fable) in architecting & engineering the right implementation — but only if you are extremely specific and provide tight guidelines and guardrails. If you give it a one-liner… you’re going to have a bad (SHA-256-hash-verified) time.
>> I think Sol is second only to Astra (and miles ahead of even Fable) in architecting & engineering the right implementation — but only if you are extremely specific and provide tight guidelines and guardrails.
To me, having to give extremely specific instructions and provide tight guidelines and guardrails defeats the purpose of agentic coding agents almost completely. At that point I might as well do the task myself.
With Fable I can start with a general ask like "I'm trying to do X, can you investigate and tell me what the shape would look like" and have it poke around and think, ask me questions with single-choice or multiple-choice answers, then break the task into small chunks, each of which becomes a ticket.
With Astra, it's like pulling teeth. It often does not understand what I'm trying to do, takes things literally, does not go above and beyond (i.e. infer intent), and stops way too short of the actual goal. I have to constantly prod it and it's frankly exhausting.
I agree somewhat with the way the agents behave but feel the opposite reaction. With Fable, I get exhausted because it's always dumping out paragraphs of text that explain one approach but have some secret gotcha thrown out in the last two sentences. Then I have to pause and consider the caveat and if it matters and it happens every single time Fable responds and that constantly needing to make a decision that could radically change the approach gives me decision fatigue. I much prefer how much more decisive Astra can be.
I'm the opposite. Every time I've let Sol/Astra be decisive, I ended up with an overengineered mess.
I much prefer getting alerted when there's more than 1 approach to the problem and it's discovered mid-implementation.
I don't want to do the grunt work of writing code, but I do want to know the architecture and be responsible for the decisions.
Fable is also very good at pushing back when I propose something that will cost me. E.g. I'm working on a configuration layer above nix to manage my homelab fleet declaratively, and I tend to get into "config as new language", where Fable just goes - let's not do that.
Oh man! This also is a pet peeve of mine with Fable. I will look at what it's doing and say "Shouldn't it be done this way?" and then it will spend forever arguing with me that it should be done the way it wanted to do it. It seems to get stuck in a certain way of thinking and will insist its way is right until I can really prove it - or just go over to Astra.
Any bench that puts GLM 5.3 ahead of 5.6 Sol is highly sus. They've been my two daily drivers since release, and I like GLM 5.3, but it's definitely not better than Sol, it's more ~Terra, while being significantly slower.
I had exactly the same thoughts. I often have similar thoughts on other benchmark sites, where supposed performance is way off base from my experience.
I’m not sure what the methodology of these are, but they certainly don’t match what I experience. Maybe I need to look deeper for relevant benchmarks.
There's an issue with GPT-5.6 Sol where it sometimes starts mixing thinking with output and stops working[1]. Once it starts doing that, the session is essentially cooked and you need to do a bit of gymnastics if you want to recover it.
This happens to me more commonly in large projects (>100k LOC) and in those projects it seems to happen every few sessions. I feel this specific benchmark would be impacted by this more than the smaller contrived benchmarks.
Honestly matches my experience with Sol and the fact that all the random message boards appearing on the internet are attributed to Sol then it makes sense that its the one which makes up most requirements.
Does this mean they ended up sharing those private codebases with OAI, Anthropic etc? Also, the ~30% number tracks with my experience. I thought I was going insane for expecting too much from the models but they are still bad, including astra. This morning it messed something pretty trivial while fixing an issue which I was shocked to see. Also2, benchmarks don't mean much these days.
Can you please share, if you are comfortable of course, what did the model(s) mess up? what were you using codex/cc/pi? did the project have a solid agent.md/claude.md? I am genuinely curious whenever someone have such a low success rate with models what is happening because it could be fixed maybe?
From my own experience using agents for the past year or so. The rate if I have to guess, is well above 70%. I mainly use claude (opus) on typescript react projects that are well setup with minimal plugins/MCPs!
> For instance the task naming in the task file starts with an optimistic 1, 2, 3, 5, 5a but then eventually gets to 8a, 8a1, and then ends up with 8b2c2b3 and “8b2c2b2b checkpoint1”. The code that it produced got ever more wild. I don’t want to bore you with what it tried to build, but here are some example pieces of the interpreter changes:
Hardcoded constants everywhere
Multiple same-line macro invocations in C
Random indexes in production code
Hideous tokenizer code in C
Just earlier Opus 5 was supposed to migrate a Python codebase from kafka-python client-per-send to a client singleton. It decided it should add a partition key "that nobody asked for" which it admitted to at the end of the implementation.
It randomly decided going from no explicit partition key which would randomly add to a Kafka partition to partioning key company was the right solution.
In fact, that was fairly counterproductive since this codebase dumps events to the topic per company so you'd get a lumpy distribution where only one partition is ever active at once (starving other consumers)
The fact it decided it should add this without prompting was a bit shocking and I'll probably try to make it explicitly clear in the context not to "do random shit that wasn't asked for"
One thing I've always been curious about is, often times it seems that models don't seem to have these addendums like "dont do excessive/random shit" by default? Or I suppose, if it did have something like that, and still screws up like in your example, it clearly isn't working, so assume that it doesn't.
But the phrase "don't do random shit", semantically, from a monkey's paw perspective, could imply curtailing the model's creativity and 'thinking out of the box' capacity, that might have existed in its 'reasoning' process. So I'm always concerned if it's possible that, adding these phrases might be part of the reason why a model performs dumber than it should.
I don't have empirical evidence to support that supposition though.
I think of it as a tradeoff between creativity and specificity. Every instruction you give reduces creativity, and at best, increases specificity (I imagine a lot of prompts like "make no mistakes" do literally nothing but pollute context, but I haven't evaled them)
So if you're doing something very ordinary, fewer instructions result in better results. If you're doing something fairly off-piste, you have to give instructions to that effect and accept less creativity. For situations where you want it to do something extremely specific, tons of instructions and accept that you're going to get much closer output but much worse "intelligence"
Another way to think about it is Type 1 and Type 2 errors or sensitivity and specificity from statistical testing - do you want an agent that solves any problem but goes off the rails 10% of the time, or do you want an agent that can only solve 10% of the problems but nails them 100% of the time (sensitivity and specificity, respectively)
This is why I like to really interrogate Claude on what it’s planning to do, what the implementation will look like etc before writing code. Without enough direction and time thinking about it, it can definitely go in an unintended direction
The high score of Gemini 3.8 Flash vibes with my experience anecdotally. While it often goes off the rails with open-ended questions (which is a strength if taken with care), it is also a good at solving issues in a well-defined environment like an enterprise codebase.
Unlike TFA, the lone lisp code is public. I suppose the models could have been trained on my codebase. Still, I think it produced interesting results.
Took months and loads and loads of tokens to do this, so I'm not gonna repeat this study as new models come out. It did anchor all of my future expectations, though. OpenAI is winning as far as I'm concerned, and their cybersecurity program is the only remaining pain point.
143 comments
[ 0.25 ms ] story [ 38.6 ms ] threadhttps://artificialanalysis.ai/evaluations/terminalbench-v4-0
"Model X performed great, but we can't possibly tell you anything about the code it was looking at apart from it was a large code base from an unknown company".
So basically pinky-promise benchmarking ?
I'm not sure I follow the value here ?
How does that work?
My gut feeling is that any serious real-world company with a proprietary codebase worth looking at would not be handing out the crown jewels to a third party. License or not.
I don't doubt somebody licensed their codebase to them, I just have my doubts about who the "who" could be.
At any rate, I'm not sure it matters whose codebase it is. I'd even say that a shitty codebase might make for a better test.
So it does not make sense that the same companies would then magically sign-off on allowing their entire codebase to be spoon-fed into a whole bunch of LLMs for benchmarking.
There's only two or three sane options here - you can easily try them all and pick yourself.
I'm not sure what others are doing that they're getting such different results, but I'll take Codex every day of the week.
Try and use gemini 3.8 yourself for any real world work and you'll see it's terrible. It'll just go in circles reading the same file 20 times for no reason making hundreds of tool calls for a simple change.
And it just BURNS tokens like crazy.
PS: Just Googled it to confirm: Gemini CLI was deprecated on May 19th, 2026. The correct harness is called agy or antigravity for Gemini 3.8 Flash.
https://developers.googleblog.com/an-important-update-transi...
One lesson of running botsbench.com, in a slightly different domain, is to measure for model contamination every time.
My own experience is opus being lousy at an extremely niche math task, but it was still easier for me to describe what it needed to do to get code and correct issues in its reasoning/working than to write myself. a minor model number change later and it's nailing everything, despite my opt-out. Its is astronomically unlikley others were working on this also, especially at that level, especially this application.
so, safe to say they _all_ train models on chats, the only difference being if you "opt out" you at least have some defence later when they steal your work and claim it as their models original output.
There's a funny middle where they may not know the exact answer confidently, but do know where to get them, or aspects of the problem. That's still a potential lift in the competition . So we also check the traces to see if there is any knowledge during intermediate step DB queries that it shouldt have based on the previous steps.
I gave a recorded talk at BSides Las Vegas a few weeks ago on 'adversarial evals' that gets into the kind of cheating we've been having to deal with.
I think the bigco folks saw all the "it wrote all this code but the tests didn't pass" or "it wrote the feature but it's super brittle" and tuned the newer model+harness combinations incredibly aggressively to try to turn a lazy prompt into "median Enterprise Architecture design suggestions" to bring up the baseline, but in a way that slows you down if you don't want that.
I'm not on big enough subscriptions to want to burn a lot time just evaluating Fable/Astra comparatively until they're cheaper, heh. I can steer any of the cheaper ones just fine anyway.
on the other hand, having fable plan and orchestrate with opus implemention + fable reviews, is my go-to. if you give fable your guidelines up front or in your {claude,agents}.md, it will keep opus on a tight leash. opus can still write great code almost on par with fable, but it needs to be tightly constrained.
Otherwise I wouldn’t see others in the exact same codebase struggle and underutilize agents while others thrive using the exact same ones.
To me, having to give extremely specific instructions and provide tight guidelines and guardrails defeats the purpose of agentic coding agents almost completely. At that point I might as well do the task myself.
With Fable I can start with a general ask like "I'm trying to do X, can you investigate and tell me what the shape would look like" and have it poke around and think, ask me questions with single-choice or multiple-choice answers, then break the task into small chunks, each of which becomes a ticket.
With Astra, it's like pulling teeth. It often does not understand what I'm trying to do, takes things literally, does not go above and beyond (i.e. infer intent), and stops way too short of the actual goal. I have to constantly prod it and it's frankly exhausting.
I much prefer getting alerted when there's more than 1 approach to the problem and it's discovered mid-implementation.
I don't want to do the grunt work of writing code, but I do want to know the architecture and be responsible for the decisions.
Fable is also very good at pushing back when I propose something that will cost me. E.g. I'm working on a configuration layer above nix to manage my homelab fleet declaratively, and I tend to get into "config as new language", where Fable just goes - let's not do that.
Oh man! This also is a pet peeve of mine with Fable. I will look at what it's doing and say "Shouldn't it be done this way?" and then it will spend forever arguing with me that it should be done the way it wanted to do it. It seems to get stuck in a certain way of thinking and will insist its way is right until I can really prove it - or just go over to Astra.
I’m not sure what the methodology of these are, but they certainly don’t match what I experience. Maybe I need to look deeper for relevant benchmarks.
This happens to me more commonly in large projects (>100k LOC) and in those projects it seems to happen every few sessions. I feel this specific benchmark would be impacted by this more than the smaller contrived benchmarks.
[1]: https://github.com/openai/codex/issues/37524
happy to share more if you are interested.
It randomly decided going from no explicit partition key which would randomly add to a Kafka partition to partioning key company was the right solution.
In fact, that was fairly counterproductive since this codebase dumps events to the topic per company so you'd get a lumpy distribution where only one partition is ever active at once (starving other consumers)
The fact it decided it should add this without prompting was a bit shocking and I'll probably try to make it explicitly clear in the context not to "do random shit that wasn't asked for"
But the phrase "don't do random shit", semantically, from a monkey's paw perspective, could imply curtailing the model's creativity and 'thinking out of the box' capacity, that might have existed in its 'reasoning' process. So I'm always concerned if it's possible that, adding these phrases might be part of the reason why a model performs dumber than it should.
I don't have empirical evidence to support that supposition though.
So if you're doing something very ordinary, fewer instructions result in better results. If you're doing something fairly off-piste, you have to give instructions to that effect and accept less creativity. For situations where you want it to do something extremely specific, tons of instructions and accept that you're going to get much closer output but much worse "intelligence"
Another way to think about it is Type 1 and Type 2 errors or sensitivity and specificity from statistical testing - do you want an agent that solves any problem but goes off the rails 10% of the time, or do you want an agent that can only solve 10% of the problems but nails them 100% of the time (sensitivity and specificity, respectively)
its under this section 'Missed requirements are the most common failure' on realswe.withspecific.com
we also have the setup in the blog. the reason for lower success rates is that we gave models ambitious tasks that real engineers worked on for weeks.
- DeepSeek V4.1 Flash
- Kimi K3
- GLM 5.3 (and flash)
- hy4-preview
- Grok 4.6
All of these can be acessed using a $10/mo OpenCode Go subscription.
1 Fable 5.1 38.8%
2 GPT-6 Astra 33.8%
3 Gemini 3.8 Flash 31.2%
4 GLM 5.3 28.8%
5 Grok 4.6 23.8%
5 Muse Spark 1.3 23.8%
7 Kimi K3 18.8%
8 GPT-5.6 Sol 16.2%
See number 4, 5 and 7.
Even posted an article about it:
https://www.matheusmoreira.com/articles/code-reviewing-lone-...
Unlike TFA, the lone lisp code is public. I suppose the models could have been trained on my codebase. Still, I think it produced interesting results.
Took months and loads and loads of tokens to do this, so I'm not gonna repeat this study as new models come out. It did anchor all of my future expectations, though. OpenAI is winning as far as I'm concerned, and their cybersecurity program is the only remaining pain point.