I know the log scale of that chart is valuable when surveying the full breadth of possibilities, but I would encourage people to make decisions between nearby models based on linear scaling. Linear cost is often more useful for making the actual decision.
Would you please elaborate on "really decent"? Really decent for what use exactly? I want to love it (and use it as a cheaper research model in my harness), but I really cannot get any real development done with it? It stumbles and fumbles and over-complicates, where astra just plows ahead (often in less overall time) with excellent solutions.
I really want to love this model, particularly since it's good to have an open weight model on my team so to speak, but I think I might be holding it wrong.
I can have 2-3 sessions running at the same time for pretty much the entire working day with DSF4.1 where the responses are fast, cost is pretty negligible, and the results are good enough 95% of the time.
On an overall speed/cost/quality comparison, it seems unmatched in my experience these days.
I've been playing with the fast mode on BaseTen - clearly a different cost bracket, but WOW - the speed is phenomenal! A real game changer to be able to stay in flow.
Yeah, good point. For HTML/CSS/JavaScript it seems to do fine. Maybe I am just asking too much of it, with a codebase spanning Rust and Elixir and doing some more low-level stuff in-between...
I use that specific model as my primary LLM for coding and diagnostics on projects in Python, C++, CUDA, and for web research.
I'm also using it with my own harness (you mentioned "in my harness"), so I'll share two things I found about using open weight model providers. The first is that getting the configuration right isn't a per model thing, its a per model per provider per point in time thing. The point in time thing comes from middleware updates as those introduce or fix defects that can very seriously impact inference quality. The second thing is inference quality varies a lot between providers, even when they are ostensibly using the same quant of the same model.
If you want to try it something else for comparison I open sourced my harness a couple of weeks ago: https://fluximp.org
I recommend trying deepseek v4.1 flash from together.ai (not the 1017 one the one released last month). deepinfra.com is cheaper for that but its noticeably slower and the inference from together.ai feels like it might be better (not quantified; just vibes).
Thank you for the link to your harness! Looks quite interesting.
What type of ongoing tuning do you do for your model+provider+harness combo, and when do you know it's time to redo it? Or more specifically, when you do the tuning, how do you actually go about doing it?
I am mostly using omp (omp.sh) which is very tunable, and I did some tuning for one of the GLM models in the past (mostly based on vibes because people said it was getting dumb like a brick after some context lengths so I had omp compress sooner).
Every single model I've tested that is supposed to be anywhere near Opus, wasn't there. There is some magic sauce in Anthropic models that these benchmarks aren't demonstrating.
I think the secret sauce is about communication. It understands you and you understand it. Just a better coworker to work with. Benchmarks don’t measure that.
Yeah, but if you need to provide less details to Opus to get the same output it means the model is better at deducing your needs, which again implies it's a higher quality model.
It's been suggested elsewhere that the AA-Omniscience benchmark [1] is more representative than the rest. It puts K3.1 about 6 months behind the frontier and the new version of DS 4.1-flash about 12 months. That roughly lines up with my gut feeling as well, at least when it comes to the trickier problems. Still rather impressive, but not quite frontier.
Although even this one needs to be taken with a grain of salt. Fable/Opus still act very differently to Astra and Argon imo. But maybe it's just because I am used to these. I wouldn't clearly put them above each other though, although I still prefer anthropics models, but that could be simply out of comfort.
Huh, this is an interesting hypothesis! It would make sense, to a degree. Part of "big model smell" to me is the model being able to pull a huge range of interesting algorithms, architectures, mathematics, and everything else that would come with greater world knowledge and having a much more sort of innate, like, full, well-rounded understanding of it instead of surface-level ICL pattern matching on web results.
That definitely matches my experience, I have found Astra to be more capable than many other benchmarks show, have been very impressed with Argon for text, and generally like Sol. I find Haiku and Sonnet to be not worth my time as compared to the speed and cheapness of something like DeepSeek.
The only exception on the chart is DeepSeek - it scores exceptionally low there, which doesn't exactly track my experience. It usually does what you tell it to, and it's not exceptionally intelligent, but I do find it quite good at what it does (the bulk of code, scripting, etc). I know a lot of folks like it too ... curious if I'm just not exploring the simpler models enough?
I skimmed through the publicly available questions. The first programming-related one I saw has the answer in the question, not a great start :D [1]
Q: In C23, attributes are allowed in an unbraced (forward) union declaration only in the form “union [[...]] Tag;”. What single character must immediately follow the identifier in that declaration for the attributes to be permitted?
A: ;
A lot of these models are likely benchmaxxing the existing benchmarks that are out there. Some perhaps even by accident.
I think looking at some more niche benchmarks helps, I created one and it does show Kimi K3.1 Pro as the most intelligent of all open-weight models, though it is still significantly below Opus 5/5.5[1]
I'd say it's both. Models should be intelligent enough to be able to tell prompt injections apart. But the language itself is also novel and not in the model training data set, so that should be a good signal for intelligence in itself.
It really depends on the type of task you're doing and how you're using it, because for the past two month the only reason I've had to use my company's paid Opus plan instead of my local Qwen-3.8-Next was speed. I use Qwen as a back-up plan when my token limit is reached and in my day to day work they have been indiscernible in terms of output.
I think also heavily depends on what you're doing.
If you're in a stable codebase with lots of best practices, patterns, controls (linters, analyzers, etc.), examples already established, then any model will do ~the same plus/minus since there's not much novelty.
If you're building ground up, entirely novel, no reference, I think results will vary with lower capability models.
For me, I do a mix, leaving implementation to Haiku/Luna/DeepSeek and research/feedback/critique to Opus/Sol/Astra/Fable
Maybe I need to try it again, but I have never liked the opus models.
I keep having problems with them not doing the thing I ask, (and also failing to call tools correctly in pi. I think they've very much trained to the harness, which is frustrating.).
I recently just crossed the threshold for 100% decompilation of a PlayStation game. I used a lot of models in the process, and I feel like this is a good demonstration the "sauce" that opus has.
The harness I've built for it is pretty straight forward: split binary into individual functions, give the agent a bunch of small tools to diff output with underlying binary, make them wrote clean c until there's a certified match.
Rin a final oracle to make sure the agent isn't cheating.
- Qwen 3.8 27b on my 5090, barely acceptable, thinks too long, sometimes freaks out.
- deepseek flash variants, work horse. Runinfra provides around 300 tok/s. Great for tearing through moderately complex functions
- glm and kimi k.3, expensive, slow, ultimately not that much better than deepseek flash for this task. Better at doing long context research into old compiler productions
- sol 6.1, best bang for buck. Creative enough to methodically work through more complex spellings
But I found there were around 1% of functions that these models simply cannot solve. Opus and Astra take them handily.
I'm still still in the process of looking through the solution traces, but from watching the missing sauce seems to be creative reasoning.
Sol 6.1 will just happily grind on permutations of a base shape spelling that M2C outputs, where opus and Astra immediately realize that they need to unwind assumptions taken for granted (and sometimes in the process endure a worst match).
I'm hoping to be able to capture that creativity step and distill it into the harness tools.
I definitely feel something is "wrong" with these benchmarks.
I don't have much experience with anything outside of Claude for frontier models (and only haiku/3.x flash/5.6+6 luna at the entry level) so take this with that in mind.
I find Fable is still SIGNIFICANTLY 'smarter' then opus 5.5 when figuring out HOW to do something that's non-mechanical (e.g. How should we handle this aspect of business logic?"). Opus comes up with perfectly usable options, however Fable more often comes up with options that are more aligned with what I expect, or accept.
29 comments
[ 2.6 ms ] story [ 44.4 ms ] threadI really want to love this model, particularly since it's good to have an open weight model on my team so to speak, but I think I might be holding it wrong.
What are you using it for?
On an overall speed/cost/quality comparison, it seems unmatched in my experience these days.
I'm also using it with my own harness (you mentioned "in my harness"), so I'll share two things I found about using open weight model providers. The first is that getting the configuration right isn't a per model thing, its a per model per provider per point in time thing. The point in time thing comes from middleware updates as those introduce or fix defects that can very seriously impact inference quality. The second thing is inference quality varies a lot between providers, even when they are ostensibly using the same quant of the same model.
If you want to try it something else for comparison I open sourced my harness a couple of weeks ago: https://fluximp.org
I recommend trying deepseek v4.1 flash from together.ai (not the 1017 one the one released last month). deepinfra.com is cheaper for that but its noticeably slower and the inference from together.ai feels like it might be better (not quantified; just vibes).
What type of ongoing tuning do you do for your model+provider+harness combo, and when do you know it's time to redo it? Or more specifically, when you do the tuning, how do you actually go about doing it?
I am mostly using omp (omp.sh) which is very tunable, and I did some tuning for one of the GLM models in the past (mostly based on vibes because people said it was getting dumb like a brick after some context lengths so I had omp compress sooner).
1. https://artificialanalysis.ai/evaluations/omniscience
Although even this one needs to be taken with a grain of salt. Fable/Opus still act very differently to Astra and Argon imo. But maybe it's just because I am used to these. I wouldn't clearly put them above each other though, although I still prefer anthropics models, but that could be simply out of comfort.
The only exception on the chart is DeepSeek - it scores exceptionally low there, which doesn't exactly track my experience. It usually does what you tell it to, and it's not exceptionally intelligent, but I do find it quite good at what it does (the bulk of code, scripting, etc). I know a lot of folks like it too ... curious if I'm just not exploring the simpler models enough?
Edit: The other programming ones look fine, bad luck I saw this one first I guess.
I think looking at some more niche benchmarks helps, I created one and it does show Kimi K3.1 Pro as the most intelligent of all open-weight models, though it is still significantly below Opus 5/5.5[1]
1 - https://bench.killswitch-lang.org/
If you're in a stable codebase with lots of best practices, patterns, controls (linters, analyzers, etc.), examples already established, then any model will do ~the same plus/minus since there's not much novelty.
If you're building ground up, entirely novel, no reference, I think results will vary with lower capability models.
For me, I do a mix, leaving implementation to Haiku/Luna/DeepSeek and research/feedback/critique to Opus/Sol/Astra/Fable
I keep having problems with them not doing the thing I ask, (and also failing to call tools correctly in pi. I think they've very much trained to the harness, which is frustrating.).
The harness I've built for it is pretty straight forward: split binary into individual functions, give the agent a bunch of small tools to diff output with underlying binary, make them wrote clean c until there's a certified match.
Rin a final oracle to make sure the agent isn't cheating.
- Qwen 3.8 27b on my 5090, barely acceptable, thinks too long, sometimes freaks out.
- deepseek flash variants, work horse. Runinfra provides around 300 tok/s. Great for tearing through moderately complex functions
- glm and kimi k.3, expensive, slow, ultimately not that much better than deepseek flash for this task. Better at doing long context research into old compiler productions
- sol 6.1, best bang for buck. Creative enough to methodically work through more complex spellings
But I found there were around 1% of functions that these models simply cannot solve. Opus and Astra take them handily.
I'm still still in the process of looking through the solution traces, but from watching the missing sauce seems to be creative reasoning.
Sol 6.1 will just happily grind on permutations of a base shape spelling that M2C outputs, where opus and Astra immediately realize that they need to unwind assumptions taken for granted (and sometimes in the process endure a worst match).
I'm hoping to be able to capture that creativity step and distill it into the harness tools.
I don't have much experience with anything outside of Claude for frontier models (and only haiku/3.x flash/5.6+6 luna at the entry level) so take this with that in mind.
I find Fable is still SIGNIFICANTLY 'smarter' then opus 5.5 when figuring out HOW to do something that's non-mechanical (e.g. How should we handle this aspect of business logic?"). Opus comes up with perfectly usable options, however Fable more often comes up with options that are more aligned with what I expect, or accept.