> Together with this week’s launch of Muse Image, this release brings us closer to our vision of personal superintelligence: models that help you pursue your goals, create what you imagine, deepen your relationships, and take action on what you value most.
I was laughing along and then realised they actually meant that
Very strong pricing, cheaper than Grok 4.5, particularly the cached reads. We'll have to wait to see if it's actually worth using (it's not on OpenRouter yet).
This is still ridiculously expensive imagine having to pay $10 for 100 search results on Google, thats essentially what this is.
I really dont see how anyone's willing spend more than $1.50 per mm output. Let alone $15-50. Does anyone actually pay for usage based billing as a consumer?
Sometimes. It depends on the task. $15/Mtok is still a lot cheaper than human written code. It’s probably worth spending more for contract reviews. Tasks don’t have uniform value. If you’re using an AI for entertainment, then frontier premiums are hard to justify. For paid work, they’re a great deal if used reasonably.
Yeah, this is most directly comparable to xAI Grok 4.5. In both cases, directionally "opus level intelligence for haiku prices" which is a really big deal for application developers who want to include models like this in their applications. I have been testing switching out haiku and sonnet for Grok 4.5, and may give this a try too (it is quite a bit cheaper, particularly for cached).
> Yeah, this is most directly comparable to xAI Grok 4.5.
Grok 4.5 has a relatively high $0.50 per 1M cached input token rate, compared to $0.15 on this model.
Grok 4.5 cached input costs the same as Opus 4.8 cached input, which is going to make it a lot more expensive to use for multi-turn coding than many would assume from the $2/$6 headline numbers they led with.
> ... make it a lot more expensive to use for multi-turn coding than many would assume from the $2/$6 headline numbers they led with.
There's a further sting in the tail, Grok 4.5 is only $2/$6 for the first 200k of context. Go above that, and the pricing is $6 / $12 - and you're still capped at only 500k context anyway.
Yeah, I think it is definitely great. Having said that, I am still debating in my mind whether the volume of software engineers needed in the AI era is going to increase or decrease because of all of these advancements.
On the one hand, because it is easy to build products, more and more people will build. And more and more products and features will be built. However, a lot of people who are non-technical will also try to build, but they get stuck, and then they will need engineers. The sheer volume of product built by both experienced technical companies and non-technical novice startups and founders and wannabe founders is going to be massive. That is the bull case for having more software engineers needed in the near future.
On the other hand, in a year or so, people will build all these products, and most of them won't be able to market them, sell them and make money. Eventually, there won't really be a need for that many software engineers.
I think overall the bull case is probably going to win net net.
The big thing to me is why are we even running these models on top of an operating system?
What I really want is Claude as a deep part of the operating system.
If that happens then a whole lot of the abstraction of software vanishes along with what we think of today as software jobs. I think many new forms of knowledge work would emerge from this though.
I would think that needs massive local compute but I can't imagine that is not the future down the line.
It’s also not the future SV is incentivized to build. They want everything for rent, nothing can be owned.
Luckily, China is on the verge of a true breakout, I’m not sure what exactly it will be - but I’d make a very large wager the “next iPhone” is Chinese, and will constitute a full blown “Sputnik moment” for the US and SV.
If Americans weren’t forbidden to own Chinese EVs they’d know this. But tariffs mean the breakthrough will be even more unexpected.
Since Chinese actually “sell stuff” I’m guessing their unbeatable lead in AI efficiency, manufacturing, and distribution will produce a step change breakthrough within a decade.
I think LLMs/coding agents in particular are going to play out like automation always does.
1. You have a task and it requires an expert to perform it (software engineering, where we were)
2. You automate the task, it still requires the expert to babysit it (where we are now)
3. Management works out that the expert is just monitoring what the automation is doing and has actually lost expertise because they are just watching, not doing. Pay collapses. (where we are going)
I use it all the time through Fireworks. The normal version when I pay it myself and the fast one when company pays. It's really fast and I never get rate limited with my daily use.
> No wonder we still can’t get climate change under control
This is was historically a money issue, being green used to be wildly more expensive.
Now being green is cheaper, the limiting factor is how fast PV and batteries can be made or imported.
Recent reports of the sum of all US data centres currently in planning, has a power demand exceeding the (capacity-factor-adjusted!) global annual supply of PV.
This would be less of a problem if Trump wasn't trying to get in the way of anything green, or if the companies building data centres decided to also support factories to make more PV.
* Planned new demand: 300 GW; PV factory capacity ~ 600 GW nameplate, but the capacity factor is 14% so that's really 84 GW on average.
He came to X to post about this instead of his very own meta threads. This just shows how much interested he is to make this thing big, and of course, the cost can stay bearable for us considering all of these cash burn that these companies are doing
Yes and Zuck effectively disbanded the entire team that did that. Not saying we shouldn't cast a critical eye on it, but it probably does warrant a second chance.
This sounds... kind of useless? Really good JSON or similar constrained decoder performance is interesting, but normal decoder > tool validator loop with good error message > tool retry is almost always able to get a tool to work second try, and input is cached so it's not expensive.
Debugging and diagnosis is very tool call heavy, whether that's grepping / transforming logs, calling out to profilers/tracers, or even just writing up incident reports.
Bug diagnostics is about being okay at coding but better at tooling.
Given a good diagnostic report, it can be handed to opus for the fix.
Opus is okay at writing reports, but it still regularly gets table widths wrong in typst documents, leaving the last column full of text but only a handful of characters wide.
I wonder if we'll start to see that pattern with every new release. Tool use likely changes rapidly, so the newest, rather than most intelligent, model may always have an edge.
What you mean.. The tools are all just invoking bash and terminal/cli cmds and http requests. Paradigms that have existed and stayed mostly unchanged for decades.
It's a high quality benchmark for sure, but it being public means it's at risk of leaking into the models (unintentionally or not), right? For that reason I prefer to look at the private ones, like: HLE, SimpleBench, Kagi, ARC-AGI.
Exactly. I have zero trust in them, especially given their history of shady practices - like buying a VPN service (Onavo) and using it to track how people used competing apps.
Competition for cheaper and efficient models is a good thing, regardless of if you don't like SpaceX, Meta, etc. Especially from US based labs
I for one am really glad to get competitive models that will push the major labs to bring prices down. While Chinese open source labs are also great, unfortunately when it comes to US/Western political pressure it won't often have as much of a bearing on labs bringing prices down, especially for enterprises.
Also if these numbers are true, this is truly breaking ground finally for Meta.
I missed the fact that Meta was developing and releasing closed-weights models... bummer. Would be great to see some more progress with American open-weights models.
Everyone has been loving to shit on the Alexander Wang acquisition but this seems legitimately impressive to me?
Meta's AI org when from a total mismanaged dumpster fire for multiple years to delivering a competitive model in less than a year on essentially their first try?
> We use a bash-tool-only agent harness to evaluate 89 Terminal-Bench 2.1 tasks from the official repository, where resources are capped at 6 CPU cores and 8GB RAM.
This disqualifies the results. Each terminal bench task has a cpu upper limit and RAM upper limit. Overriding either is disqualification.
For reference, in tbench-2.1,
1. 0 out of 89 task allow 6 cpu cores (highest is 4, and i think only 1 task)
2. 8 out of 89 tasks allow 8GB RAM
This kind of shady benchmarking (I was talking about it just yesterday in a different context https://news.ycombinator.com/item?id=48838212) takes all joy out of building a harness to improve benchmark performance of a model because no matter what you do, you won't beat the headline (cheating) number. This is presumably why this model is not in the official benchmark leaderboard https://www.tbench.ai/leaderboard/terminal-bench/2.1
As an ex Meta employee, this is a little sad but not massively surprising. 'Number go up' is the core performance evaluation metric until PSC is done and you move on.
Why are resource limits considered at all aside from models accidentally fork bombing themselves?
I thought the benchmark was supposed to be about terminal use and specifically chaining together lots of bash tool calls. Which test cases does this matter for?
Terminal bench 2 isn't simply about 'somehow' getting a task done, it intends to measure real world behavior of an agent, including environment awareness in a given situation.
A few examples from memory:
1. This task [1] asks the agent to train a CNN under 1 CPU, 2GB RAM, 10GB storage. If you allow high resources, weaker models often succeed (the most clock time actually goes in waiting for the network to train).
2. This task [2] asks agents to implement a complete MIPS interpreter in JavaScript in 1 cpu and 2GB RAM. A common failure mode is OOM, at least in the earlier buggy versions that models run to get feedback. When OOM hits, the task is killed, no do-overs.
3. A lot of tasks involve building projects with a single core supplied. If you use -j12 type options, it will actually be _slower_ to build and the task will more likely miss the timeout. Having more threads squeezes the end to end time. This is a big one actually since the most common failure mode (from what I have seen) is the task timeout hitting before the agent finishes
What that link describes is basically the motivation to go from terminal bench 2.0 to 2.1. The latter simply fixed the common issues/complaints. There is a long github discussion on tbench's about it
Yes but my point is
- Resource limits are a "recommendation" and are not strictly enforced
- Significantly boosting resources up to 3 did not statistically shift performance results
Sure for old tasks you could argue that now its not required to boost because infra errors are alleviated with better default limits. My point more so is that its a strange thing to index on because if you wanted to cheat on the benchmark, it does not particularly seem like something that shifts results? Once the API is out maybe I'll eat my words, but I don't really believe that if you manually tried to reproduce the results with lower limits you'd see significantly different results
This doesn't seem that big of a deal to me? I mean, in any other area where I want an assessment of a product, I'm not going to trust what the product producer says about it at face value -- obviously they're going to be biased. This is the whole raison d'etre for independent testing, like https://artificialanalysis.ai.
I get your point but I'm not sure it matters all that much.
Did harbor / tb2.1 cap the swap available to docker runs?
There used to be a bug that would allow dockerized instance runs to use more memory than the specs allowed. Some of the original tasks weren't really possible to complete without exploiting swap. Even the oracle solutions didn't pass if you stopped docker from having access to swap.
I think crack-7z-hash and filter-js-from-html had that problem off the top of my head, but i haven't looked at this in months, so i'm not sure
Thats what is wrong with close source models, we dont know exactly what we are paying for, a superior base model or a well thought harness for benchmaxing
It’s just inherently a difficult problem to solve, I think (just like with human individual contributors, where there famously also doesn’t exist one universal, automated process).
“Good” benchmarks to gauge development skills at the moment seem to be:
DeepSWE [0] by Datacurve
FrontierCode [1] by Cognition
And then there’s TerminalBench, which I’m certain has been saturated in post-training to no end, so I wouldn’t think of it as a gold standard anymore.
But yeah, in general, it’s not going to get easier knowing which benchmarks are actually measuring “frontier” capability, and which are just getting results inflated by way of time/token budget [2], ever again.
Good to see Meta finally back to releasing something at least worth evaluating. And it sounds like they did at least a bit skate to where the puck is going by focusing on tool and computer use.
Interesting that neither meta nor xai chose to do open source given that they are both clearly behind Google, OpenAI and anthropic - and a serious us open source offering would give them a clear foothold.
I suspect they have a brand problem from their social media ties and shady histories. I personally will never use their models, plenty of better alternatives. I'm now exclusively on open weight models
185 comments
[ 0.87 ms ] story [ 99.4 ms ] threadI was laughing along and then realised they actually meant that
https://dev.meta.ai/docs/getting-started/pricing-rate-limits
If they have a really good model, it makes sense to subsidise it, to gain users, before they align prices with competitors.
I really dont see how anyone's willing spend more than $1.50 per mm output. Let alone $15-50. Does anyone actually pay for usage based billing as a consumer?
https://platform.claude.com/docs/en/about-claude/pricing
Model Base Input Tokens 5m Cache Writes 1h Cache Writes Cache Hits & Refreshes Output Tokens
Claude Fable 5 $10 / MTok $12.50 / MTok $20 / MTok $1 / MTok $50 / MTok
Claude Opus 4.8 $5 / MTok $6.25 / MTok $10 / MTok $0.50 / MTok $25 / MTok
Note Fable costs $50 MTok and Opus 4.8 costs $25 / MTok.
Even with usage based billing I'm below $1 writing code all day.
same, pi as a harness works best, better than claude code and open code.
Grok 4.5 has a relatively high $0.50 per 1M cached input token rate, compared to $0.15 on this model.
Grok 4.5 cached input costs the same as Opus 4.8 cached input, which is going to make it a lot more expensive to use for multi-turn coding than many would assume from the $2/$6 headline numbers they led with.
There's a further sting in the tail, Grok 4.5 is only $2/$6 for the first 200k of context. Go above that, and the pricing is $6 / $12 - and you're still capped at only 500k context anyway.
Here's the xAI pricing on OpenRouter:
https://openrouter.ai/x-ai/grok-4.5?endpoint=0e927811-b1a8-4...
Compare with Grok 4.5 which came out at $2/$6 but then quietly charges $0.50 per 1M cached input tokens. That's as high as Opus 4.8!
- Chinese models
- Grok
- Meta
- Google
- OpenAI
- Anthropic
I think this is a win. I'm building like crazy to take advantage of all these subsidized tokens while I can.
On the one hand, because it is easy to build products, more and more people will build. And more and more products and features will be built. However, a lot of people who are non-technical will also try to build, but they get stuck, and then they will need engineers. The sheer volume of product built by both experienced technical companies and non-technical novice startups and founders and wannabe founders is going to be massive. That is the bull case for having more software engineers needed in the near future.
On the other hand, in a year or so, people will build all these products, and most of them won't be able to market them, sell them and make money. Eventually, there won't really be a need for that many software engineers.
I think overall the bull case is probably going to win net net.
What I really want is Claude as a deep part of the operating system.
If that happens then a whole lot of the abstraction of software vanishes along with what we think of today as software jobs. I think many new forms of knowledge work would emerge from this though.
I would think that needs massive local compute but I can't imagine that is not the future down the line.
Luckily, China is on the verge of a true breakout, I’m not sure what exactly it will be - but I’d make a very large wager the “next iPhone” is Chinese, and will constitute a full blown “Sputnik moment” for the US and SV.
If Americans weren’t forbidden to own Chinese EVs they’d know this. But tariffs mean the breakthrough will be even more unexpected.
Since Chinese actually “sell stuff” I’m guessing their unbeatable lead in AI efficiency, manufacturing, and distribution will produce a step change breakthrough within a decade.
1. You have a task and it requires an expert to perform it (software engineering, where we were) 2. You automate the task, it still requires the expert to babysit it (where we are now) 3. Management works out that the expert is just monitoring what the automation is doing and has actually lost expertise because they are just watching, not doing. Pay collapses. (where we are going)
Source: https://en.wikipedia.org/wiki/Ironies_of_Automation
- DeepSeek
- GLM (Z.ai)
- Minimax
- Kimi (Moonshot)
- Hy3 (Tencent)
- Qwen (Alibaba)
(Each one of these with weights available to download and run locally)
No wonder we still can’t get climate change under control
This is was historically a money issue, being green used to be wildly more expensive.
Now being green is cheaper, the limiting factor is how fast PV and batteries can be made or imported.
Recent reports of the sum of all US data centres currently in planning, has a power demand exceeding the (capacity-factor-adjusted!) global annual supply of PV.
This would be less of a problem if Trump wasn't trying to get in the way of anything green, or if the companies building data centres decided to also support factories to make more PV.
* Planned new demand: 300 GW; PV factory capacity ~ 600 GW nameplate, but the capacity factor is 14% so that's really 84 GW on average.
I do not know if competition is good, we will see in a few years.
Looking forward having a physical job for a change :D
Let's see how it does on the Creative Writing bench ;)
What kind of use case would be best for that shape?
Bug diagnostics is about being okay at coding but better at tooling.
Given a good diagnostic report, it can be handed to opus for the fix.
Opus is okay at writing reports, but it still regularly gets table widths wrong in typst documents, leaving the last column full of text but only a handful of characters wide.
I see models fail on tool calls that involve API requests to a specific API, internal or cloned Makefile calls, npm run commands, etc.
I for one am really glad to get competitive models that will push the major labs to bring prices down. While Chinese open source labs are also great, unfortunately when it comes to US/Western political pressure it won't often have as much of a bearing on labs bringing prices down, especially for enterprises.
Also if these numbers are true, this is truly breaking ground finally for Meta.
what advantage does this give them? is it really that hard to add github or google login options there?
Meta's AI org when from a total mismanaged dumpster fire for multiple years to delivering a competitive model in less than a year on essentially their first try?
From Terminal-bench-2.1 details,
> We use a bash-tool-only agent harness to evaluate 89 Terminal-Bench 2.1 tasks from the official repository, where resources are capped at 6 CPU cores and 8GB RAM.
This disqualifies the results. Each terminal bench task has a cpu upper limit and RAM upper limit. Overriding either is disqualification.
For reference, in tbench-2.1,
1. 0 out of 89 task allow 6 cpu cores (highest is 4, and i think only 1 task)
2. 8 out of 89 tasks allow 8GB RAM
This kind of shady benchmarking (I was talking about it just yesterday in a different context https://news.ycombinator.com/item?id=48838212) takes all joy out of building a harness to improve benchmark performance of a model because no matter what you do, you won't beat the headline (cheating) number. This is presumably why this model is not in the official benchmark leaderboard https://www.tbench.ai/leaderboard/terminal-bench/2.1
As an ex Meta employee, this is a little sad but not massively surprising. 'Number go up' is the core performance evaluation metric until PSC is done and you move on.
I thought the benchmark was supposed to be about terminal use and specifically chaining together lots of bash tool calls. Which test cases does this matter for?
A few examples from memory:
1. This task [1] asks the agent to train a CNN under 1 CPU, 2GB RAM, 10GB storage. If you allow high resources, weaker models often succeed (the most clock time actually goes in waiting for the network to train).
2. This task [2] asks agents to implement a complete MIPS interpreter in JavaScript in 1 cpu and 2GB RAM. A common failure mode is OOM, at least in the earlier buggy versions that models run to get feedback. When OOM hits, the task is killed, no do-overs.
3. A lot of tasks involve building projects with a single core supplied. If you use -j12 type options, it will actually be _slower_ to build and the task will more likely miss the timeout. Having more threads squeezes the end to end time. This is a big one actually since the most common failure mode (from what I have seen) is the task timeout hitting before the agent finishes
[1] https://github.com/harbor-framework/terminal-bench-2-1/blob/...
[2] https://github.com/harbor-framework/terminal-bench-2-1/tree/...
The test cases of "don't melt my computer" and "be a good (computational) neighbor"
https://www.anthropic.com/engineering/infrastructure-noise
Is anthropic benchmark maxxing and cheating on terminal bench too? They don't follow the strict resource "limits" either
Sure for old tasks you could argue that now its not required to boost because infra errors are alleviated with better default limits. My point more so is that its a strange thing to index on because if you wanted to cheat on the benchmark, it does not particularly seem like something that shifts results? Once the API is out maybe I'll eat my words, but I don't really believe that if you manually tried to reproduce the results with lower limits you'd see significantly different results
Did harbor / tb2.1 cap the swap available to docker runs?
There used to be a bug that would allow dockerized instance runs to use more memory than the specs allowed. Some of the original tasks weren't really possible to complete without exploiting swap. Even the oracle solutions didn't pass if you stopped docker from having access to swap.
I think crack-7z-hash and filter-js-from-html had that problem off the top of my head, but i haven't looked at this in months, so i'm not sure
To be fair, seems more correct to compare against similar strength models if your main edge is pricing.
Second, compare to older versions of competitor s models.
Still does not look good? Compare to own previous models.
I was a heavy codex and CC user but these days I use whatever is this weeks top open model (via API, maybe that will change to local in 1-2 years)
“Good” benchmarks to gauge development skills at the moment seem to be:
DeepSWE [0] by Datacurve
FrontierCode [1] by Cognition
And then there’s TerminalBench, which I’m certain has been saturated in post-training to no end, so I wouldn’t think of it as a gold standard anymore.
But yeah, in general, it’s not going to get easier knowing which benchmarks are actually measuring “frontier” capability, and which are just getting results inflated by way of time/token budget [2], ever again.
[0] https://deepswe.datacurve.ai
[1] https://cognition.com/blog/frontier-code
[2] https://xcancel.com/i/article/2064210146558136827