Related to this, for our use case, setting thinking to high instead of low made tasks complete faster and cheaper (Gemini 3.0 flash).
Other aspects are caching, often at 0.1X cost, where providers really differ in how efficient they are (Anthropic really good, Google not so much) and how chatty a model is (costing output tokens).
Well yeah this benchmark I've quoted is quite large, I believe they use thousands of diverse tasks, so this average cost per task could be a more accurate representation of how expensive each model actually is to run.
I think that the future is something middleware, I'm working in a hibrid app in order to reduce the tokens consumption of our company, and It works with on premise computing from our datacenter, and laptops from our community, maybe it could fit for you, check the code if you are interested https://github.com/vfalbor/hibrid
Price per token is meaningless for more reasons than this, because all of the provider monthly subscriptions price tokens _extremely_ differently than their per-token billing rates. It's stupid to look only at what you get when paying more than you need to for a given service.
Yes that's true, although calculating the exact discount you get using a subscription could be difficult, especially since the labs often don't state their actual limits and just tell you you get "more usage" or "even more usage" or "5x the usage" (but how much exactly is x?).
It is clear we are getting a very meaningful discount by using a subscription though. I once checked how much would I pay for Codex after about 3 months of non-daily use if I only bought "credits". Couldn't believe it when it came out to be north of $1000 (I paid them $20/month in that period).
Cost per token doesn't say a lot, but "Cost per benchmark task" is also meaningless if your task is difficult enough that the cheaper model has no chance of cracking it.
My advice to any CEO / individual - throw your hands in the air and bring it in-house. Yeah the performance can dip depending on what GPUs you can salvage these days but the uncertainty over price is almost nothing compared to the uncertainty over the effective use of AI. It’s not just coding (do I go partly agentic or all out Steve Yegge). This is all over the enterprise - do we parse every email, rewrite PowerPoints? Or just stop using PowerPoints at all. Do we throw LLMs at the mess of wikis and word docs, do we pretend that the policies no-one has ever read actually are how the LLM should think or is it how the work actually gets done - barely documented
The uncertainty of how to use this vastly vastly outweighs the price in a data centre - so buckle up, buy enoughbGPUs to experiment at a known cost and one day you will find the approach that gives you 10x returns - at that point pay any price per token but not till then
Concrete example: I’ve been trying to use Claude to generate all my commit messages, but it takes 5-10x longer than if I just write them myself. Mine are less detailed, but one line changes are sometimes inconsequential (especially white space reformatting). I wish there was a model that understood the codebase well enough to generate commit messages in half the time.
Pricing based on tokens always seemed a little weird to me.“Tokens” was and still is an engineering concept. The fundamental unit of transformer encoding and decoding.
But I have a sinking feeling that many AI developers think “tokens” got their name from the same idea as “virtual tokens in a casino” which is more related to product pricing and business.
Tokens in a casino is pretty accurate if you think about it. You never really know what you'll get so it's tempting to "roll" over and over, thinking every roll puts you closer to a bellringer. It can even get addicting for some people.
Tokens do reflect the provider's cost though - each token output required them to execute the model once, normally incurring a fixed amount of compute per token.
Apparently hardware depreciation is the dominant cost rather that operational cost (electricity etc), and this is occuring at a fixed rate per the planned replacement lifetime.
So, the cost to provide the service is essentially fixed regardless of load, but the revenue they are generating is variable. In practice most GPU's are going to be capacity-maxxed since the providers sell cheap batch APIs that they queue to keep the machine loaded. They'd be losing money over a given time interval if revenue generated during that interval wasn't greater than depreciation (etc), but it seems that will rarely happen.
The Sonnet 5 comment is spot on. Even Anthropic's own graph initially showed lower performance at higher costs. Only thing I notice about Sonnet 5 is that it does appear to hand off tasks to agents more frequently similar to Fable, but of course nowhere near the quality of Fable. My guess is that Opus 5 will do similar but just isn't ready yet.
Sonnet 5 is a huge regression and many times it performs worst than deepseek. I believe Antrophic staff themself don't use Sonnet and use Fable for everything.
Given the capability of fable and the shockingly repetitive silly mistakes they made when publishing/updating something, I am starting to wondering whether Anthrophic can afford Fable for everything themselves.
tool use is another factor, every time the agent uses a tool the entire context is priced at cache rate on top. the same happens when it asks you for input.
I keep trying to convince directors and executives at my company to look past the cost per token amount but they refuse to do so. Those are the only things that actually give any sort of measurement of the monetary value of a token by these labs, and so its what many go by.
For example there's some benchmarks that show that Opus for any task that requires a higher than `high` level of effort, may have actually been cheaper to use Fable on low even though the cost per token is drastically higher
Similarly with GPT 5.5 vs Opus. They simply look at the dollar amounts the labs assign to each model and run with it.
But part of the issue compounds on the fact that there are many people who simply default to the smartest model/effort and don't actually vary their model per task. So in some sense I don't actually blame them very much.
In the context of local LLMs on limited hardware I've ran to the exact same conclusion: "tok/s" isn't the most useful metric when my personal North star metric, given my fixed hardware is: Model smart enough to execute my goals _in the minimum amount of time_.
Some models I tried (Mistral I think) had better tok/s, and roughly same billion parameters / scores on various benchmark... But they were _so_ verbose, that they generated many more tokens compared to a Qwen model of same caliber to answer the same thing.
So even though it had better generated tok/s, because so many more were generated, the clock time was longer.
And this compounds over mutli-turns: more generated token means more context used in the next turn (until some compaction or something runs)
Even more important in a local context is the difference between token generation and prompt processing speed. We tend to focus on the former, but for multi-turn/agentic workflows the latter can dominate.
I feel like we need to see more proliferation of local LLMs to start seeing ones turned to be terse, rather than maxing the amount of tokens user pays for
Yup. I’ve been evaluating several on openrouter and find token cost meaningless for my work. I haven’t found a great alternative, though the “cost per task” he uses makes some sense.
An LLM is an extremely complex thing used for all manner of purposes. The hope that there would be some simple pricing construct that would map nicely to value provided is a pipe dream.
Pricing per token is at least reasonably straight forward. If you aren't getting value, you don't use the service. One doesn't buy a Ferrari and then complain that in their town Ferrari doesn't help them pick up women and hence it should cost less.
I'd say it's more like going to a Ferrari dealership and they tell you they will build a car for you and bill you per gram of parts used. It might also not work. And they might also not build it ever, really — but they will bill you for any attempts to build it.
That's fine though. If that doesn't work for you, don't buy. There are all manner of situations where what one wants or needs and what they get don't match up well. You don't price out every situation - it's take it or leave it. Pricing in a way that is somehow based on cost structure at least enables the provider to work to reduce the cost and hence price and win. Costco prices at a small margin above cost, they don't price "if this meets value prop X, pay Y and if only value prop A, pay B".
Another important benchmark would be — cost per benchmark task using subscription tokens. Since most of us are using subscriptions and cost per token there is quite different from API costs.
Cost per tokens is as valid as price per unit volume of fuel.
Changing the fuel type, efficiency of your vehicle, driving distance, or driving conditions will all change how much it will cost you.
Fuel cost per unit volume does not become meaningless just because you are neglecting all of the other factors involved. That would be throwing away the only data point you have been using.
This is just asking for someone to amalgamate all of the factors involved into one simple, easy to game, index.
Except, a gallon is a gallon no matter which gas station I'm at. Also I know my car's gas mileage, and it doesn't change when I visit a Shell station instead of a Chevron. The composition of the gas is regulated, as are the pumps that dispense it. There are inspectors from the state whose job it is to ensure that when I buy a gallon, I really get a gallon.
Tokenizers aren't standardized to anywhere near that level. A "token" from one isn't the same as a token from another.
That’s not a good analogy because a gallon of gasoline has a known amount of energy in it. The efficiency of each vehicle is also known, at least in a way that is easy to compare on a relative basis.
I can go to 10 different gas stations and buy the same amount of energy from them. When I put it in my car I’m going to get the same result out. The differences are very small.
I feel we are caught in a "this is fine, pay more and we may turn down the fire" situation.
The LLM itself produces one token. Some tool adds that token to the input and runs it again, flogging the horse. Downstream another tool, some kind of harness, tries to control this stream by injecting tokens into the context and then sending it to the inference tool, and then trying to pattern-match the output.
Finally, there you are on CodePorn.yata paying for an agent to generate code, paying for an agent to tell you what's wrong with it, and paying for an agent to make it differently bad, and hopefully move on to the next task.
If it still hasn't dawned on you that this isn't just a bubble, but a snake-oil-bubble-bath, just try to imagine the paradigm shift whereby you go on github.com, assign an issue to an agent, the agent fixes it by rewriting the application in Pascal but a reviewing agent catches that you wanted it to print a measurement in Pascals (pa), and you don't pay for the work or the review, you only pay for work that one or two reviewing agents determine is up to par.
Nobody is going to do that because as soon as they test it they're going to have to do some math that won't make sense without admitting/realizing it's not some near-sentient, AGI rating 0.9 intelligence, it's just a text prediction algorithm that can pull out entire sentences when you use it to infer output on topics it trained on.
Maybe, but I think the downplay is worth it to emphasize that the things have to be done. The point is precisely that token generation is a phenomenal power source but it does not give you anything but debt unless you build the tools to leverage it.
None of the big players involved right now seem to be doing anything to disarm the perception that the LLM does all the work.
I feel like we're at a state fair 180 years ago having just seen an impressive demonstration of small animals momentarily coming back to life, someone in the crowd said 'eternal life' and everyone lost their minds trying to give money to the 'inventor' on the stage who applied the electric current.
Ask some lay people or junior devs how they feel about LLM mistakes, listen for answers about "learning" etc. Do they realize the model doesn't change? The same set of weights and tensors are deployed to every GPU hosting it. If you use a power tool that lets you specify the temperature and seed, it will generate the exact same output time after time.
"But I told it to always speak like a pirate, and it does": No, you signed up to pay for the model to be told to speak like a pirate every time you use it.
There - that's the curtain I'm talking about that you're lifting/tearing. I think momentarily downplaying the potential of LLMs themselves is worth it to expose that.
I suspect the next real advance will be the LLM equivalent of thin-client/fat-server evolution, a sort of "local lora" system that forms the unique part of your own agent(s) distinct from the underlying static model, and capable of dynamic learning/processing.
Could it be cloud-based? That changes the risk and scale calculus in a way that's going to take a long time to get funding into.
>Careful you don't "it's just a text predictor" yourself into unemployment .
Says the guy who is "I-can-now-just-push-buttons-and-get-paid-forever"-ing into not only into unemployment, but also being totally redundant.
Before you come back with, "Oh I review everything LLM does": Only in your dreams you ll gain or retain experience by just reviewing stuff. That is even if you actually review every line that the LLM writes...which have literally zero chance of happening. People have hard time keeping their eye open even when their life is on the line when using a pseudo self driving car...
I do have an entirely AI written project, and I don't think I've reviewed even 1% of the code the bots have generated; it's been my pure vibe project. It's about what you'd expect, too.
Elsewhere, I still haven't reviewed the vast majority of AI code I've generated, but all of the AI-generated code I've submitted I did review exhaustively. I have a couple of past roles to thank for drilling some strong code-review strategies and - thus far - have had the willpower to reject an AI generation when I know I can't/won't diligently review.
LLMs do not do anything but generate a token. It's a trivial but critical distinction like RNG vs pRNG. The LLM doesn't learn, it doesn't do backtracking, their output is deterministic.
Not "chatgpt.com"s output, but the LLM powering it; Anthropic aren't customizing a version of Claude's weights/tensors on your inputs, they're adding billable input tokens to the baseline of your subsequent contexts.
Its very financially convenient for Anthropic and OpenAI if people think of the LLM as doing the work, because for a lot of people that conjures a sense of a system that inherently learns and improves.
Even some of the people I've worked with on low-level LLM harnessing tools/systems since early 2023 fall for it and start thinking of the LLM as an AI with the connotations of back-propagation, weight adjustment, ... learning.
The "AI" is in what the harness software does with the inference output and the context it feeds back to it.
It took forever to convince people that quality degrades with context length but then all we got was compaction, there's still huge resistance to harnesses actively curating the context.
A lot of non-devs I've spoken to assumed when their tool said "Now I need to re-read" it was overwriting the 'in memory' copy.
You open aider or code or claude and ask it to fix a bug in file.<c|cpp|js|py>, it reads the entire 8k file, reasons about it (in which it sometimes echoes specific lines of code), and then it "edits the file".
To edit a file, inference currently injects a stream of tokens into the context that your software identifies as a tool call.
And frequently the tooling will guide the model to re-read the file, in some pathological cases by prompting the model to generate a tool instruction to do it.
Your context now contains: pre-prompting, your prompt, first file read (8k tokens), reasoning, all the tool calls to make the changes, re-read (8k tokens), conclusory reasoning, tooling prompts llm to describe what has been done, work-done-summary.
Because of the way the underlying LLM works and because of the way the APIs are presented, tools are discouraged from redacting or eliding the original read.
Otherwise, what they'd already be doing is curating the context: read file once, append output from inference/tool calls, and when it's done, replace the original insert of the file.
Having 2 or more disagreeing versions of a significant source file in your context is detrimental to model attention and output quality, and you either see people who get it and manage it, or who wallow in superstition and hand-waving and a near eagerness to pay for "mistakes" that the "model is making" because they believe it is learning, on the fly.
The only metric that really matters is 'profit per amount invested'. This is very difficult to quickly evaluate, and therefore we resolve to use simplified metrics such as cost per unit tokens.
The point at which the metrics become meaningless is when others become aware of them, and begin to optimise for them. Lines per code is is not a bad insight for development activity, only when the developers are not aware of the metric. Price per 1M tokens became meaningless when LLM providers started to optimise for it. It seems to be that Sonnet 5 is optimised to score well on AA intelligence whilst seemingly having a lower price per 1M tokens.
I think generally we are in an AI bubble, and it will at some point pop. The numbers simply don't make sense. I would gamble heavily on local cost per task to survive the LLM winter. Given that hardware is pretty much a fixed overhead, you probably want to optimise for task per kW - that's where I'm betting.
74 comments
[ 2.6 ms ] story [ 40.5 ms ] threadOther aspects are caching, often at 0.1X cost, where providers really differ in how efficient they are (Anthropic really good, Google not so much) and how chatty a model is (costing output tokens).
i've always wanted cost per prompt, but even that has too much variation.
It is clear we are getting a very meaningful discount by using a subscription though. I once checked how much would I pay for Codex after about 3 months of non-daily use if I only bought "credits". Couldn't believe it when it came out to be north of $1000 (I paid them $20/month in that period).
Yeah. My $200/month subscription would easily cost $5000/month or more if I were paying per token.
Why can't you derive this discount from experiments? Example of such research
https://she-llac.com/claude-limits
The uncertainty of how to use this vastly vastly outweighs the price in a data centre - so buckle up, buy enoughbGPUs to experiment at a known cost and one day you will find the approach that gives you 10x returns - at that point pay any price per token but not till then
We've started trying to do some comparison videos to capture more of the UX vs speed vs cost stuff e.g. https://www.linkedin.com/feed/update/urn:li:activity:7479891... which one of my team did for my LinkedIn account (disclaimer: marketing)
(In this particular case Deepseek was way slower than GPT 5.5 but I think that's because it installed Libreoffice half-way through the task!)
But I have a sinking feeling that many AI developers think “tokens” got their name from the same idea as “virtual tokens in a casino” which is more related to product pricing and business.
If yours is the only request in the batch it will cost them one full pass through the model.
If yours is one of 1024 inputs in the batch the per token cost is 1024x less.
So, the cost to provide the service is essentially fixed regardless of load, but the revenue they are generating is variable. In practice most GPU's are going to be capacity-maxxed since the providers sell cheap batch APIs that they queue to keep the machine loaded. They'd be losing money over a given time interval if revenue generated during that interval wasn't greater than depreciation (etc), but it seems that will rarely happen.
For example there's some benchmarks that show that Opus for any task that requires a higher than `high` level of effort, may have actually been cheaper to use Fable on low even though the cost per token is drastically higher
Similarly with GPT 5.5 vs Opus. They simply look at the dollar amounts the labs assign to each model and run with it.
But part of the issue compounds on the fact that there are many people who simply default to the smartest model/effort and don't actually vary their model per task. So in some sense I don't actually blame them very much.
Some models I tried (Mistral I think) had better tok/s, and roughly same billion parameters / scores on various benchmark... But they were _so_ verbose, that they generated many more tokens compared to a Qwen model of same caliber to answer the same thing.
So even though it had better generated tok/s, because so many more were generated, the clock time was longer.
And this compounds over mutli-turns: more generated token means more context used in the next turn (until some compaction or something runs)
Pricing per token is at least reasonably straight forward. If you aren't getting value, you don't use the service. One doesn't buy a Ferrari and then complain that in their town Ferrari doesn't help them pick up women and hence it should cost less.
Changing the fuel type, efficiency of your vehicle, driving distance, or driving conditions will all change how much it will cost you.
Fuel cost per unit volume does not become meaningless just because you are neglecting all of the other factors involved. That would be throwing away the only data point you have been using.
This is just asking for someone to amalgamate all of the factors involved into one simple, easy to game, index.
Tokenizers aren't standardized to anywhere near that level. A "token" from one isn't the same as a token from another.
I can go to 10 different gas stations and buy the same amount of energy from them. When I put it in my car I’m going to get the same result out. The differences are very small.
The LLM itself produces one token. Some tool adds that token to the input and runs it again, flogging the horse. Downstream another tool, some kind of harness, tries to control this stream by injecting tokens into the context and then sending it to the inference tool, and then trying to pattern-match the output.
Finally, there you are on CodePorn.yata paying for an agent to generate code, paying for an agent to tell you what's wrong with it, and paying for an agent to make it differently bad, and hopefully move on to the next task.
If it still hasn't dawned on you that this isn't just a bubble, but a snake-oil-bubble-bath, just try to imagine the paradigm shift whereby you go on github.com, assign an issue to an agent, the agent fixes it by rewriting the application in Pascal but a reviewing agent catches that you wanted it to print a measurement in Pascals (pa), and you don't pay for the work or the review, you only pay for work that one or two reviewing agents determine is up to par.
Nobody is going to do that because as soon as they test it they're going to have to do some math that won't make sense without admitting/realizing it's not some near-sentient, AGI rating 0.9 intelligence, it's just a text prediction algorithm that can pull out entire sentences when you use it to infer output on topics it trained on.
This downplays the incredible things that can be done with it.
There's a lot of noise, yes. How long has the web existed? And yet we're still figuring out how to optimize (HTTP/3).
Disregard the signal at your own expense.
None of the big players involved right now seem to be doing anything to disarm the perception that the LLM does all the work.
I feel like we're at a state fair 180 years ago having just seen an impressive demonstration of small animals momentarily coming back to life, someone in the crowd said 'eternal life' and everyone lost their minds trying to give money to the 'inventor' on the stage who applied the electric current.
Ask some lay people or junior devs how they feel about LLM mistakes, listen for answers about "learning" etc. Do they realize the model doesn't change? The same set of weights and tensors are deployed to every GPU hosting it. If you use a power tool that lets you specify the temperature and seed, it will generate the exact same output time after time.
"But I told it to always speak like a pirate, and it does": No, you signed up to pay for the model to be told to speak like a pirate every time you use it.
There - that's the curtain I'm talking about that you're lifting/tearing. I think momentarily downplaying the potential of LLMs themselves is worth it to expose that.
Could it be cloud-based? That changes the risk and scale calculus in a way that's going to take a long time to get funding into.
Careful you don't "it's just a text predictor" yourself into unemployment .
says more than anything else could
Says the guy who is "I-can-now-just-push-buttons-and-get-paid-forever"-ing into not only into unemployment, but also being totally redundant.
Before you come back with, "Oh I review everything LLM does": Only in your dreams you ll gain or retain experience by just reviewing stuff. That is even if you actually review every line that the LLM writes...which have literally zero chance of happening. People have hard time keeping their eye open even when their life is on the line when using a pseudo self driving car...
Elsewhere, I still haven't reviewed the vast majority of AI code I've generated, but all of the AI-generated code I've submitted I did review exhaustively. I have a couple of past roles to thank for drilling some strong code-review strategies and - thus far - have had the willpower to reject an AI generation when I know I can't/won't diligently review.
LLMs do not do anything but generate a token. It's a trivial but critical distinction like RNG vs pRNG. The LLM doesn't learn, it doesn't do backtracking, their output is deterministic.
Not "chatgpt.com"s output, but the LLM powering it; Anthropic aren't customizing a version of Claude's weights/tensors on your inputs, they're adding billable input tokens to the baseline of your subsequent contexts.
Its very financially convenient for Anthropic and OpenAI if people think of the LLM as doing the work, because for a lot of people that conjures a sense of a system that inherently learns and improves.
Even some of the people I've worked with on low-level LLM harnessing tools/systems since early 2023 fall for it and start thinking of the LLM as an AI with the connotations of back-propagation, weight adjustment, ... learning.
The "AI" is in what the harness software does with the inference output and the context it feeds back to it.
It took forever to convince people that quality degrades with context length but then all we got was compaction, there's still huge resistance to harnesses actively curating the context.
A lot of non-devs I've spoken to assumed when their tool said "Now I need to re-read" it was overwriting the 'in memory' copy.
You open aider or code or claude and ask it to fix a bug in file.<c|cpp|js|py>, it reads the entire 8k file, reasons about it (in which it sometimes echoes specific lines of code), and then it "edits the file".
To edit a file, inference currently injects a stream of tokens into the context that your software identifies as a tool call.
[tool name="edit" file="..."] @120,131 - halt_and_catch_fire() + exit_nicely() [/tool]
And frequently the tooling will guide the model to re-read the file, in some pathological cases by prompting the model to generate a tool instruction to do it.
Your context now contains: pre-prompting, your prompt, first file read (8k tokens), reasoning, all the tool calls to make the changes, re-read (8k tokens), conclusory reasoning, tooling prompts llm to describe what has been done, work-done-summary.
Because of the way the underlying LLM works and because of the way the APIs are presented, tools are discouraged from redacting or eliding the original read.
Otherwise, what they'd already be doing is curating the context: read file once, append output from inference/tool calls, and when it's done, replace the original insert of the file.
Having 2 or more disagreeing versions of a significant source file in your context is detrimental to model attention and output quality, and you either see people who get it and manage it, or who wallow in superstition and hand-waving and a near eagerness to pay for "mistakes" that the "model is making" because they believe it is learning, on the fly.
The point at which the metrics become meaningless is when others become aware of them, and begin to optimise for them. Lines per code is is not a bad insight for development activity, only when the developers are not aware of the metric. Price per 1M tokens became meaningless when LLM providers started to optimise for it. It seems to be that Sonnet 5 is optimised to score well on AA intelligence whilst seemingly having a lower price per 1M tokens.
I think generally we are in an AI bubble, and it will at some point pop. The numbers simply don't make sense. I would gamble heavily on local cost per task to survive the LLM winter. Given that hardware is pretty much a fixed overhead, you probably want to optimise for task per kW - that's where I'm betting.