The article does a great job of highlighting the core disconnect in the LLM API economy: linear pricing for a service with non-linear, quadratic compute costs. The traffic analogy is an excellent framing.
One addition: the O(n^2) compute cost is most acute during the one-time prefill of the input prompt. I think the real bottleneck, however, is the KV cache during the decode phase.
For each new token generated, the model must access the intermediate state of all previous tokens. This state is held in the KV Cache, which grows linearly with sequence length and consumes an enormous amount of expensive GPU VRAM. The speed of generating a response is therefore more limited by memory bandwidth.
Viewed this way, Google's 2x price hike on input tokens is probably related to the KV Cache, which supports the article’s “workload shape” hypothesis. A long input prompt creates a huge memory footprint that must be held for the entire generation, even if the output is short.
"In a move that at first went unnoticed, Google significantly increased the price of its popular Gemini 2.5 Flash model"
It's not quite that simple. Gemini 2.5 Flash previously had two prices, depending on if you enabled "thinking" mode or not. The new 2.5 Flash has just a single price, which is a lot more if you were using the non-thinking mode and may be slightly less for thinking mode.
Another way to think about this is that they retired their Gemini 2.5 Flash non-thinking model entirely, and changed the price of their Gemini 2.5 Flash thinking model from $0.15/m input, $3.50/m output to $0.30/m input (more expensive) and $2.50/m output (less expensive).
Another minor nit-pick:
> For LLM providers, API calls cost them quadratically in throughput as sequence length increases. However, API providers price their services linearly, meaning that there is a fixed cost to the end consumer for every unit of input or output token they use.
That's mostly true, but not entirely: Gemini 2.5 Pro (but oddly not Gemini 2.5 Flash) charges a higher rate for inputs over 200,000 tokens. Gemini 1.5 also had a higher rate for >128,000 tokens. As a result I treat those as separate models on my pricing table on https://www.llm-prices.com
One last one:
> o3 is a completely different class of model. It is at the frontier of intelligence, whereas Flash is meant to be a workhorse. Consequently, there is more room for optimization that isn’t available in Flash’s case, such as more room for pruning, distillation, etc.
OpenAI are on the record that the o3 optimizations were not through model changes such as pruning or distillation. This is backed up by independent benchmarks that find the performance of the new o3 matches the previous one: https://twitter.com/arcprize/status/1932836756791177316
> This is the first time a major provider has backtracked on the price of an established model
Arguably that was Haiku 3.5 in October 2024.
I think the same hypothesis could apply though, that you price your model expecting a certain average input size, and then adjust price up to accommodate the reality that people use that cheapest model when they want to throw as much as they can into the context.
Llama 4 maverick is 16x 17b. So 67GB of size. The equivalency is 400billion.
Llama 4 behemoth is 128x 17b. 245gb size. The equivalency is 2 trillion.
I dont have the resources to be able to test these unfortunately; but they are claiming behemoth is superior to the best SAAS options via internal benchmarking.
Comparatively Deepseek r1 671B is 404gb in size; with pretty similar benchmarks.
But you compare deepseek r1 32b to any model from 2021 and it's going to be significantly superior.
So we have quality of models increasing, resources needed decreasing. In 5-10 years, do we have an LLM that loads up on a 16-32GB video card that is simply capable of doing it all?
> If you’re building batch tasks with LLMs and are looking to navigate this new cost landscape, feel free to reach out to see how Sutro can help.
I don't have any reason to doubt the reasoning this article is doing or the conclusions it reaches, but it's important to recognize that this article is part of a sales pitch.
Unfounded extrapolation from a minor pricing update. I am sure every generation of chips also came with “end of Moore’s law” articles for the actual Moore’s law.
FWIW Gemini 2.5 Flash Lite is still very good; I used it in my latest side project to generate entire web sites and it outputs great content and markup every single time.
>By embracing batch processing and leveraging the power of cost-effective open-source models, you can sidestep the price floor and continue to scale your AI initiatives in ways that are no longer feasible with traditional APIs.
Context size is the real killer when you look at running open source alternatives on your own hardware. Has anything even come close to the 100k+ range yet?
Is there math backing up the “quadratic” statement with LLM input size? At least in the traffic analogy, I imagine it’s exponential, but for small amounts exceeding some critical threshold, a quadratic term is sufficient
Stopped reading here, if you're positioning yourself
as if you have some kind of unique insight when there is none in order to boost youe credentials and sell your product there's little chance you have anything actually insightful to offer. Might sound like an overreaction/nitpicking but it's entirely needless LinkedIn style "thought leader" nonsense.
In reality it was immediately noticed by anyone using these models, have a look at the HN threads at the time, or even on Reddit, let alone the actual spaces dedicated to AI builders.
What is holding back AI is this business necessity that models must perform everything. Nobody can push for a smaller model that learns a few simple tasks and then build upon that, similar to the best known intelligent machine: the human.
If these corporations had to build a car they would make the largest possible engine, because "MORE ENGINE MORE SPEED", just like they think that bigger models means bigger intelligence, but forget to add steering, or even a chassi.
Extremely doubtful that it boils down to quadratic scaling of attention. That whole issue is a leftover from the days of small bert models with very few parameters.
Google is raising prices for most of their services. I do not agree that this is due to the cost of compute or that this is the end of Moore’s Law. I don’t think we have scratched the surface.
Basing anything on Google's pricing is folly. Quite recently Google offered several of their preview models at a price of $0.00.
Because they were the underdog. Everyone was talking about ChatGPT, or maybe Anthropic. Then Deepseek. Google were the afterthought that was renowned for that ridiculous image generator that envisioned 17th century European scientists as full-headdress North American natives.
There has been absolute 180 since then, and Google now has the ability to set their pricing similar to the others. Indeed, Google's pricing still has a pretty large discount over similarly capable model levels, even after they raised prices.
The warning is that there is no free lunch, and when someone is basically subsidizing usage to get noticed, they don't have to do that once their offering is good.
Is this overthinking it? Google had a huge incentive to outprice Anthropic and OAI to join the "conversation". I was certainly attracted to the low price initially, but I'm staying because it's still affordable and I still think the Gemini 2.5 options are the best simple mix of models available.
This is a marketing blog[^1], written with AI[^2], heavily sensationalized, & doesn't understand much in the first place.
We don't have accurate price signals externally because Google, in particular, had been very aggressive at treating pricing as a competition exercise than anything that seemed tethered to costs.
For quite some time, their pricing updates would be across-the-board exactly 2/3 of the cost of OpenAI's equivalent mode.
[^1] "If you’re building batch tasks with LLMs and are looking to navigate this new cost landscape, feel free to reach out to see how Sutro can help."
[^2] "Google's decision to raise the price of Gemini 2.5 Flash wasn't just a business decision; it was a signal to the entire market." by far the biggest giveaway, the other tells are repeated fanciful descriptions of things that could be real, that when stacked up, indicate a surreal, artifical, understanding of what they're being asked to write about, i.e. "In a move that at first went unnoticed,"
One of the clearest example is Deepseek v3. Deepseek has mentioned its price of 0.27/1.10 has 80% profit margin, so it cost them 90% lesser than the price of Gemini flash. And Gemini flash is very likely smaller model than Deepseek v3.
Oh, I noticed. I've also complained how Gemini 2.0 Flash is 50% more expensive than Gemini 1.5 Flash for small requests.
Also I'm sure if Google wanted to price Gemini 2.5 Flash cheaper they could. The reason they won't is because there is almost zero competition at the <10 cents per million input token area. Google's answer to the 10 cents per million input token area is 2.5 Flash Lite which they say is equivalent to 2.0 Flash at the same cost. Might be a bit cheaper if you factor in automatic context caching.
Also the quadratic increase is valid but it's not as simple as the article states due to caching. And if it was a bit issue Google would impose tiered pricing like they do for Gemini 2.5 Pro.
And for what it's worth I've been playing around with Gemma E4B on together.ai. It takes 10x as long as Gemini 2.5 Flash Lite and it sucks at multilingual. But other than that it seems to produce acceptable results and is way cheaper.
I think providers are making a mistake in simplifying prices at all costs, hiding the quadratic nature of attention. People can understand the pricing anyway, even if more complex, by having a tool that let them select a prompt and a reply length and see the cost, or fancy 3D graphs that capture the cost surface of different cases. People would start sending smaller prompts and less context when less is enough, and what they pay would be more related to the amount of GPU/TPU/... power they use.
I feel that the details regarding the type of the model and the purpose it serves are underrepresented here. Yes existing models will get cheaper over time as they become more obsolete but to be at the forefront of innovation or models costs will only increase due to the these mentioned bottlenecks. Also there is the basic law of supply and demand coming into play here. As models get more advanced more industries will be exposed to them and see the potential cost savings compared to the current alternative. This will further increase demand and with further innovation, there will be further capability in turn again increasing demand. I only see this reversing is you are not at the forefront of innovation and many people using these LLMs at this point are close at least compared to many "normal" people and their understandings of LLMs.
Can anyone explain the economics of Anthropic's Max plan pricing to me? I have friends on the $100/month plan using well over $800 of tokens per month with Claude Code (according to ccusage). I certainly don't use Claude Code as much if I'm not on a flat rate plan, the cost spirals out of control very quickly. I understand that a subscription makes for more predictable revenue and that there will be people on the Max plan not using Claude Code 24/7, but the delta between what the API costs and what using the Max plan with Claude Code costs just seems too great for that to be an explanation. I don't think that user/mindshare capture can fully explain it either, Code is free and the cost of switching to something else if pricing later changes is just too low. I don't get it.
We’re in an LLM bubble and their money is cheap as they’re drowning in investor money and have to spend it + show growth. If it doesn’t make economic sense you probably can’t count on it to last once the bubble bursts.
The article assumes that there will be no architectural improvements / migrations in the future, & that Sparse MoE will always stay. Not a great foundation to build upon.
Personally, I'm rooting for RWKV / Mamba2 to pull through, somehow. There's been some work done to increase their reasoning depths, but transformers still beat them without much effort.
29 comments
[ 2.4 ms ] story [ 36.3 ms ] threadOne addition: the O(n^2) compute cost is most acute during the one-time prefill of the input prompt. I think the real bottleneck, however, is the KV cache during the decode phase.
For each new token generated, the model must access the intermediate state of all previous tokens. This state is held in the KV Cache, which grows linearly with sequence length and consumes an enormous amount of expensive GPU VRAM. The speed of generating a response is therefore more limited by memory bandwidth.
Viewed this way, Google's 2x price hike on input tokens is probably related to the KV Cache, which supports the article’s “workload shape” hypothesis. A long input prompt creates a huge memory footprint that must be held for the entire generation, even if the output is short.
It's not quite that simple. Gemini 2.5 Flash previously had two prices, depending on if you enabled "thinking" mode or not. The new 2.5 Flash has just a single price, which is a lot more if you were using the non-thinking mode and may be slightly less for thinking mode.
Another way to think about this is that they retired their Gemini 2.5 Flash non-thinking model entirely, and changed the price of their Gemini 2.5 Flash thinking model from $0.15/m input, $3.50/m output to $0.30/m input (more expensive) and $2.50/m output (less expensive).
Another minor nit-pick:
> For LLM providers, API calls cost them quadratically in throughput as sequence length increases. However, API providers price their services linearly, meaning that there is a fixed cost to the end consumer for every unit of input or output token they use.
That's mostly true, but not entirely: Gemini 2.5 Pro (but oddly not Gemini 2.5 Flash) charges a higher rate for inputs over 200,000 tokens. Gemini 1.5 also had a higher rate for >128,000 tokens. As a result I treat those as separate models on my pricing table on https://www.llm-prices.com
One last one:
> o3 is a completely different class of model. It is at the frontier of intelligence, whereas Flash is meant to be a workhorse. Consequently, there is more room for optimization that isn’t available in Flash’s case, such as more room for pruning, distillation, etc.
OpenAI are on the record that the o3 optimizations were not through model changes such as pruning or distillation. This is backed up by independent benchmarks that find the performance of the new o3 matches the previous one: https://twitter.com/arcprize/status/1932836756791177316
Arguably that was Haiku 3.5 in October 2024.
I think the same hypothesis could apply though, that you price your model expecting a certain average input size, and then adjust price up to accommodate the reality that people use that cheapest model when they want to throw as much as they can into the context.
Llama 4 maverick is 16x 17b. So 67GB of size. The equivalency is 400billion.
Llama 4 behemoth is 128x 17b. 245gb size. The equivalency is 2 trillion.
I dont have the resources to be able to test these unfortunately; but they are claiming behemoth is superior to the best SAAS options via internal benchmarking.
Comparatively Deepseek r1 671B is 404gb in size; with pretty similar benchmarks.
But you compare deepseek r1 32b to any model from 2021 and it's going to be significantly superior.
So we have quality of models increasing, resources needed decreasing. In 5-10 years, do we have an LLM that loads up on a 16-32GB video card that is simply capable of doing it all?
I don't have any reason to doubt the reasoning this article is doing or the conclusions it reaches, but it's important to recognize that this article is part of a sales pitch.
FWIW Gemini 2.5 Flash Lite is still very good; I used it in my latest side project to generate entire web sites and it outputs great content and markup every single time.
Context size is the real killer when you look at running open source alternatives on your own hardware. Has anything even come close to the 100k+ range yet?
Stopped reading here, if you're positioning yourself as if you have some kind of unique insight when there is none in order to boost youe credentials and sell your product there's little chance you have anything actually insightful to offer. Might sound like an overreaction/nitpicking but it's entirely needless LinkedIn style "thought leader" nonsense.
In reality it was immediately noticed by anyone using these models, have a look at the HN threads at the time, or even on Reddit, let alone the actual spaces dedicated to AI builders.
If these corporations had to build a car they would make the largest possible engine, because "MORE ENGINE MORE SPEED", just like they think that bigger models means bigger intelligence, but forget to add steering, or even a chassi.
For large models, compute is very rarely dominated by attention. Take, for example, this FLOPs calculation from https://www.adamcasson.com/posts/transformer-flops
Compute per token = 2(P + L × W × D)
P: total parameters L: Number of Layers W: context size D: Embedding dimension
For Llama 8b, the window size starts dominating compute cost per token only at 61k tokens.
Because they were the underdog. Everyone was talking about ChatGPT, or maybe Anthropic. Then Deepseek. Google were the afterthought that was renowned for that ridiculous image generator that envisioned 17th century European scientists as full-headdress North American natives.
There has been absolute 180 since then, and Google now has the ability to set their pricing similar to the others. Indeed, Google's pricing still has a pretty large discount over similarly capable model levels, even after they raised prices.
The warning is that there is no free lunch, and when someone is basically subsidizing usage to get noticed, they don't have to do that once their offering is good.
We don't have accurate price signals externally because Google, in particular, had been very aggressive at treating pricing as a competition exercise than anything that seemed tethered to costs.
For quite some time, their pricing updates would be across-the-board exactly 2/3 of the cost of OpenAI's equivalent mode.
[^1] "If you’re building batch tasks with LLMs and are looking to navigate this new cost landscape, feel free to reach out to see how Sutro can help."
[^2] "Google's decision to raise the price of Gemini 2.5 Flash wasn't just a business decision; it was a signal to the entire market." by far the biggest giveaway, the other tells are repeated fanciful descriptions of things that could be real, that when stacked up, indicate a surreal, artifical, understanding of what they're being asked to write about, i.e. "In a move that at first went unnoticed,"
One of the clearest example is Deepseek v3. Deepseek has mentioned its price of 0.27/1.10 has 80% profit margin, so it cost them 90% lesser than the price of Gemini flash. And Gemini flash is very likely smaller model than Deepseek v3.
Oh, I noticed. I've also complained how Gemini 2.0 Flash is 50% more expensive than Gemini 1.5 Flash for small requests.
Also I'm sure if Google wanted to price Gemini 2.5 Flash cheaper they could. The reason they won't is because there is almost zero competition at the <10 cents per million input token area. Google's answer to the 10 cents per million input token area is 2.5 Flash Lite which they say is equivalent to 2.0 Flash at the same cost. Might be a bit cheaper if you factor in automatic context caching.
Also the quadratic increase is valid but it's not as simple as the article states due to caching. And if it was a bit issue Google would impose tiered pricing like they do for Gemini 2.5 Pro.
And for what it's worth I've been playing around with Gemma E4B on together.ai. It takes 10x as long as Gemini 2.5 Flash Lite and it sucks at multilingual. But other than that it seems to produce acceptable results and is way cheaper.
It could just as well have been Google reducing subsidisation. From the outside that would look exactly the same
Personally, I'm rooting for RWKV / Mamba2 to pull through, somehow. There's been some work done to increase their reasoning depths, but transformers still beat them without much effort.
https://x.com/ZeyuanAllenZhu/status/1918684269251371164
In terms of microbiology, the architecture of Transformer is more in line with the highly interconnected global receptive field of neurons
https://github.com/dmf-archive/PILF