A major current problem is that we're smashing gnats with sledgehammers via undifferentiated model use.
Not every problem needs a SOTA generalist model, and as we get systems/services that are more "bundles" of different models with specific purposes I think we will see better usage graphs.
Because none of them are good enough yet to trust completely with any task. Even the absolute best ones still fart out at surprising times, and for most stuff I have an AI that's always on, it requires no cognitive overhead to delegate to my own brain. So to delegate, it has to be a reliable win: I'm not here to make AI look good, I'm here to make my own performance be good, only a sure thing is a candidate for reflexive delegation.
AI companies advertise peak AI performance, users select AI tools on worst case AI fuckups: hence, only SOTA is ever in demand. TFA illustrates this well.
AI will be judged on it's worst performance, just like people are fired for their worst showing, not their best. No one cares about AI performance in ideal (read: carefully contrived) settings. We care how bad it fucks up when we take our eyes off it for 2 seconds.
This is a place testing and benchmarking can definitely save you money.
It's the same as compute--you can skip testing and throw money at the problem but you're going to end up paying more.
We have some pretty basic guidelines at work and I think that's a decent starting point. They amount to a few example prompts/problem types and which OpenAI model to try using first for best bang for your buck.
I think some of it also comes down to scale. Buying a 5 pack of sledgehammers isn't a terrible value when everything comes in a "5 pack" and you only need <= 5 tools total. Or more practically, on the small end it's more economical to run general purpose models than tailor more specific models. Once you start invoking them enough, there's a break even and flip point where spending more time on the tailored or custom model is cheaper.
I'm kind of curious what IntelliJ's deal is with the different providers. I usually just keep it set to Claude but there are others that you can pick. I don't pay extra for the AI assistant - it's part of my regular subscription. I don't think I use the AI features as heavily as many others, but it does feed my code base to whoever I'm set to...
Mathematics are not relevant when we have hype and vibes. We can't have facts and projections and no path to profitability distract us from our final goal.
Which, of course, is to donate money to Sama so he can create AGI and be less lonely with his robotic girlfriend, I mean...change the world for the better somehow. /s
The truth is we're brute forcing some problems via tremendous amount of compute. Especially for apps that use AI backends (rather than chats where you interface with the LLM directly), there needs to be hybridization. I haven't used Claude Code myself but I did a screenshare session with someone who does and I think I saw it running old fashioned keyword search on the codebase. That's much more effective than just pushing more and more raw data into the chat context.
On one of the systems I'm developing I'm using LLMs to compile user intents to a DSL, without every looking at the real data to be examined. There are ways; increased context length is bad for speed, cost and scalability.
I have already thought a lot about the large packaged inference companies hitting a financial brick wall, but I was surprised by material near the end of the article: the discussions of lock in for companies that can’t switch and about Replit making money on the whole stack. Really interesting.
I managed a deep learning team at Capital One and the lock-in thing is real. Replit is an interesting case study for me because after a one week free agent trial I signed up for a one year subscription, had fun the their agent LLM-based coding assistant for a few weeks, and almost never used their coding agent after that, but I still have fun with Replit as an easy way to spin up Nix based coding environments. Replit seems to offer something for everyone.
First of all the title is click-bait. Tokens are getting cheaper and cheaper. People just use more and more tokens.
And everything, I mean everything after the title is only a downhill:
> saying "this car is so much cheaper now!" while pointing at a 1995 honda civic misses the point. sure, that specific car is cheaper. but the 2025 toyota camry MSRPs at $30K.
Cars got cheaper. The only reason you don't feel it is trade barrier that stops BYD from flooding your local dealers.
> charge 10x the price point
> $200/month when cursor charges $20. start with more buffer before the bleeding begins.
What does this even mean? The cheapest Cursor plan is $20, just like Claude Code. And the most expensive Cursor plan is $200, just like Claude Code. So clearly they're at the exact same price point.
> switch from opus ($75/m tokens) to sonnet ($15/m) when things get heavy. optimize with haiku for reading. like aws autoscaling, but for brains.
> they almost certainly built this behavior directly into the model weights, which is a paradigm shift we’ll probably see a lot more of
"I don't know how Claude built their models and I have no insider knowledge, but I have very strong opinions."
> 3. offload processing to user machines
What?
> ten. billion. tokens. that's 12,500 copies of war and peace. in a month.
Unironically quoting data from viberank leaderboard, which is just user-submitted number...
> it's that there is no flat subscription price that works in this new world.
The author doesn't know what throttling is...?
I've stopped reading here. I should've just closed the tab when I saw the first letter in each sentence isn't capitalized. This is so far the most glaring signal of slop. More than the overuse of em-dash and lists.
> the first letter in each sentence isn't capitalized. This is so far the most glaring signal of slop.
just fyi, it's a very common manner of writing for younger folk online. more so in informal contexts, but as with everything else, once it's widely adopted it starts to creep into the more formal communication. it's not about "slop", it's just a cultural convention.
i should also note that many languages that got their orthographies defined relatively recently (e.g. various native american languages) use all-lowercase as well, by design. so there's no inherent reason why english can't do that either.
Over the past year or two I've just been paying for the API access and using open source frontends like LibreChat to access these models.
This has been working great for the occasional use, I'd probably top up my account by $10 every few months. I figured the amount of tokens I use is vastly smaller than the packaged plans so it made sense to go with the cheaper, pay-as-you-go approach.
But since I've started dabbling in tooling like Claude Code, hoo-boy those tokens burn _fast_, like really fast. Yesterday I somehow burned through $5 of tokens in the space of about 15 minutes. I mean, sure, the Code tool is vastly different to asking an LLM about a certain topic, but I wasn't expecting such a huge leap, a lot of the token usage is masked from you I guess wrapped up in the ever increasing context + back/forth tool orchestration, but still
Insisting on flaunting English spelling rules (by not starting a sentence with a capital letter) in a think piece is a dead giveaway that the author thinks too highly of themselves, and results in me automatically discounting whatever they're saying.
If I (and billions others) can be bothered to learn your damn language so we can all communicate, do us a service and actually use it properly, FFS.
> claude code has had to roll back their original unlimited $200/mo tier this week
The article repeats this throughout but isn't it a straight lie? The plan was named 20x because it's 20x usage limits, it always had enforced 5 hour session limits, it always had (unenforced? soft?) 50 session per month limits.
It was limited, but not enough and very very probably still isn't, judging by my own usage. So I don't think the argument would even suffer from telling the truth.
Also the “walkback” of the unlimited Max plan was anything but. The plan is still exactly the same for 95% of the subscribers. It just turns out that less than 5% were doing things like scripting claude to run multiple sessions 24/7 with the same login credentials. The Max plan was supposed to be a single user plan, not an infinite-parallel-sessions plan.
Probably they asked AI if it was unlimited and it responded something like "oh wise and sagacious user what an amazingly insightful question! Yes, yes Max is unlimited! Would you like me to help you use an infinite amount of tokens?"
My team is debating this exact question for a new product we have in early access. Ultimately we realized the issue early on, so even our plans option would include at-cost usage limits.
> consumers hate metered billing. they'd rather overpay for unlimited than get surprised by a bill.
Yes and no.
Take Amazon. You think your costs are known and WHAMMO surprise bill. Why do you get a surprise bill? Because you cannot say 'Turn shit off at X money per month'. Can't do it. Not an option.
All of these 'Surprise Net 30' offerings are the same. You think you're getting a stable price until GOTAHCA.
Now, metered billing can actually be good, when the user knows exactly where they stand on the metering AND can set maximums so their budget doesn't go over.
Taken realistically, as an AI company, you provide a 'used tokens/total tokens' bar graph, tokens per response, and estimated amount of responses before exceeding.
Again, don't surprise the user. But that's an anathema to companies who want to hide tokens to dollars, the same way gambling companies obfuscate 'corporate bux' to USD.
Metered billing makes sense for B2B infrastructure-as-a-service type products (AWS), where as your company grows both you and the infra provider know the bill will grow manageably over time. Infra is set it and forget it.
But for AI in the context of point-solutions and on-the-job use cases, metered billing is a death blow.
In this context, metered is a massive incentive to not use the product and requires the huge friction of having to do a cost/benefit analysis before every task. And if you're using it at work you may even need management sign-off before you can use it again.
For a tool that's intended to amplify productivity, very few humans want to make a cost/benefit analysis 250 times a day whether it's worth $3 to code up a boilerplate or not. On metered billing, they just wont use it.
> consumers hate metered billing. they'd rather overpay for unlimited than get surprised by a bill.
Standard packages are like insurance. Everyone pays more or less the same premium, but some claim more than others. On average people always overpay for insurance.
The upside is that it's a predictable cost for the users, and also means predictable cash flow for the provider.
You get a surprise bill because of surprise usage of services billed based on usage.
If you ask anyone how much water or electricity they use per month, the first thing they're going to do is look at last month's usage.
Estimating what you need ahead of time is a hard problem.
In fairness, AWS doesn't give you a lot of tools to help you measure and predict how many "units" you'll use outside of running your thing and measuring. On the other hand, running your thing and measuring is the defacto way to figure out how much of something you'll use.
Finally, there are AWS services like EC2 and RDS you can run at a fixed cost to help you stay within budget. Traffic/bandwidth is the only thing that comes to mind that you're pretty much required to use without a way to fix the cost (although you can get pretty close with bandwidth limits on EC2 interfaces)
That's like the introductory prices of the incumbent giant telecom here in Canada: he's $60/month for gigabit internet + phone + TV for 1 year + a $250 prepaid Visa gift card! Oh, you didn't want the bill for $300 on month 13?
While reading this, every time I started a paragraph and saw a lowercase, my brain and eyes were stalling or jumping up, to reflexively look for the text that got cut off. My brain has been trained for decades that, when reading full prose, a paragraph starting with lowercase means I'm starting in the middle of a sentence, and something happened in the layout or HTML to interrupt it.
And, I know this seems dramatic, but besides being cognitively distracting, it also makes me feel sad. Chatroom formatting in published writings is clearly a developing trend at this point, and I love my language so much. Not in a linguistic capacity - I'm not an English expert or anything, nor do I follow every rule - I mean in an emotional capacity.
I'm not trying to be condescending. This is a style choice, not "bad writing" in the typical sense. I realize there is often a lot of low-quality bitterness on both sides about this kind of thing.
Edit:
I also fear that this is exactly the kind of thing where any opinion in opposition to this style will feel like the kind of attack that makes a writer want to push back in a "oh yeah? fuck you" kind of way. I.e. even just my writing this opinion may give an author using the style in question the desire to "double down". Though this conundrum is appropriate (ironic?) - the intensely personal nature of language is part of why I love it.
We haven't reached a peak on scaling/performance, so even if an old model can be commoditized, a new one will be created to take advantage of the newly freed infra. Until we hit a ceiling on scaling, tokens are going to remain expensive relative to what people are trying to do with them because the underlying compute is expensive.
On the topic of cost per token, is it accurate to represent a token as, ideally, a composable atomic unit of information. But because we’re (often) using English as the encoding format, it can only be as efficient as English can encode the data.
Does this mean that other languages might offer better information density per token? And does this mean that we could invent a language that’s more efficient for these purposes, and something humans (perhaps only those who want a job as a prompt engineer) could be taught?
In principle, yes. And you could do the same for programming languages.
In practice, the problem is that any such constructed language wouldn't have a corpus large enough to train on.
It's really unfortunate that we ended up with English as the global lingua franca right at the time generative AI came about, because it is effectively cementing that dominance. Even Chinese models are trained mostly on English AFAIK.
But also, arguably, Lojban is the language you want to use for LLMs. Especially for the chain of thought.
And the interesting property of Lojban is that it has unambiguous grammar that can be syntax-checked by tools and enforced by schemas, and machine-translated back to English. I experimented with it a bit and found that large SOTA models can generate reasonably accurate translations if you give them tools like dictionary and parser and tell them to iterate until they get a syntactically valid translation that parses into what they meant to say. So perhaps there is a way to generate a large enough dataset to train a model on; I wish I had enough $$$ to try this on a lark.
I tried Gemini CLI and in 2 hours somehow spent $22 just messing around with a very small codebase. I didn’t find out until the next day from Google’s billing system. That was enough for me - I won’t touch it again.
Same. 30m tokens in a few hours (of which many were cached it seems and only a few output tokens). I use Gemini to solve only specific problems and in my case $20+ was worth it.
Interesting article, full of speculation and some logical follows, but feels like it feels short of admitting what the true conclusion is. Model building companies can build thinner wrapper / harness and can offer better prices than third party companies (the article assumes it costs anthropic same price per token as it does for their customers) because their costs per token is lower than app layer companies. Anthropic has a decent margin (likely higher than openai) on sale of every token, and with more scale, they can sell at a lower cost (or some unlimited plans with limits that keeps out 1%-5% of the power users).
I don't agree with the Cognition conclusion either. Enterprises are fighting super hard to not have a long term buying contract when they know SOTA (app or model) is different every 6 months. They are keeping their switching costs low and making sure they own the workflow, not the tool. This is even more prominent after Slack restricted API usage for enterprise customers.
Making money on the infra is possible, but that again misunderstands the pricing power of Anthropic. Lovable, Replit etc. work because of Claude. Openai had codex, google had jules, both aren't as good in terms of taste compared to Claude. It's not the cli form factor which people love, it's the outcome they like. When Anthropic sees the money being left on the table in infra play, they will offer the same (at presumably better rates given Amazon is an investor) and likely repeat this strategy. Abstraction is a good play, only if you abstract it to the maximum possible levels.
This is the moment an open source solution could pop in and say just "uv add aider" and then make sure you have a 24gb card for Qwen3 for each dev, and you are future proofed for at least the next year. It seems like the only way out.
> when a new model is released as the SOTA, 99% of the demand immediately shifts over to it
99% is in the wrong ballpark. Lots of users use Sonnet 4 over Opus 4, despite Opus being 'more' SOTA. Lots of users use 4o over o3 or Gemini over Claude. In fact it's never been a closer race on who is the 'best': https://openrouter.ai/rankings
>switch from opus ($75/m tokens) to sonnet ($15/m) when things get heavy. optimize with haiku for reading. like aws autoscaling, but for brains.
they almost certainly built this behavior directly into the model weights
???
Overall the article seems to argue that companies are running into issues with usage-based pricing due to consumers not accepting or being used to usage based pricing and it's difficult to be the first person to crack and switch to usage based.
I don't think it's as big of an issue as the author makes it out to be. We've seen this play out before in cloud hosting.
- Lots of consumers are OK with a flat fee per month and using an inferior model. 4o is objectively inferior to o3 but millions of people use it (or don't know any better). The free ChatGPT is even worse than 4o and the vast majority of chatgpt visitors use it!
- Heavy users or businesses consume via API and usage based pricing (see cloud). This is almost certainly profitable.
- Fundamentally most of these startups are B2B, not B2C
Personally I just avoid OpenAIs models entirely because I have absolutely no way of telling how their products stack up against one another or which to use for what. In what world does o3 sort higher than 4o?
If I have to research your products by name to determine what to use for something that is already a commodity, you've already lost and are ruled out.
> now look at the actual pricing history of frontier models, the ones that 99% of the demand is for at any given time:
The meaningful frontier isn't scalar on just the capability, it's on capability for a given cost. The highest capability models are not where 99% of the demand is on. Actually the opposite.
To get an idea of what point on the frontier people prefer, have a look at the OpenRouter statistics (https://openrouter.ai/rankings). Claude Opus 4 has about 1% of their total usage, not 99%. Claude Sonnet 4 is the single most popular model at about 18%. The runners up in volume are Gemini Flash 2.0 and 2.5, which are in turn significantly cheaper than Sonnet 4.
68 comments
[ 2.9 ms ] story [ 79.9 ms ] threadNot every problem needs a SOTA generalist model, and as we get systems/services that are more "bundles" of different models with specific purposes I think we will see better usage graphs.
AI companies advertise peak AI performance, users select AI tools on worst case AI fuckups: hence, only SOTA is ever in demand. TFA illustrates this well.
AI will be judged on it's worst performance, just like people are fired for their worst showing, not their best. No one cares about AI performance in ideal (read: carefully contrived) settings. We care how bad it fucks up when we take our eyes off it for 2 seconds.
It's the same as compute--you can skip testing and throw money at the problem but you're going to end up paying more.
We have some pretty basic guidelines at work and I think that's a decent starting point. They amount to a few example prompts/problem types and which OpenAI model to try using first for best bang for your buck.
I think some of it also comes down to scale. Buying a 5 pack of sledgehammers isn't a terrible value when everything comes in a "5 pack" and you only need <= 5 tools total. Or more practically, on the small end it's more economical to run general purpose models than tailor more specific models. Once you start invoking them enough, there's a break even and flip point where spending more time on the tailored or custom model is cheaper.
Which, of course, is to donate money to Sama so he can create AGI and be less lonely with his robotic girlfriend, I mean...change the world for the better somehow. /s
On one of the systems I'm developing I'm using LLMs to compile user intents to a DSL, without every looking at the real data to be examined. There are ways; increased context length is bad for speed, cost and scalability.
I managed a deep learning team at Capital One and the lock-in thing is real. Replit is an interesting case study for me because after a one week free agent trial I signed up for a one year subscription, had fun the their agent LLM-based coding assistant for a few weeks, and almost never used their coding agent after that, but I still have fun with Replit as an easy way to spin up Nix based coding environments. Replit seems to offer something for everyone.
And everything, I mean everything after the title is only a downhill:
> saying "this car is so much cheaper now!" while pointing at a 1995 honda civic misses the point. sure, that specific car is cheaper. but the 2025 toyota camry MSRPs at $30K.
Cars got cheaper. The only reason you don't feel it is trade barrier that stops BYD from flooding your local dealers.
> charge 10x the price point > $200/month when cursor charges $20. start with more buffer before the bleeding begins.
What does this even mean? The cheapest Cursor plan is $20, just like Claude Code. And the most expensive Cursor plan is $200, just like Claude Code. So clearly they're at the exact same price point.
> switch from opus ($75/m tokens) to sonnet ($15/m) when things get heavy. optimize with haiku for reading. like aws autoscaling, but for brains.
> they almost certainly built this behavior directly into the model weights, which is a paradigm shift we’ll probably see a lot more of
"I don't know how Claude built their models and I have no insider knowledge, but I have very strong opinions."
> 3. offload processing to user machines
What?
> ten. billion. tokens. that's 12,500 copies of war and peace. in a month.
Unironically quoting data from viberank leaderboard, which is just user-submitted number...
> it's that there is no flat subscription price that works in this new world.
The author doesn't know what throttling is...?
I've stopped reading here. I should've just closed the tab when I saw the first letter in each sentence isn't capitalized. This is so far the most glaring signal of slop. More than the overuse of em-dash and lists.
just fyi, it's a very common manner of writing for younger folk online. more so in informal contexts, but as with everything else, once it's widely adopted it starts to creep into the more formal communication. it's not about "slop", it's just a cultural convention.
i should also note that many languages that got their orthographies defined relatively recently (e.g. various native american languages) use all-lowercase as well, by design. so there's no inherent reason why english can't do that either.
This has been working great for the occasional use, I'd probably top up my account by $10 every few months. I figured the amount of tokens I use is vastly smaller than the packaged plans so it made sense to go with the cheaper, pay-as-you-go approach.
But since I've started dabbling in tooling like Claude Code, hoo-boy those tokens burn _fast_, like really fast. Yesterday I somehow burned through $5 of tokens in the space of about 15 minutes. I mean, sure, the Code tool is vastly different to asking an LLM about a certain topic, but I wasn't expecting such a huge leap, a lot of the token usage is masked from you I guess wrapped up in the ever increasing context + back/forth tool orchestration, but still
If I (and billions others) can be bothered to learn your damn language so we can all communicate, do us a service and actually use it properly, FFS.
To flaunt:
> display (something) ostentatiously, especially in order to provoke envy or admiration or to show defiance.
To flout:
> openly disregard (a rule, law, or convention).
(I'm also a non-native speaker)
The article repeats this throughout but isn't it a straight lie? The plan was named 20x because it's 20x usage limits, it always had enforced 5 hour session limits, it always had (unenforced? soft?) 50 session per month limits.
It was limited, but not enough and very very probably still isn't, judging by my own usage. So I don't think the argument would even suffer from telling the truth.
you shouldn't be pricing compute directly (by charging for tokens yourself)
I am seeing problems with formatting that seemed 'solved' already.
I mean, I have seen "the same" model get better and worse already.
clearly somebody is calibrating the stupidity level relative to energy cost and monetary gain
You definitely want that for some tasks, but for the majority of tasks there is a lot of space for cheap & cheerful (and non-thinking)
> consumers hate metered billing. they'd rather overpay for unlimited than get surprised by a bill.
Yes and no.
Take Amazon. You think your costs are known and WHAMMO surprise bill. Why do you get a surprise bill? Because you cannot say 'Turn shit off at X money per month'. Can't do it. Not an option.
All of these 'Surprise Net 30' offerings are the same. You think you're getting a stable price until GOTAHCA.
Now, metered billing can actually be good, when the user knows exactly where they stand on the metering AND can set maximums so their budget doesn't go over.
Taken realistically, as an AI company, you provide a 'used tokens/total tokens' bar graph, tokens per response, and estimated amount of responses before exceeding.
Again, don't surprise the user. But that's an anathema to companies who want to hide tokens to dollars, the same way gambling companies obfuscate 'corporate bux' to USD.
But for AI in the context of point-solutions and on-the-job use cases, metered billing is a death blow.
In this context, metered is a massive incentive to not use the product and requires the huge friction of having to do a cost/benefit analysis before every task. And if you're using it at work you may even need management sign-off before you can use it again.
For a tool that's intended to amplify productivity, very few humans want to make a cost/benefit analysis 250 times a day whether it's worth $3 to code up a boilerplate or not. On metered billing, they just wont use it.
Standard packages are like insurance. Everyone pays more or less the same premium, but some claim more than others. On average people always overpay for insurance.
The upside is that it's a predictable cost for the users, and also means predictable cash flow for the provider.
You get a surprise bill because of surprise usage of services billed based on usage.
If you ask anyone how much water or electricity they use per month, the first thing they're going to do is look at last month's usage.
Estimating what you need ahead of time is a hard problem.
In fairness, AWS doesn't give you a lot of tools to help you measure and predict how many "units" you'll use outside of running your thing and measuring. On the other hand, running your thing and measuring is the defacto way to figure out how much of something you'll use.
Finally, there are AWS services like EC2 and RDS you can run at a fixed cost to help you stay within budget. Traffic/bandwidth is the only thing that comes to mind that you're pretty much required to use without a way to fix the cost (although you can get pretty close with bandwidth limits on EC2 interfaces)
And, I know this seems dramatic, but besides being cognitively distracting, it also makes me feel sad. Chatroom formatting in published writings is clearly a developing trend at this point, and I love my language so much. Not in a linguistic capacity - I'm not an English expert or anything, nor do I follow every rule - I mean in an emotional capacity.
I'm not trying to be condescending. This is a style choice, not "bad writing" in the typical sense. I realize there is often a lot of low-quality bitterness on both sides about this kind of thing.
Edit:
I also fear that this is exactly the kind of thing where any opinion in opposition to this style will feel like the kind of attack that makes a writer want to push back in a "oh yeah? fuck you" kind of way. I.e. even just my writing this opinion may give an author using the style in question the desire to "double down". Though this conundrum is appropriate (ironic?) - the intensely personal nature of language is part of why I love it.
Does this mean that other languages might offer better information density per token? And does this mean that we could invent a language that’s more efficient for these purposes, and something humans (perhaps only those who want a job as a prompt engineer) could be taught?
Kevin speak good? https://youtu.be/_K-L9uhsBLM?si=t3zuEAmspuvmefwz
In practice, the problem is that any such constructed language wouldn't have a corpus large enough to train on.
It's really unfortunate that we ended up with English as the global lingua franca right at the time generative AI came about, because it is effectively cementing that dominance. Even Chinese models are trained mostly on English AFAIK.
And the interesting property of Lojban is that it has unambiguous grammar that can be syntax-checked by tools and enforced by schemas, and machine-translated back to English. I experimented with it a bit and found that large SOTA models can generate reasonably accurate translations if you give them tools like dictionary and parser and tell them to iterate until they get a syntactically valid translation that parses into what they meant to say. So perhaps there is a way to generate a large enough dataset to train a model on; I wish I had enough $$$ to try this on a lark.
I don't agree with the Cognition conclusion either. Enterprises are fighting super hard to not have a long term buying contract when they know SOTA (app or model) is different every 6 months. They are keeping their switching costs low and making sure they own the workflow, not the tool. This is even more prominent after Slack restricted API usage for enterprise customers.
Making money on the infra is possible, but that again misunderstands the pricing power of Anthropic. Lovable, Replit etc. work because of Claude. Openai had codex, google had jules, both aren't as good in terms of taste compared to Claude. It's not the cli form factor which people love, it's the outcome they like. When Anthropic sees the money being left on the table in infra play, they will offer the same (at presumably better rates given Amazon is an investor) and likely repeat this strategy. Abstraction is a good play, only if you abstract it to the maximum possible levels.
> when a new model is released as the SOTA, 99% of the demand immediately shifts over to it
99% is in the wrong ballpark. Lots of users use Sonnet 4 over Opus 4, despite Opus being 'more' SOTA. Lots of users use 4o over o3 or Gemini over Claude. In fact it's never been a closer race on who is the 'best': https://openrouter.ai/rankings
>switch from opus ($75/m tokens) to sonnet ($15/m) when things get heavy. optimize with haiku for reading. like aws autoscaling, but for brains.
they almost certainly built this behavior directly into the model weights
???
Overall the article seems to argue that companies are running into issues with usage-based pricing due to consumers not accepting or being used to usage based pricing and it's difficult to be the first person to crack and switch to usage based.
I don't think it's as big of an issue as the author makes it out to be. We've seen this play out before in cloud hosting.
- Lots of consumers are OK with a flat fee per month and using an inferior model. 4o is objectively inferior to o3 but millions of people use it (or don't know any better). The free ChatGPT is even worse than 4o and the vast majority of chatgpt visitors use it!
- Heavy users or businesses consume via API and usage based pricing (see cloud). This is almost certainly profitable.
- Fundamentally most of these startups are B2B, not B2C
How much of that is the naming?
Personally I just avoid OpenAIs models entirely because I have absolutely no way of telling how their products stack up against one another or which to use for what. In what world does o3 sort higher than 4o?
If I have to research your products by name to determine what to use for something that is already a commodity, you've already lost and are ruled out.
The meaningful frontier isn't scalar on just the capability, it's on capability for a given cost. The highest capability models are not where 99% of the demand is on. Actually the opposite.
To get an idea of what point on the frontier people prefer, have a look at the OpenRouter statistics (https://openrouter.ai/rankings). Claude Opus 4 has about 1% of their total usage, not 99%. Claude Sonnet 4 is the single most popular model at about 18%. The runners up in volume are Gemini Flash 2.0 and 2.5, which are in turn significantly cheaper than Sonnet 4.