Two issues with this. One, it's profitable assuming you just keep serving the same model forever, which is not realistic in this market. A given model has a shelf-life, which these days is measured in months, not years. Which means that trying to separate the cost of training the model from the cost of serving it doesn't make much business sense. And two, for providers that provide inference only via open weight models, the margins quickly move to commoditization. The "someday" when frontier model providers can enjoy their current high inference margins without the burden of significant training costs is never going to arrive.
>Commoditization means there's price competition. From a consumer perspective, that's good. You want it to be a low-margin, high volume, competitive business.
You and I may want it to be a low-margin, high-volume business. But the valuations of OpenAI, Anthropic, and much of the rest of the AI industry are not based on that assumption. They are based on the assumption that there will be a couple of winners, like in the smartphone wars, and that those winners will be able to maintain good margins.
There are low-end and premium smartphones, with different profit margins. Similarly, it seems like open weights models and high-end models could co-exist?
>Similarly, it seems like open weights models and high-end models could co-exist?
Low-end and premium smartphones are still overwhelmingly either Android or Apple. And I'm sure open weights and premium models will co-exist. But that's not the future the current valuations are predicated on. I agree there's no way to say where the market structure will end, but I think we know enough to say where it won't end up. And some of the supporting players are starting to see the handwriting on the wall:
> A given model has a shelf-life, which these days is measured in months, not years.
Not all new models are trained from scratch. ChatGPT 5.3 to 5.4 (and likely 5.5) was basically the same model, but probably trained a bit more, not a new model from scratch.
> The "someday" when frontier model providers can enjoy their current high inference margins without the burden of significant training costs is never going to arrive.
That is debatable. I believe the moat for the frontier model providers is the compute. At the level of 10 trillion parameters (that Fable/Mythos are rumored to have), you need serious compute to serve inference, and you also need serious compute to train. Will DeepSeek, Qwen, Kimi, GLM come up with a 10T new model anytime soon? I doubt that. People keep saying that the Chinese labs are catching up to the US big 3, and measured in months the gap is now only 4-6 months. I doubt a Chinese version of Fable/Mythos will be released in the next 12 months.
>ChatGPT 5.3 to 5.4 (and likely 5.5) was basically the same model, but probably trained a bit more, not a new model from scratch.
Then those models have an eroding moat and will be quickly driven down to commodity pricing. The only thing propping up inference margins are the cap-ex costs of training. That's the moat. That's why there's no way to win this game. You cannot have low training/infrastructure cost and high margins (such as would justify today's valuations).
>I doubt a Chinese version of Fable/Mythos will be released in the next 12 months.
Profitable on a per token basis is meaningless. Ed Zitron doesn’t argue that it is impossible to offer profitable inference, he argues that the business as it stands today is deeply unprofitable and only getting worse because it isn’t a high-margin inference business.
Play out the most likely pessimist’s scenario: LLMs are useful but frontier models are overkill so businesses just use dirt cheap open weight models on their own hardware and/or they rent hardware instead of paying per token. Then what for OpenAI and Anthropic?
OpenAI’s business collapses if customers are happy with an LLM that costs $0.10 per million tokens even if it only costs OpenAI $0.05 in inference per million tokens. The insane bonkers claim from Garry Tan that in 2 years we will be using 90,000x as many tokens as today is… well, obviously not true.
The fixed costs that OpenAI and Anthropic have created need inference demand far beyond what is plausible.
How is this sustainable in the long run tho? Major enterprises are already making the headlines as they cut licenses and reduce token consumptions. The conversation is moving a lot toward return on token... is automating this activity with an agent giving me a higher return rather than having a human run it? Plus the advent of local AI and open source models are creating new opportunities to reduce token costs
13 comments
[ 3.0 ms ] story [ 53.3 ms ] threadTOO BAD YOU IDIOTS CAN'T FIGURE OUT HOW TO BAN PEOPLE. RETARDS. LEARN TO CODE.
The smaller models will only get better which push out the usefulness of older gpus.
You and I may want it to be a low-margin, high-volume business. But the valuations of OpenAI, Anthropic, and much of the rest of the AI industry are not based on that assumption. They are based on the assumption that there will be a couple of winners, like in the smartphone wars, and that those winners will be able to maintain good margins.
Low-end and premium smartphones are still overwhelmingly either Android or Apple. And I'm sure open weights and premium models will co-exist. But that's not the future the current valuations are predicated on. I agree there's no way to say where the market structure will end, but I think we know enough to say where it won't end up. And some of the supporting players are starting to see the handwriting on the wall:
https://www.reuters.com/business/blackstones-qts-terminates-...
Not all new models are trained from scratch. ChatGPT 5.3 to 5.4 (and likely 5.5) was basically the same model, but probably trained a bit more, not a new model from scratch.
> The "someday" when frontier model providers can enjoy their current high inference margins without the burden of significant training costs is never going to arrive.
That is debatable. I believe the moat for the frontier model providers is the compute. At the level of 10 trillion parameters (that Fable/Mythos are rumored to have), you need serious compute to serve inference, and you also need serious compute to train. Will DeepSeek, Qwen, Kimi, GLM come up with a 10T new model anytime soon? I doubt that. People keep saying that the Chinese labs are catching up to the US big 3, and measured in months the gap is now only 4-6 months. I doubt a Chinese version of Fable/Mythos will be released in the next 12 months.
Then those models have an eroding moat and will be quickly driven down to commodity pricing. The only thing propping up inference margins are the cap-ex costs of training. That's the moat. That's why there's no way to win this game. You cannot have low training/infrastructure cost and high margins (such as would justify today's valuations).
>I doubt a Chinese version of Fable/Mythos will be released in the next 12 months.
I would take that bet.
Play out the most likely pessimist’s scenario: LLMs are useful but frontier models are overkill so businesses just use dirt cheap open weight models on their own hardware and/or they rent hardware instead of paying per token. Then what for OpenAI and Anthropic?
OpenAI’s business collapses if customers are happy with an LLM that costs $0.10 per million tokens even if it only costs OpenAI $0.05 in inference per million tokens. The insane bonkers claim from Garry Tan that in 2 years we will be using 90,000x as many tokens as today is… well, obviously not true.
The fixed costs that OpenAI and Anthropic have created need inference demand far beyond what is plausible.