Super awesome. Wish they would release the paper about what they did to achieve this. I remember nous released the token superposition paper which improved pretraining FLOPs some, but not 50x: https://nousresearch.com/token-superposition. Wondering if they also found some cool tokenization strategiesa
If you're like me, a SWE who is curious about ML/LLM training but unfamiliar with the terms, I got an agent to explain to me how to read the charts.
Basically, you can think of a LLM as a function which generates a probability distribution of words. If the next word in a series is "they", and one model predicts that word 40% of the time, and another model predicts that word 1% of the time, the latter model is worse as it is more surprised by the true distribution.
You can convert these probabilities into "bits":
surprise in bits = −log₂(probability of the actual token)
Probability of actual token,Surprise
1,0 bits
1/2,1 bit
1/8,3 bits
1/1024,10 bits
This is then normalised by text length:
Bits per byte = total next-token surprise in bits / number of bytes in the evaluated text
So the lower you go on the charts, the less surprises in the LLMs distribution (a better model).
Cost will keep dropping, but the frontier will keep getting pushed. The whole "give me today's model 10x cheaper and I'm good" line is a fallacy. It isn't true now and it never will be for the top 1% of tasks, which will create the most economic gains.
There are an enormous number of tasks that can get by on good enough.
If you need image recognition, and a 30B model saturates the use case with 100% accuracy, you absolutely wouldn't continue to use the next frontier model as they come out.
And I'd argue most economically meaningful tasks will be saturated by cheaper models than those requiring frontier.
Think about what today's models can do with pretty close to 100% accuracy, and then consider that they will be orders of magnitudes cheaper over the years.
5.6 Sol can already obviate tons of labor, and why would you pay 2x or more for no meaningful gain?
The relative gap between frontier and non frontier also continues to shrink, so it's not like you take a meaningful performance loss by rewinding to models from 3-6 months ago.
That can all be true but the frontier models will still have a huge market. You're thinking of all tasks as a fixed pie. The top 1% of intelligence opens up a whole new pie, stuff nobody does today because it's too expensive: daily cancer scans instead of one every few years, asteroid mining missions that need ten thousand PhD-hours of planning, custom drugs designed for your specific tumor, a personal lawyer and doctor for every person on earth, auditing every line of code in every bank and hospital continuously and so on.
> daily cancer scans instead of one every few years
Silly nitpick: the reason we don’t do daily cancer scans isn’t the cost, it’s the false positive rate. Invasive procedures like biopsies come with complications like infections that happen at a higher rate and do more damage than the cancer that doesn’t even exist. This dilemma is pervasive in medicine, because our tests aren’t perfect but the thing they’re testing for is rare.
The whole paradigm changes, though, when you can do daily cancer scans. You don't get a biopsy when the scan shows a lump. You get a biopsy after a couple weeks of daily scans showing the lump growing. Plus, having all the data from the daily scans improves your testing accuracy so false positives and negatives are more rare.
The errors are correlated, not random. Lumps are usually benign cysts. If you start cutting people open for every cyst on a scan, you’d kill a lot more people than you’d save.
This isn’t something you can solve with more scans because the tests test for data that is indistinguishable. They look the same on a scan, there’s an overlap in the assay with some random protein with the same binding sites that is only present in 1% of the population, the coding gene in one person gets repeated in a noncoding region in another, and so on. The “more data” that works is a doctor applying professional judgement (which they’re also famously bad at because biology is a fickle mistress).
More data (daily scans) can mean we get better at medicine, though, and more accurate. You're assuming "daily cancer scans" look just like they do today, rather than eg unobtrusive devices in our daily lives that measure changes over time.
The frequent "muh false positives" comment we hear from doctors appears to be a lack of imagination?
They “can mean” that, but they don’t. Empirically.
For example: you can get an unobtrusive device to measure your blood sugar, like a diabetic. Result: a bunch of hypochondriacs flooding doctors with spurious blood sugar readings that don’t mean anything because spoiler alert the research is entirely on actual diabetics, not healthy people. Repeat this kafkaesque pattern all over medicine, and you get to where we are today.
If you're doing non-redundant tests and your uncertainty bars aren't shrinking, it's usually a skill issue.
If it looks like a duck, it might be a duck - or a painting of one. If it looks like a duck, swims like a duck, and quacks like a duck? The joint duck estimation is much more confident now. There might be a few more observational tests one should administer before committing to a duck decision, but each tests pins down variables and rejects confounders. Uncertainties are cut down, and we get closer to crossing the threshold between "informative" and "actionable".
Thus, it's often worth it to improve observability. If you managed to make a certain test more reliable, or cheaper to administer, or reduced the chance of adverse effects? Or, in other words, improved SNR, reduced costs, and reduced costs? You can get more information for your buck. Paired with good knowledge: you can make better decisions more easily.
The fact that the thought of "having more information might be bad actually" even occurs in the field of medicine shows just how far it is from being optimal. Having more information isn't always beneficial - some information is genuinely redundant. Some information is not worth the effort of gathering and integrating it. But if you get more information and it results in worse outcomes? You're doing something wrong.
> If you're doing non-redundant tests and your uncertainty bars aren't shrinking, it's usually a skill issue.
They’re not non-redundant. It’s not a “skill issue” (wtf? How can an assay have a skill issue?)
Biology is messy, and all of the assumptions in your comment are dead wrong (empirically). More perfect data is good, but we live in the real world not some made up fantasy land.
I think there's an intelligence limit, or at least asymptote. It may be above human intelligence, but I don't think it's miles above it (at least not the kind of intelligence humans can create, recognize, or use). For example in Go, most estimates place God or "perfect play" three ranks above top professionals[0]. In the latest human-AI Go match, the human got a 2 stone handicap. So it's not like we have a lot more frontier to push there.
this assumes our whole universe and what we can do in it is a finite go board. maybe it is. but we are no where close to exploring even a fraction of it. lots of things need to happen before any limit is reached. the cavemen would also probably think we saturated tools once they saw bows and arrows.
Intelligence is spiky. In some things humans may play near the limit (Go possibly), but if you look at parts of mathematics like addition, humans can add in their head just fine but its a few trillion times faster to use a computer on addition problems of any size.
And that's not even really touching societal/network intelligence. A single human isn't that smart and can't accomplish that much. Hence we form families, and companies, and societies, and governments. What does a society of AIs look like?
Note that addition that's "trillion times faster" is purely an optimization, not greater intelligence.
And a society of humans does not increase our overall intelligence. It allows all of humanity to access the accomplishments of our most intelligent members throughout history (which is huge, don't get me wrong!) but it seems almost self-evident that our civilization as a whole is not smarter than Newton/Einstein/von Neumann/etc.
Intelligence is also access to data, along with all of it's other requirements. Society as a whole is far smarter than Einstein.
If I dump Einstein on an island at 5 you don't get a theory of relativity. The data provided by a civilization is required, hence why people 10,000 years ago didn't go to the stars. It is unlikely they were significantly less intelligent than us. They just didn't have the language of data we do now.
Just FTR, Shin Jinseo's result in that latest match is seen as extraordinary. In the last few years it's been common for top pros to play bots on 3 or even 4 stones.
The thing about playing strength in games like go is that it doesn't map at all linearly to results. It could very well be the case that the bots are only a few points of handicap away from perfect play, but thousands of points of ELO.
Intuitively I agree with you, but you've cited evidence that doesn't actually prove what you said. Yes, openrouter users choose cheaper open weights. If you're on openrouter, you're already shopping around. We'd need stats for claude, codex, gemini, grok (yuck), AND openrouter, to determine what people in general (not just the subset of people who decided to shop around) are actually doing.
> It isn't true now and it never will be for the top 1% of tasks, which will create the most economic gains
Vulcan Materials Company has produced some of the strongest and most resilient economic gains for investors at around 25% gross margins for 50 years.
Vulcan’s business is crushing rocks, and then driving those rocks to where people need crushed rocks.
I mention it because Vulcan isn’t a sophisticated business at its core, but its economic returns are exceptional, because they do their unsophisticated work exceptionally well, and exceptionally efficiently.
Across the economy, most economic gains are created by companies like Vulcan, who do boring repetitive work exceptionally well and efficiently.
I expect this to hold true into the era of AI. Most stuff probably doesn’t need an exceptional model, and paying for an exceptional model to do unsophisticated work will leave your business vulnerable to competitors who take time to find the most efficient model for the task, and undercut you.
I just wonder whether the analogy holds from atom based businesses to byte based businesses. At the end of the day, there are pretty hard constraints on how much a competitor could undercut Vulcan - they'd need to be better a) crushing rocks and b) driving those rocks to where people need it. Something about that, while simple, seems really nonfungible to do well. The business may traffic in commodities, but the service and quality it provides ironically seems to be not really a commodity.
Conversely, AI models feel pretty different. People will swap models at the drop of a hat and while not perhaps fully fungible, the cost of switching can be as low as one engineers it to be. In this case, the service does really seem to feel more like a commodity.
Not sure where this leads or how this ends. Perhaps those two will swap poles for me (IE as robotic automation improves, there are more commoditized Vulcans, and as metered intelligence improves, they become less fungible). I just can't quite see how yet. Just food for thought.
I agree, but I also think as AI gets better we're going to see Jevons paradox in full swing, which might delay lower costs. We've seen this with Astra according to Tibo: https://x.com/thsottiaux/status/2097559315150426222
I don't know enough about the specific models they're comparing against to say this definitively, but it looks to me like they're comparing their pre-trained models with others' post-trained models.
The metric upon which their 10x claim is based (bits-per-byte) is exactly the metric which is optimized during pre-training. Post-trained models are fine-tuned to optimize other metrics, which is known to be detrimental to performance on bits-per-byte evaluations. So bits-per-byte evaluations will always make a pre-trained model look favorable in comparison to a comparable model which has also undergone post-training.
Can someone confirm whether the models they are comparing against (DeepSeek V4, Kimi K2, and Nemotron 3 Ultra) have been post-trained?
> “Base models” are pretrained models that have not yet undergone reinforcement learning, SFT, or other post-training. They are highly sensitive to prompting, making sampling-based evals unreliable. Instead, we measured bits-per-byte loss on heldout data, which does not suffer from prompt sensitivity and smooths measurement of otherwise emergent abilities. As a side note, we were surprised that Nemotron 3 outperforms DeepSeek V4 Pro across the board but found this to be consistent across domains and inference engines. This might indicate that Nemotron’s weak performance on benchmarks after RL is due to weaker post-training, but the pretrain was ahead of Chinese open-weight competitors.
this is basically the only thing pre-training teams work on in labs. compute efficiency is the metric, the assumption that scaling = intelligence is considered a given.
This is great. All algorithmic efficiencies are amazing!
One thing I'd remind all scientists and the wonderful people here is this wonderful meme/line from Jurassic Park: "Your scientists were so preoccupied with whether they could they didn't stop to think if they should."
What is the actual amount of data that needs to be pre-trained and what is not? Nobody has come up with great answers to this question, and I'm already seeing amazing 0.5b-2b parameter models working very well with n-Gram corpuses of data.
Imagine yourself the CEO of a big AI company. It takes about a month to develop and train a model, so you release a new model every month. A startup says they can 10x your efficiency. What does that get you? You can't release a new model every three days. You can't 10x R&D either. You definitely can't tell investors that you are growing at the same rate, but selling off assets and cancelling purchasing contracts. So you just never improve efficiency enough to use less energy than the previous model version.
This seems weirdly pessemistic: frontier labs have much stronger pretraining than most open weights models
And reading the release it feels very obvious this is also a ton of aligning their data mix with coding and science: we don't know that this model doesn't have terrible world knowledge or is ruined for anything related to subjective preference
They also repeatedly mention knowledge almost as if they saw that skepticism coming, but then limit knowledge to topics where more understanding of how code/scientific writing looks would produce the same graph as having actual world knowledge maintained.
That's not nefarious (they literally build coding models), but it also means the resulting model isn't necessarily competitive with a frontier model in a broader way.
This feels like the inverse approach to what Thinking Machines did with Inkling (trying to train as "un-spikey" a base model as possible)
the obvious answer is that you can try more experiments over the same period of time, so you find more improvements per month, and the rate of improvement of the models you release increases
A 10x reduction in pre-training means a 10x faster feedback loop. I'm pretty sure any lab would sign up for that. You can start experimenting on different approaches much more aggressively.
48 comments
[ 0.32 ms ] story [ 5.6 ms ] threadIf this holds up that's a really big deal.
Going to be interesting to see what happens to discoveries like this in the future.
Basically, you can think of a LLM as a function which generates a probability distribution of words. If the next word in a series is "they", and one model predicts that word 40% of the time, and another model predicts that word 1% of the time, the latter model is worse as it is more surprised by the true distribution.
You can convert these probabilities into "bits":
surprise in bits = −log₂(probability of the actual token)
This is then normalised by text length:Bits per byte = total next-token surprise in bits / number of bytes in the evaluated text
So the lower you go on the charts, the less surprises in the LLMs distribution (a better model).
For more info: https://smalldocs.org/s/DfvdGuFsiR3LlzXw1H5J0K#k=AJ8V1AQECYj...
I think cost will decrease forever.
If you need image recognition, and a 30B model saturates the use case with 100% accuracy, you absolutely wouldn't continue to use the next frontier model as they come out.
And I'd argue most economically meaningful tasks will be saturated by cheaper models than those requiring frontier.
Think about what today's models can do with pretty close to 100% accuracy, and then consider that they will be orders of magnitudes cheaper over the years.
5.6 Sol can already obviate tons of labor, and why would you pay 2x or more for no meaningful gain?
The relative gap between frontier and non frontier also continues to shrink, so it's not like you take a meaningful performance loss by rewinding to models from 3-6 months ago.
Silly nitpick: the reason we don’t do daily cancer scans isn’t the cost, it’s the false positive rate. Invasive procedures like biopsies come with complications like infections that happen at a higher rate and do more damage than the cancer that doesn’t even exist. This dilemma is pervasive in medicine, because our tests aren’t perfect but the thing they’re testing for is rare.
This isn’t something you can solve with more scans because the tests test for data that is indistinguishable. They look the same on a scan, there’s an overlap in the assay with some random protein with the same binding sites that is only present in 1% of the population, the coding gene in one person gets repeated in a noncoding region in another, and so on. The “more data” that works is a doctor applying professional judgement (which they’re also famously bad at because biology is a fickle mistress).
The frequent "muh false positives" comment we hear from doctors appears to be a lack of imagination?
For example: you can get an unobtrusive device to measure your blood sugar, like a diabetic. Result: a bunch of hypochondriacs flooding doctors with spurious blood sugar readings that don’t mean anything because spoiler alert the research is entirely on actual diabetics, not healthy people. Repeat this kafkaesque pattern all over medicine, and you get to where we are today.
If it looks like a duck, it might be a duck - or a painting of one. If it looks like a duck, swims like a duck, and quacks like a duck? The joint duck estimation is much more confident now. There might be a few more observational tests one should administer before committing to a duck decision, but each tests pins down variables and rejects confounders. Uncertainties are cut down, and we get closer to crossing the threshold between "informative" and "actionable".
Thus, it's often worth it to improve observability. If you managed to make a certain test more reliable, or cheaper to administer, or reduced the chance of adverse effects? Or, in other words, improved SNR, reduced costs, and reduced costs? You can get more information for your buck. Paired with good knowledge: you can make better decisions more easily.
The fact that the thought of "having more information might be bad actually" even occurs in the field of medicine shows just how far it is from being optimal. Having more information isn't always beneficial - some information is genuinely redundant. Some information is not worth the effort of gathering and integrating it. But if you get more information and it results in worse outcomes? You're doing something wrong.
They’re not non-redundant. It’s not a “skill issue” (wtf? How can an assay have a skill issue?)
Biology is messy, and all of the assumptions in your comment are dead wrong (empirically). More perfect data is good, but we live in the real world not some made up fantasy land.
[0]https://senseis.xmp.net/?HandOfGod
And that's not even really touching societal/network intelligence. A single human isn't that smart and can't accomplish that much. Hence we form families, and companies, and societies, and governments. What does a society of AIs look like?
And a society of humans does not increase our overall intelligence. It allows all of humanity to access the accomplishments of our most intelligent members throughout history (which is huge, don't get me wrong!) but it seems almost self-evident that our civilization as a whole is not smarter than Newton/Einstein/von Neumann/etc.
If I dump Einstein on an island at 5 you don't get a theory of relativity. The data provided by a civilization is required, hence why people 10,000 years ago didn't go to the stars. It is unlikely they were significantly less intelligent than us. They just didn't have the language of data we do now.
The thing about playing strength in games like go is that it doesn't map at all linearly to results. It could very well be the case that the bots are only a few points of handicap away from perfect play, but thousands of points of ELO.
I wrote about all of this on Stack Exchange, along with some napkin math, a couple years back: https://boardgames.stackexchange.com/a/61058
https://analyticsindiamag.com/ai-features/almost-nobody-is-u...
Vulcan Materials Company has produced some of the strongest and most resilient economic gains for investors at around 25% gross margins for 50 years.
Vulcan’s business is crushing rocks, and then driving those rocks to where people need crushed rocks.
I mention it because Vulcan isn’t a sophisticated business at its core, but its economic returns are exceptional, because they do their unsophisticated work exceptionally well, and exceptionally efficiently.
Across the economy, most economic gains are created by companies like Vulcan, who do boring repetitive work exceptionally well and efficiently.
I expect this to hold true into the era of AI. Most stuff probably doesn’t need an exceptional model, and paying for an exceptional model to do unsophisticated work will leave your business vulnerable to competitors who take time to find the most efficient model for the task, and undercut you.
Conversely, AI models feel pretty different. People will swap models at the drop of a hat and while not perhaps fully fungible, the cost of switching can be as low as one engineers it to be. In this case, the service does really seem to feel more like a commodity.
Not sure where this leads or how this ends. Perhaps those two will swap poles for me (IE as robotic automation improves, there are more commoditized Vulcans, and as metered intelligence improves, they become less fungible). I just can't quite see how yet. Just food for thought.
The metric upon which their 10x claim is based (bits-per-byte) is exactly the metric which is optimized during pre-training. Post-trained models are fine-tuned to optimize other metrics, which is known to be detrimental to performance on bits-per-byte evaluations. So bits-per-byte evaluations will always make a pre-trained model look favorable in comparison to a comparable model which has also undergone post-training.
Can someone confirm whether the models they are comparing against (DeepSeek V4, Kimi K2, and Nemotron 3 Ultra) have been post-trained?
> “Base models” are pretrained models that have not yet undergone reinforcement learning, SFT, or other post-training. They are highly sensitive to prompting, making sampling-based evals unreliable. Instead, we measured bits-per-byte loss on heldout data, which does not suffer from prompt sensitivity and smooths measurement of otherwise emergent abilities. As a side note, we were surprised that Nemotron 3 outperforms DeepSeek V4 Pro across the board but found this to be consistent across domains and inference engines. This might indicate that Nemotron’s weak performance on benchmarks after RL is due to weaker post-training, but the pretrain was ahead of Chinese open-weight competitors.
One thing I'd remind all scientists and the wonderful people here is this wonderful meme/line from Jurassic Park: "Your scientists were so preoccupied with whether they could they didn't stop to think if they should."
What is the actual amount of data that needs to be pre-trained and what is not? Nobody has come up with great answers to this question, and I'm already seeing amazing 0.5b-2b parameter models working very well with n-Gram corpuses of data.
I don't believe this is actually happening.
And reading the release it feels very obvious this is also a ton of aligning their data mix with coding and science: we don't know that this model doesn't have terrible world knowledge or is ruined for anything related to subjective preference
They also repeatedly mention knowledge almost as if they saw that skepticism coming, but then limit knowledge to topics where more understanding of how code/scientific writing looks would produce the same graph as having actual world knowledge maintained.
That's not nefarious (they literally build coding models), but it also means the resulting model isn't necessarily competitive with a frontier model in a broader way.
This feels like the inverse approach to what Thinking Machines did with Inkling (trying to train as "un-spikey" a base model as possible)
[1]: https://magic.dev/blog/100m-token-context-windows (also linked to in their blogpost)