We already knew that kind of stuff from AlphaZero vs AlphaGo
AlphaGo Zero is a version of DeepMind's Go software AlphaGo. AlphaGo's team published an article in Nature in October 2017 introducing AlphaGo Zero, a version created without using data from human games, and stronger than any previous version.[1] By playing games against itself, AlphaGo Zero: surpassed the strength of AlphaGo Lee in three days by winning 100 games to 0; reached the level of AlphaGo Master in 21 days; and exceeded all previous versions in 40 days.[2]
Training artificial intelligence (AI) without datasets derived from human experts has significant implications for the development of AI with superhuman skills, as expert data is "often expensive, unreliable, or simply unavailable."[3] Demis Hassabis, the co-founder and CEO of DeepMind, said that AlphaGo Zero was so powerful because it was "no longer constrained by the limits of human knowledge".[4] Furthermore, AlphaGo Zero performed better than standard deep reinforcement learning models (such as Deep Q-Network implementations[5]) due to its integration of Monte Carlo tree search. David Silver, one of the first authors of DeepMind's papers published in Nature on AlphaGo, said that it is possible to have generalized AI algorithms by removing the need to learn from humans.[6]
Google later developed AlphaZero, a generalized version of AlphaGo Zero that could play chess and Shōgi in addition to Go.[7] In December 2017, AlphaZero beat the 3-day version of AlphaGo Zero by winning 60 games to 40, and with 8 hours of training it outperformed AlphaGo Lee on an Elo scale. AlphaZero also defeated a top chess program (Stockfish) and a top Shōgi program (Elmo).[8][9]
It’s great, but it only works for problems where there is exactly one correct solution and it’s possible to automatically verify the solution - like math and programming. So far these reasoning models have not shown much transfer learning of reasoning to other domains, and are often worse at non-math/code tasks than standard models.
Not sure if this a trivial or naive thought - is that perhaps because in non-discrete ideas there is more granularity of information encoded compared to a numerical solution?
Separatelt, but on a related note - do we need analog or quantum computing to "truly" scale?
To ”truly” scale we need a system directly doing calculation on some fundamental property of matter. Quantum computing is _close_ but suffers from a lack of interpretability: how would we know if our quantum simulation of the universe would include the quantum computer doing the simulation, as well as another recursive copy of the simulation, and another, and another…
We can probably make it work at more nebulous problems by using a big LLM to judge the quality of the answers as well. It should be easier to recognise e.g. a really good poetic translation than to make one, and as long as that's true it could benefit from internal monologue "reasoning" in those domains as well.
This has been done, and doesn't work especially well. See, for example, GANs, which are difficult to train and vulnerable to mode collapse.
Not saying that it's impossible, but reinforcement learning of the form shown by DeepSeek was particularly well-established and robust. It's sort of ironic, actually...IIRC, OpenAI started in the world of this kind of reinforcement learning.
LLMs can verify math problems if you give it a calculator tool. Coding problems if you give it a Linux environment.
So why can’t we give LLMs other virtual environments to verify if the solution is correct? For example, a stock simulator, a physics simulator, a driving simulator, etc.
Because simulations are not perfect models. A particle physics simulation simply cannot be accurate enough to provide the 5-sigma (99.99994%) confidence typically required in particle physics. Not even measurement is simple when the requirements are so strict, that’s why they spend billions building those huge accelerators.
A physics simulator's rules will be derivative of known physics. If we ask it to push the boundaries of known physics, then we can't verify it without real-world experiment. Absent that then it's basically Chegg on steroids: "Tell me, based on what we believe about how the universe works, about X." is the implicit preface to every physics question.
To the extent that someone can figure out a way out of this epistemic box, I'm interested.
If the current hubbub around DeepSeen is really because they ”created their model” with like $5M when previously ”creating a model” cost $500B, it is rather obvious that ”creating the model” with just $30 implies the meanings of the three ”creating a model” expressions are highly divergent.
They aren’t though. $5M is the cost of a single training run. $500B includes the cost of operations, data center, a lot more failed runs because they weren’t sure that they’d were on the right path etc.
Right. It's like building a large model rocket and saying that you've cracked rocketry for a fraction of the cost that was required in the 1940's and 1950's.
Well yes, yes you did, because all you had to do was follow the existing instructions, guidelines, and use easily available materials. You didn't go down any dead ends, didn't have to work your way from propellants like high test peroxide, dangerous hypergolics, and eventually develop solid rocket boosters.
It's like making the generic of a drug someone else developed.
Probably just that it's not as impressive as it appears because it didn't innovate. Which is of course irrelevant since the innovative leap here were the optimizations that let them make their model with an order of magnitude fewer materials, regardless of whatever innovation costs OpenAI ate.
No, just that what DeepSeek did is not as valuable as what the first movers did, because it did not advance the state of the art nearly as much. It's a new cheaper way to go from Base LM -> CoT "reasoning" LM. We already had CoT "reasoning" LMs, so while the new cheaper path to get to them is interesting, it's not necessarily groundbreaking either. Also, R1 only works with the "cold start" data that they distilled from o1, so it's not quite clear that it'll ever be able to exceed o1's capabilities. We already know it's much cheaper to distill new, smaller models from large already pretained and well-performing models—in fact, $5M sounds like a very expensive way to do so. So while these new techniques are probably going to have some impact, OpenAI is far from quaking in their boots
Um it’s very valuable, maybe even more valuable… Companies can now have a private LLM with o1 quality without having to send data to OpenAI or pay for their API.
I don’t think anyone is claiming that Deepseek didn’t produce a very impressive frontier model. They’re just saying it’s not surprising that, in your analogy, flipping the single bit was cheaper than the prior work.
DeepSeek never claimed to have trained the base models they used. Now maybe a lot of people inferred that they did, but that's not what they claimed.
Their breakthrough was more along the lines of, "given a model, we can train it to reason using a certain class of RL techniques". Which is, to my mind, more useful in any case.
But yeah, if there are people out there thinking they can train base models from scratch for USD6 Million with no data, they're likely to be disagreeably surprised when they make the attempt.
> DeepSeek never claimed to have trained the base models they used
Isn't that exactly what they have done? Maybe you are confused with the distilled models?
DeepSeek-R1: "DeepSeek-R1-Zero & DeepSeek-R1 are trained based on DeepSeek-V3-Base"
DeepSeek-V3: "At an economical cost of only 2.664M H800 GPU hours, we complete the pre-training of DeepSeek-V3 on 14.8T tokens, producing the currently strongest open-source base model. The subsequent training stages after pre-training require only 0.1M GPU hours"
Aren't they based on qwen? I thought the clever bit with DeepSeek is a way to fine tune with reinforcement learning, not training a huge model from scratch.
They developed their “zero” model, which (they claim) is a from-scratch model. Base models are typically not fine-tuned for particular applications (eg chat).
They trained their Zero model into a Chat+Reasoning model, which is what is attracting news.
They ALSO fine-tuned small Qwen models using their big models as a teacher (distillation technique).
deepseek-r1 (reasoning) is a post-trained (not simply finetuned) deepseek-r3 (base model) by deepseek. qwen is a completely separate 3rd party model by alibaba. deepseek distilled (finetuned on the outputs of) r1 into qwen. They also did the same with llama.
https://newsletter.languagemodels.co/p/the-illustrated-deeps... is a really fascinating overview of what those new steps were! The innovation here seems to be that the creation of a cutting-edge reasoning model could be made from a relatively inexpensive base model, regardless of how much RL is needed. Part of this seems to be a novel methodology for generating large volumes of chain-of-thought training data; it's left ambiguous exactly how the bulk of this data was generated, though, and it could have been a long process of manual curation.
I'd also take the assertion that they started R1 with V3-Base in its release form with a grain of salt. We'll see whether people can reproduce similar results. Either way, there's a new chain-of-thought model with open weights, and that's an incredible achievement regardless of the context - and one that will be meaningful to future research.
So far it is literally 3 different uses of the phrase “creating a model”. I could further obfuscate it by fine tuning the work of these researchers on a single step and claimed I “created a model” for $0.05. :(
I feel like that’s what everyone who is naming their model “R1” is doing right now…
This article is trying to make the headline sound as if they replicated the model, when they “just” replicated the theory of one of their findings. It’s a great result even without the author’s obfuscation.
These models are highly specialized models that reason similarly to DeepSeek's new model except instead of reasoning over the general domain they do something specific like arithmetic.
They're using a similar model and a similar training method to do something simple. It is to DeepSeek what DeepSeek is to ChatGPT-4o, and this kind of reduction to the simplest case is exactly what makes progress go.
(e.g. ChatGPT-4 is a dead end because it is so expensive to train that you can't do training experiments. At $30 a model it is reasonable to train a model 1000s of times)
But how they built on other open concepts like Cold Start + RL + Rejection Sampling etc... is the big thing.
I am not at all connected to whomever this is, but as I am on a "confidential project for a confidential client" it seems to simplify what I would overcomplicate anyway.
Some of us need to improve domain specificity, and are more interested in targeted capabilities.
DeepSeek-R1-Distill-Qwen-32B itself is pretty good to be honest, but it is more aspirational how easy it is to add reasoning to any base model IMHO.
Those of us who remember Simon’s satisficing principle, realize that often complex problems sub-optimal solutions.
We don't need, want or expect some central omniscient and omnipotent AGI, we want tools that are sufficient to solve real world problems.
Obviously having good reliable domain specific CoT for cold start etc... is hard.
But not nearly as hard as hitching the companies success to MS/OpenAI.
While there is a lot of noise...I think some people are missing the forrest for the trees here. Deepseek is more about moving the decision to use OpenAI for some needs from a single source vendor to one of convenience for us.
For those of us who are in the camp that OpenAI was never going to reach their "AGI" claims because of fundamental issues like the frame problem, this is huge.
Ya...we won't be training our own models for some silly budget of ~$5M like the press is claiming.
But we won't have to for some use cases...
I think it will take time and someone on a project with not such a strict NDA to demonstrate just how much this 'completes' things in ways that RAG, functions etc... didn't.
Or I just possibly got lucky with my needs? who knows.
First graph tells the story - below a certain model size (500m params), reinforcement learning is close to useless. Above this (task-dependent) model size threshold, reinforcement learning basically works.
I suspect this is what we saw play out with math/coding reasoning models - until recently, the base models were not good enough for ~random output search to hit on a correct path with any reasonable frequency. Below this threshold of base model intelligence, the only efficient way forward was to collect plain supervised data (either through human labeled math problem solutions [1] or meticulous filtering of web text [2].
But as soon the base model (in this case Deepseek V3) breaks through and can actually solve a decent fraction of math problems, then reinforcement learning (plus other simple tricks like chain-of-thought prompting, simple ensemble voting, etc.) can easily juice the results through the following loop:
1) random search through different solution paths
2) identify the correct solution paths based on the final answer
3) train on the correct solution paths
The exciting thing is that not only can RL bump up the performance of the current base model, but it can be used to generate new high-quality reasoning trace data, which was in painfully short-supply for training the initial models. This leads to a new wave of base models with better one-pass intuition, which leads to more efficient reinforcement learning search on harder problems, which leads to better training data...
Note that this was basically impossible for non-LLM models in the past. You could always juice ImageNet classification performance with a simple ensemble of identically trained models, but that path didn't lead anywhere interesting because a juiced model didn't allow the creation of new synthetic data that was superior to the data it was trained on. The key difference is that LLMs not only output the solution but also output a solution path with all the intermediate steps - and these searched-and-filtered solution paths are much more valuable than the vast majority of the model's initial training data.
Would it be correct to summarize that the general conceptual shift is optimizing MOEs on more specific smaller tasks? It smells like borderline overfitting to me for some reason.
Thank you for the reply. Maybe I'm daft or out of the loop, I don't see the difference between this method and what I have seen previously as "fine tuning on synthetic data".
They also mention in the source tweet that the quality of the base model is important, which is not really accounted for in the $30 figure.
'finetuning on synthetic data' = same LM task with BCE objective, ie. training on next word prediction on good text.
'rl on sparse rewards' = RL objective - reward is given by deterministic evaluation, reward is passed back as sparse signal. there is no 'ground truth' data to compare against.
This reads like an AI hallucination. I'm willing to steak-out this ground even if I'm wrong because of the glaring lack of skepticism. I don't even know when dollars became a concrete compute measure. We used to use FLOPs before we were trying to pull headlines like it were a claw machine game.
"TinyZero is a reproduction of DeepSeek R1 Zero in countdown and multiplication tasks." Does that mean that this has very limited utility (to certain math problems)?
72 comments
[ 6.9 ms ] story [ 169 ms ] threadThat’s nuts and brings forward the idea that an AI is close to self improvement.
AlphaGo Zero is a version of DeepMind's Go software AlphaGo. AlphaGo's team published an article in Nature in October 2017 introducing AlphaGo Zero, a version created without using data from human games, and stronger than any previous version.[1] By playing games against itself, AlphaGo Zero: surpassed the strength of AlphaGo Lee in three days by winning 100 games to 0; reached the level of AlphaGo Master in 21 days; and exceeded all previous versions in 40 days.[2]
Training artificial intelligence (AI) without datasets derived from human experts has significant implications for the development of AI with superhuman skills, as expert data is "often expensive, unreliable, or simply unavailable."[3] Demis Hassabis, the co-founder and CEO of DeepMind, said that AlphaGo Zero was so powerful because it was "no longer constrained by the limits of human knowledge".[4] Furthermore, AlphaGo Zero performed better than standard deep reinforcement learning models (such as Deep Q-Network implementations[5]) due to its integration of Monte Carlo tree search. David Silver, one of the first authors of DeepMind's papers published in Nature on AlphaGo, said that it is possible to have generalized AI algorithms by removing the need to learn from humans.[6]
Google later developed AlphaZero, a generalized version of AlphaGo Zero that could play chess and Shōgi in addition to Go.[7] In December 2017, AlphaZero beat the 3-day version of AlphaGo Zero by winning 60 games to 40, and with 8 hours of training it outperformed AlphaGo Lee on an Elo scale. AlphaZero also defeated a top chess program (Stockfish) and a top Shōgi program (Elmo).[8][9]
Source: https://en.wikipedia.org/wiki/AlphaGo_Zero
Separatelt, but on a related note - do we need analog or quantum computing to "truly" scale?
You could have lots of comments and pass statements and unnecessary conditionals.
I don't think it matters that there is only one correct answer.
It matters that you can verify reliably if the answer is correct enough.
Not saying that it's impossible, but reinforcement learning of the form shown by DeepSeek was particularly well-established and robust. It's sort of ironic, actually...IIRC, OpenAI started in the world of this kind of reinforcement learning.
So why can’t we give LLMs other virtual environments to verify if the solution is correct? For example, a stock simulator, a physics simulator, a driving simulator, etc.
[0]https://en.wikipedia.org/wiki/WarGames
To the extent that someone can figure out a way out of this epistemic box, I'm interested.
It was orders of magnitude more before DeepSeek.
[0] https://newsletter.languagemodels.co/p/the-illustrated-deeps...
Well yes, yes you did, because all you had to do was follow the existing instructions, guidelines, and use easily available materials. You didn't go down any dead ends, didn't have to work your way from propellants like high test peroxide, dangerous hypergolics, and eventually develop solid rocket boosters.
It's like making the generic of a drug someone else developed.
Even if you merely flipped a single bit and created AGI based on existing tech, you're still the legitimate creator of AGI.
DeepSeek never claimed to have trained the base models they used. Now maybe a lot of people inferred that they did, but that's not what they claimed.
Their breakthrough was more along the lines of, "given a model, we can train it to reason using a certain class of RL techniques". Which is, to my mind, more useful in any case.
But yeah, if there are people out there thinking they can train base models from scratch for USD6 Million with no data, they're likely to be disagreeably surprised when they make the attempt.
Isn't that exactly what they have done? Maybe you are confused with the distilled models?
DeepSeek-R1: "DeepSeek-R1-Zero & DeepSeek-R1 are trained based on DeepSeek-V3-Base"
DeepSeek-V3: "At an economical cost of only 2.664M H800 GPU hours, we complete the pre-training of DeepSeek-V3 on 14.8T tokens, producing the currently strongest open-source base model. The subsequent training stages after pre-training require only 0.1M GPU hours"
They developed their “zero” model, which (they claim) is a from-scratch model. Base models are typically not fine-tuned for particular applications (eg chat).
They trained their Zero model into a Chat+Reasoning model, which is what is attracting news.
They ALSO fine-tuned small Qwen models using their big models as a teacher (distillation technique).
There are two papers: https://arxiv.org/pdf/2412.19437v1 for DeepSeek-V3-Base, which makes claims about 2.8M H800 GPU hours, and https://arxiv.org/pdf/2501.12948 for DeepSeek-R1, which was based on V3-Base but doesn't seem to quote the cost of its RL steps.
https://newsletter.languagemodels.co/p/the-illustrated-deeps... is a really fascinating overview of what those new steps were! The innovation here seems to be that the creation of a cutting-edge reasoning model could be made from a relatively inexpensive base model, regardless of how much RL is needed. Part of this seems to be a novel methodology for generating large volumes of chain-of-thought training data; it's left ambiguous exactly how the bulk of this data was generated, though, and it could have been a long process of manual curation.
I'd also take the assertion that they started R1 with V3-Base in its release form with a grain of salt. We'll see whether people can reproduce similar results. Either way, there's a new chain-of-thought model with open weights, and that's an incredible achievement regardless of the context - and one that will be meaningful to future research.
This article is trying to make the headline sound as if they replicated the model, when they “just” replicated the theory of one of their findings. It’s a great result even without the author’s obfuscation.
They're using a similar model and a similar training method to do something simple. It is to DeepSeek what DeepSeek is to ChatGPT-4o, and this kind of reduction to the simplest case is exactly what makes progress go.
(e.g. ChatGPT-4 is a dead end because it is so expensive to train that you can't do training experiments. At $30 a model it is reasonable to train a model 1000s of times)
But how they built on other open concepts like Cold Start + RL + Rejection Sampling etc... is the big thing.
I am not at all connected to whomever this is, but as I am on a "confidential project for a confidential client" it seems to simplify what I would overcomplicate anyway.
https://youtu.be/Pabqg33sUrg
Some of us need to improve domain specificity, and are more interested in targeted capabilities.
DeepSeek-R1-Distill-Qwen-32B itself is pretty good to be honest, but it is more aspirational how easy it is to add reasoning to any base model IMHO.
Those of us who remember Simon’s satisficing principle, realize that often complex problems sub-optimal solutions.
We don't need, want or expect some central omniscient and omnipotent AGI, we want tools that are sufficient to solve real world problems.
Obviously having good reliable domain specific CoT for cold start etc... is hard.
But not nearly as hard as hitching the companies success to MS/OpenAI.
While there is a lot of noise...I think some people are missing the forrest for the trees here. Deepseek is more about moving the decision to use OpenAI for some needs from a single source vendor to one of convenience for us.
For those of us who are in the camp that OpenAI was never going to reach their "AGI" claims because of fundamental issues like the frame problem, this is huge.
Ya...we won't be training our own models for some silly budget of ~$5M like the press is claiming.
But we won't have to for some use cases...
I think it will take time and someone on a project with not such a strict NDA to demonstrate just how much this 'completes' things in ways that RAG, functions etc... didn't.
Or I just possibly got lucky with my needs? who knows.
I suspect this is what we saw play out with math/coding reasoning models - until recently, the base models were not good enough for ~random output search to hit on a correct path with any reasonable frequency. Below this threshold of base model intelligence, the only efficient way forward was to collect plain supervised data (either through human labeled math problem solutions [1] or meticulous filtering of web text [2].
But as soon the base model (in this case Deepseek V3) breaks through and can actually solve a decent fraction of math problems, then reinforcement learning (plus other simple tricks like chain-of-thought prompting, simple ensemble voting, etc.) can easily juice the results through the following loop:
1) random search through different solution paths
2) identify the correct solution paths based on the final answer
3) train on the correct solution paths
The exciting thing is that not only can RL bump up the performance of the current base model, but it can be used to generate new high-quality reasoning trace data, which was in painfully short-supply for training the initial models. This leads to a new wave of base models with better one-pass intuition, which leads to more efficient reinforcement learning search on harder problems, which leads to better training data...
Note that this was basically impossible for non-LLM models in the past. You could always juice ImageNet classification performance with a simple ensemble of identically trained models, but that path didn't lead anywhere interesting because a juiced model didn't allow the creation of new synthetic data that was superior to the data it was trained on. The key difference is that LLMs not only output the solution but also output a solution path with all the intermediate steps - and these searched-and-filtered solution paths are much more valuable than the vast majority of the model's initial training data.
[1] https://arxiv.org/abs/2305.20050
[2] https://arxiv.org/abs/2402.03300 and https://arxiv.org/abs/2206.14858
https://news.ycombinator.com/item?id=42837349
https://github.com/Jiayi-Pan/TinyZero
that MoEs are better for a given compute budget has been known for a while.
They also mention in the source tweet that the quality of the base model is important, which is not really accounted for in the $30 figure.
'rl on sparse rewards' = RL objective - reward is given by deterministic evaluation, reward is passed back as sparse signal. there is no 'ground truth' data to compare against.
more like 'demonstrates the technique generalizes' here. HN has really been inundated with blogspam recently
This is blogspam of https://github.com/Jiayi-Pan/TinyZero and https://nitter.lucabased.xyz/jiayi_pirate/status/18828393705.... This also doesn't mention that it's for one specific domain (playing Countdown). See also https://news.ycombinator.com/item?id=42819262.
Thank you for pointing this out!
or the primary source twitter thread: https://x.com/jiayi_pirate/status/1882839370505621655
Credit
GitHub: https://github.com/Jiayi-Pan/TinyZero
Source on X: https://x.com/jiayi_pirate/status/1882839370505621655