Seems like thinking should both expose solutions to head towards but also expose problems to flag/navigate around. It would be interesting get a measure of both. In my experience, preventing absolutely silly logic flaws, that even a child would avoid, seems to be the biggest benefit for enabling thinking. And, looking at the thinking output (for models that expose it), this seems to follow the output patterns of "but wait!" that fills the thinking output.
Wang's argument contradicts the view that intelligence necessarily improves through more reasoning and undermines any assumption that recursive self-improvement (RSI) will automatically converge toward greater intelligence.
It doesn't contradict either of these things. It doesn't even say that reasoning doesn't necessarily lead to more intelligence - only that reasoning is not guaranteed to converge to a solution, only tends to in expectation.
Normalizes user voice to goal so the action has more chances to success you can see this on model that dont hide traces sending very badly spelled or incomplete sentences
I'm pretty sure that the current approach of baking in chain-of-thought reasoning into weights is not the end of the line. It just seems to accidental and ad-hoc.
During training you're trying to find paths to success, and reinforcing those.
There is going to be some minimum length path (assuming the model is generating some semblance of real reasoning), but more long ones given the possibilities for less optimal solves and backtracking after errors, so it stands to reason that if you think longer during training you'll find more paths to success.
At inference time you are hoping to follow one of the reasoning paths discovered during training. You need to generate enough tokens to follow the path (or one of the paths), but continuing to think past this point can only be detrimental - if you are lucky the model will realize the error and circle back, but if you are unlucky it will snatch defeat from the jaws of victory and wander off to who knows where.
There was a story recently linked here about one model "editing the weights" of another and thereby improving it's math performance, which was exactly about this - the need to stop generating at inference time at the right point.
IMO LLM's RL acquired reasoning, even if it gets the job done, sometimes spectacularly (math, coding successes), is really a very dumb cargo-cult process. At inference time the model is just trying to copy what it "saw" at training, without actually understanding it, which may even include duplicating errors and subsequent backtracking that had lead to success during training.
It solves for the same thing a "train of thought" (internal vocalization) solves for in the human brain: there's a lot of stuff you "know" as episodic memories of words you've heard/read, but haven't yet internalized via practice into reflex reactions. It's a lot faster and cheaper to acquire a memory of some words, than it is to acquire an entire new skill, so you end up picking up far more memories-of-words than concrete skills from a given training corpus.
Reasoning (i.e. bubbling those remembered words up into your mental context, and then using a special "acting as constrained by words" bridging reflex on them, a.k.a. "system-2 thinking") is how you leverage memories-of-words into useful behavior.
An especially useful version of this is planning, where you chain through "mere words" until you end up at words that declare to yourself what you're about to do; and then you rely on your bridging reflex to react in a way that's constrained by the plan you generated.
Can you explain why your view, which is the opposite of the OP's, is correct?
"RL training directly raises the success probability , by spending a lot of compute on searching for successful paths and then training the model to reproduce them. The gains we credit to “thinking longer” come from the search we paid for during training, rather than at inference."
I get that this is the goal, but having looked at reasoning traces they kinda read like baby babble. The model is just kinda sitting there restating the question and itself over and over. It'll sometimes kinda state/clarify assumptions but it feels nothing like actual train of thought
ISTR some researchers did a test replacing the reasoning tokens with ".", and reasoning still improved the results because dummy tokens are extra passes through the same network, allowing for more computation.
I'm confused on how/why that would ever work for a regular feed-forward transformer model, given that the "more computation" that happens per output token in a transformer, just gets discarded after that token is emitted. The only thing the iteration-loop-step after the dot sees is a context that now has a dot in it.
I think the caveat is that the amount of computation involved in producing a token actually increases with the prefix size (linearly assuming KV caching) so a large enough meaningless CoT can still result in some extra "work" done. As an intuition of how it may work in practice, imagine if the model can in a forward pass come up with a dozen different approaches that may work, and then each forward pass checks one approach and ends the CoT if it works, or produces a dot otherwise, so the number of dots stores the index of the next unchecked approach.
Note also that the paper in question involves training tiny 30M-parameter models from scratch to solve narrow tasks while reasoning this way, so there's not that much applicability to LLMs.
I remember years ago now, trying to feed a text in twice etc., hoping that the representation in the KV cache of the second would end up better. I don't think it worked all that well.
But maybe that is most of the function of restating things babbling. To look at things multiple times.
It's copying a summary of what it needs to know to produce a set of tokens to solve the problem at hand to the front of its context window, so that it doesn't slide off the back and the model doesn't lose track of why it walked into the kitchen in the first place once it gets there.
If all you have is words and you want to make your ball role to the right place in the loss landscape, doesn’t make sense to babble them in a way similar to blowing on the ball to direct ot.
Because it seems they aren't using words the way we humans do. Instead, they're using them the way you would use numbers or logical equations. So the whole trace is just a search through solution space using calculations.
AFAICT those are various intermediate representations it uses for what you wrote / things it already stated in other ways; and when you see it restating things multiple times, it's due to a kind of swapping where it's struggling to juggle those representations.
If you think about how the algorithms the LLM has learned to do what it does must be represented across its layers, each such algorithm must be spread across a certain number of layers. But this includes the algorithms the LLM uses to understand (recognize, parse) the context; and the algorithms the LLM uses to turn its semantic idea of how it should respond into tokens appropriate to the "word game" being played at the moment. Every single token-generation step is therefore its own independent scheduling problem, of selecting a particular recognizer algorithm, some number of analysis algorithms, and a generation algorithm, such that they can all take the previous step's output as their input, despite each such algorithm only existing such that it can start on a particular layer.
One consequence of this is that complex recognition/generation, and complex analysis, trade off against one-another. The more a given token-generation step tries to "think", the less-sophisticated an understanding that step can have of the context, and the less-sophisticated a speech pattern that step can have for its output. And vice-versa.
In all parts of the LLM's output other than reasoning, a trained expectation of consistently high fluency in both recognition and production is expected/required. So analytical capacity is always sacrificed.
But reasoning has no such requirement. And so LLMs learn that during reasoning, they're free to flip between advancing with "NLP-focused" steps vs. "analysis-focused" steps.
And because the LLM's "verbal IQ" is temporarily decreased during analysis-focused steps, for those steps to make successful progress, they need the assistance of previous NLP-focused bridging steps to rephrase anything in the context that's too complex for the simpler NLP algorithms that get used alongside those complex analytical algorithms, into intermediate representations that those simple NLP algorithms can handle.
And then, for associational-memory reasons, you might see more bridging steps where the "simple language" the high-analysis-low-NLP algorithms output gets turned back into "fluent language". And on it goes, back and forth, during reasoning.
If this is true, I wonder why LLM safety people put so much weight on the ability to analyze the reasoning traces? If restating the same content multiple times has some useful state change inside the model, it's not obvious from the trace what exactly that is, and so the leap to "less interpretable" model designs is not so severe. Maybe there's more nuance here :)
I don't know about my other fellow humans, but it's honestly not too different than my inner monologue which feels very chaotic and would make me look insane if I spoke any of it outloud. Mine's full of back tracking, random things that are barely related to the task at hand (tbf that may just be adhd; idk), repeating myself (which often really does help with recall when struggling with that), rewording things I've already thought, telling myself to stop over thinking then not, etc. It may not be super efficient, but at least for me it eventually works most of the time.
I think it also addresses the issue that there is a finite amount of computation you can do in one pass.
So you use one pass to compute one relevant word, let's say you are working on a sound analysis task, and you output "FFT".
Now in the second pass, FFT is right there at the input, so you can push that idea forward more until you are out of layers again and then you use the output token to prime the next pass.
That is how I have been thinking about LLM reasoning pretty much since it arrived.
I am not sure I support his statement about RL “searching for successful paths and then training the model to reproduce them”.
Don’t we train the model to use a certain method to explore the space in search for an answer?
As soon as you have tool calling and a way to verify your result, such as a theorem prover for example you are essentially running an optimization algorithm.
> Consider the problem: “what’s 1 +2 +3 +⋯ +10”? One solution is to add the numbers one at a time: 1, 3, 6, 10, and so on up to 55. Another way is to pair them from the outside in: 1 +10, 2 +9, 3 +8, 4 +7 and 5 +6 each make 11, and five pairs of 11 make 55. Both of these paths land on the correct answer, 55.
Watching my son work through difficult math problems has forced me into this realization. In my head there is one "correct" path, and I often have to stop myself from interrupting his chain of thought. He often is just getting there a different way.
>I'm a researcher based in San Francisco. I like to imagine where the world is headed, and what it takes to get there.
>I did my PhD in computer science at Stanford, where I designed long-context architectures for sequence modeling and memory. Before that, I studied math & physics at Cornell. Most recently, I was a research scientist at Cartesia, working on in-context tokenization and novel architectures (HNets).
Initially, I typed "intelligence is a path in token space" but then I decided that was too abstract. In case relevant, I was typing on a device with Muse installed.
I don’t understand this part: if we assume a function that can validate an answer… what do we need AI to do? The existence of a function that can validate an answer means we have a proven way of finding the answer.
Imagine you are in a room with a lamp that has one hundred switches. Any wrong switch pressed will short the lamp and break it. You don’t know which switch turns on the lamp, but you know that when you see it light up, the problem is solved.
In this situation you have a validation function: “Is the lamp illuminated?”; While you don’t know how to find the answer to how to turn it on: “Which switch illuminates the lamp?”
Ok. I get that. You can query the lamp’s state. Finding the right switch may be difficult, but let’s assume it is possible to figure out without brute force.
So then queries like “design an effective weight loss program” can be validated observing the weight going down. It can’t be prevalidated, the proof is in the pudding.
But a query such as “what is the shortest route between these 50 cities” doesn’t have a validation function unless it CAN compute the shortest route, or if the shortest route was somehow precomputed and given to the validation function. In this sense, it could be used to calibrate the AI. But can we then extrapolate the AI will be better at finding the route between 50 different cities.
There's no efficient validation function for optimization problems. If there was, most NP-hard problems wouldn't exist.
You have to approach those with a goal of "be better than x" rather than "be exact".
Caveat that I am an amateur in this field, but given the Traveling Salesman problem in your example, if we aren't allowed to use an existing solver to check against, I'd aim for a suite of validation functions that certify different parts of reaching the solution like:
- validate properties of the logic that have to hold up, such as "shifting the order of my list of cities alone should never change the outcome"
- validate that the outcome never reaches above or below certain bounds that we know aren't possible due to trivial understanding of the problem
- validate that the outcome is always below some other algorithm we know is inefficient
Essentially, for most logic problems, checking validity is exponentially simpler than deriving a solution, but with optimization problems specifically, we are validating that the outcome is not wrong rather than is correct.
This is precisely what humans do. We use small world statistics to drastically reduce the search space of problems. Of course there’s the whole issue of contextual search which LLMs address in no way whatsoever, so even improving sampling won’t help much, as inevitably one starts to generate too many hypotheses (reasoning traces) to get a good average.
Sorry, are we back to “This site works best in IE 6.0”? At minimum one should and easily could check in safari, if already on an ios device, or a mac. Preferably FF as well, of course, though I understand it’s too much to ask.
To the point, I first saw this in experienced this in brave on iphone, and wrote the comment some time after, from memory. I just checked again and it turns out that the /bottom/ half of equations are cut off, in both brave and safari on iphone. Not sure if my memory is failing or whether it switched around.
Not sure whether you are actually the author and don’t want to go around throwing accusations, but - If you care and if this isn’t the case - i now consider sites where author states “works for me in chrome” as very likely to be slop. That said, due to issue above i didn’t spend much time on it to make a judgement based on content.
This is a great article. In our RL work, we think of reasoning as a skill -- a set of tools and protocols to leverage knowledge a model already has to improve the probability of creating knowledge the model doesn't have.
Although specifically defining reasoning as a skill is hard, we've seen success with partners who agree that you don't need to enable reasoning on every single rollout -- you get diminishing returns past a certain point (the model often already has a complete toolkit for how to explore ideas and run calculations inside of its reasoning budget). Instead, it's more compute efficient to spend some percentage of rollouts training models with reasoning disabled, on problems that they are calibrated to have a good solve rate on without reasoning. This improves the cognitive core/latent reasoning directly which better translates to both total tokens needed to solve problems, and the hardest problems a model can solve. And it's more realistic, considering that humans don't need an inner monologue to solve most problems (even hard ones, e.g. a flow state).
This is fascinating to me, I imagine this is in large part the future. This along with the weight space programming work that came out a few days ago are where I think we are headed.
I wonder if ideally we could skip ALL the "thinking/reasoning" steps if you trained a model to jump from "question space" to "answer space". Imagine a diffusion model or something that learns to map the start of the path to the destination. Very interesting
I use Qwen3.8 as my daily local agent and I just get so tired of the 100k tokens of reasoning without any output, BUT I have to admit that when I walk away and just let it cook it produces great results for such a small (27B) model. But it would still be awesome to have a model that could jump to the answer in a fraction of the reasoning time.
Oh, I should perhaps make multiple requests to my GPUs, say best of 3 and then run a reduction run on that. Since my GPUs are only on my use-cases this should work. Interesting.
72 comments
[ 3.8 ms ] story [ 50.1 ms ] threadSounds like it's hillclimbing all the way down!
Cool framing. I'm excited for the parallel construction of ___ as pathfinding through weight space.
P. S. The animated bird is truly magical!
1) Pretraining creates a bias _towards_ "correct answers" with each inference step
2) A longer walk is more likely to land in the correct region, given it's easier to verify you have the correct answer than to create it
3) RL selects the subset of walks that lands in the correct region and tilts the distribution in favor of such walks
4) By 2, that trains towards longer walks
...Is that correct?
There is going to be some minimum length path (assuming the model is generating some semblance of real reasoning), but more long ones given the possibilities for less optimal solves and backtracking after errors, so it stands to reason that if you think longer during training you'll find more paths to success.
At inference time you are hoping to follow one of the reasoning paths discovered during training. You need to generate enough tokens to follow the path (or one of the paths), but continuing to think past this point can only be detrimental - if you are lucky the model will realize the error and circle back, but if you are unlucky it will snatch defeat from the jaws of victory and wander off to who knows where.
There was a story recently linked here about one model "editing the weights" of another and thereby improving it's math performance, which was exactly about this - the need to stop generating at inference time at the right point.
IMO LLM's RL acquired reasoning, even if it gets the job done, sometimes spectacularly (math, coding successes), is really a very dumb cargo-cult process. At inference time the model is just trying to copy what it "saw" at training, without actually understanding it, which may even include duplicating errors and subsequent backtracking that had lead to success during training.
Reasoning (i.e. bubbling those remembered words up into your mental context, and then using a special "acting as constrained by words" bridging reflex on them, a.k.a. "system-2 thinking") is how you leverage memories-of-words into useful behavior.
An especially useful version of this is planning, where you chain through "mere words" until you end up at words that declare to yourself what you're about to do; and then you rely on your bridging reflex to react in a way that's constrained by the plan you generated.
"RL training directly raises the success probability , by spending a lot of compute on searching for successful paths and then training the model to reproduce them. The gains we credit to “thinking longer” come from the search we paid for during training, rather than at inference."
https://arxiv.org/pdf/2404.15758
Note also that the paper in question involves training tiny 30M-parameter models from scratch to solve narrow tasks while reasoning this way, so there's not that much applicability to LLMs.
I remember years ago now, trying to feed a text in twice etc., hoping that the representation in the KV cache of the second would end up better. I don't think it worked all that well.
But maybe that is most of the function of restating things babbling. To look at things multiple times.
If you think about how the algorithms the LLM has learned to do what it does must be represented across its layers, each such algorithm must be spread across a certain number of layers. But this includes the algorithms the LLM uses to understand (recognize, parse) the context; and the algorithms the LLM uses to turn its semantic idea of how it should respond into tokens appropriate to the "word game" being played at the moment. Every single token-generation step is therefore its own independent scheduling problem, of selecting a particular recognizer algorithm, some number of analysis algorithms, and a generation algorithm, such that they can all take the previous step's output as their input, despite each such algorithm only existing such that it can start on a particular layer.
One consequence of this is that complex recognition/generation, and complex analysis, trade off against one-another. The more a given token-generation step tries to "think", the less-sophisticated an understanding that step can have of the context, and the less-sophisticated a speech pattern that step can have for its output. And vice-versa.
In all parts of the LLM's output other than reasoning, a trained expectation of consistently high fluency in both recognition and production is expected/required. So analytical capacity is always sacrificed.
But reasoning has no such requirement. And so LLMs learn that during reasoning, they're free to flip between advancing with "NLP-focused" steps vs. "analysis-focused" steps.
And because the LLM's "verbal IQ" is temporarily decreased during analysis-focused steps, for those steps to make successful progress, they need the assistance of previous NLP-focused bridging steps to rephrase anything in the context that's too complex for the simpler NLP algorithms that get used alongside those complex analytical algorithms, into intermediate representations that those simple NLP algorithms can handle.
And then, for associational-memory reasons, you might see more bridging steps where the "simple language" the high-analysis-low-NLP algorithms output gets turned back into "fluent language". And on it goes, back and forth, during reasoning.
So you use one pass to compute one relevant word, let's say you are working on a sound analysis task, and you output "FFT".
Now in the second pass, FFT is right there at the input, so you can push that idea forward more until you are out of layers again and then you use the output token to prime the next pass.
I am not sure I support his statement about RL “searching for successful paths and then training the model to reproduce them”.
Don’t we train the model to use a certain method to explore the space in search for an answer?
As soon as you have tool calling and a way to verify your result, such as a theorem prover for example you are essentially running an optimization algorithm.
Watching my son work through difficult math problems has forced me into this realization. In my head there is one "correct" path, and I often have to stop myself from interrupting his chain of thought. He often is just getting there a different way.
>I did my PhD in computer science at Stanford, where I designed long-context architectures for sequence modeling and memory. Before that, I studied math & physics at Cornell. Most recently, I was a research scientist at Cartesia, working on in-context tokenization and novel architectures (HNets).
Not the MSL Alex Wang
Initially, I typed "intelligence is a path in token space" but then I decided that was too abstract. In case relevant, I was typing on a device with Muse installed.
That's a remarkable assertion.
In this situation you have a validation function: “Is the lamp illuminated?”; While you don’t know how to find the answer to how to turn it on: “Which switch illuminates the lamp?”
So then queries like “design an effective weight loss program” can be validated observing the weight going down. It can’t be prevalidated, the proof is in the pudding.
But a query such as “what is the shortest route between these 50 cities” doesn’t have a validation function unless it CAN compute the shortest route, or if the shortest route was somehow precomputed and given to the validation function. In this sense, it could be used to calibrate the AI. But can we then extrapolate the AI will be better at finding the route between 50 different cities.
You have to approach those with a goal of "be better than x" rather than "be exact".
Caveat that I am an amateur in this field, but given the Traveling Salesman problem in your example, if we aren't allowed to use an existing solver to check against, I'd aim for a suite of validation functions that certify different parts of reaching the solution like:
- validate properties of the logic that have to hold up, such as "shifting the order of my list of cities alone should never change the outcome"
- validate that the outcome never reaches above or below certain bounds that we know aren't possible due to trivial understanding of the problem
- validate that the outcome is always below some other algorithm we know is inefficient
Essentially, for most logic problems, checking validity is exponentially simpler than deriving a solution, but with optimization problems specifically, we are validating that the outcome is not wrong rather than is correct.
This is precisely what humans do. We use small world statistics to drastically reduce the search space of problems. Of course there’s the whole issue of contextual search which LLMs address in no way whatsoever, so even improving sampling won’t help much, as inevitably one starts to generate too many hypotheses (reasoning traces) to get a good average.
To the point, I first saw this in experienced this in brave on iphone, and wrote the comment some time after, from memory. I just checked again and it turns out that the /bottom/ half of equations are cut off, in both brave and safari on iphone. Not sure if my memory is failing or whether it switched around.
Not sure whether you are actually the author and don’t want to go around throwing accusations, but - If you care and if this isn’t the case - i now consider sites where author states “works for me in chrome” as very likely to be slop. That said, due to issue above i didn’t spend much time on it to make a judgement based on content.
Although specifically defining reasoning as a skill is hard, we've seen success with partners who agree that you don't need to enable reasoning on every single rollout -- you get diminishing returns past a certain point (the model often already has a complete toolkit for how to explore ideas and run calculations inside of its reasoning budget). Instead, it's more compute efficient to spend some percentage of rollouts training models with reasoning disabled, on problems that they are calibrated to have a good solve rate on without reasoning. This improves the cognitive core/latent reasoning directly which better translates to both total tokens needed to solve problems, and the hardest problems a model can solve. And it's more realistic, considering that humans don't need an inner monologue to solve most problems (even hard ones, e.g. a flow state).
I wonder if ideally we could skip ALL the "thinking/reasoning" steps if you trained a model to jump from "question space" to "answer space". Imagine a diffusion model or something that learns to map the start of the path to the destination. Very interesting
I suspect weight space programming is not going to be the way models are practically optimized ever.
https://x.com/NeelNanda5/status/2117375052618539485