The paper is hard to read. There is no concrete worked-through example, the prose is over the top, and the equations don't really help. I can't make head or tail of this paper.
There is a formal extensional equivalence between Markov chains & LLMs but the only person who seems to be saying anything about this is Gary Marcus. He is constantly making the point that symbolic understanding can not be reduced to a probabilistic computation regardless of how large the graph gets it will still be missing basic stuff like backtracking (which is available in programming languages like Prolog).
I think that Gary is right on basically all counts. Probabilistic generative models are fun but no amount of probabilistic sequence generation can be a substitute for logical reasoning.
Humans can do symbolic understanding that seems to rest on a rather flakey probabilistic neural network in our brains, or at least mine does. I can do maths and the like but there's quite a lot of trial and error and double checking things involved.
GPT5 said it thinks it's fixable when I asked it:
>Marcus is right that LLMs alone are not the full story of reasoning. But the evidence so far suggests the gap can be bridged—either by scaling, better architectures, or hybrid neuro-symbolic approaches.
> Probabilistic generative models are fun but no amount of probabilistic sequence generation can be a substitute for logical reasoning.
Unless you either claim that humans can't do logical reasoning, or claim humans exceed the Turing computable, then given you can trivially wire an LLM into a Turing complete system, this reasoning is illogical due to Turing equivalence.
There's a formal equivalence between Markov chains and literally any system. The entire world can be viewed as a Markov chain. This doesn't tell you anything of interest, just that if you expand state without bound you eventually get the Markov property.
Why can't an LLM do backtracking? Not only within its multiple layers but across token models as reasoning models already do.
You are a probabilistic generative model (If you object, all of quantum mechanics is). I guess that means you can't do any reasoning!
1. You are a neural net and you can backtrack. But unlike an algorithm space search, you’lol go “hmm. That doesn’t look right. Let me try it another way. “
2. Agentic AI already does this in the way that you do it.
If you want to understand SOTA systems then I don't think you should study their formal properties in isolation, i.e. it's not useful to separate them from their environment. Every LLM-based tool has access to code interpreters these days which makes this kind of a moot point.
All execution has to be serial in one way or the other, even if you backtrack you still spent time at T-i at one state, and then go back to a previous state. An LLM could very well act as simply having a tape of the computations it executed and evaluated in the sense that tokens J-K correspond to state at T-i, and the tokens after correspond to the previous state.
In essence, thinking of the computations performed as if logged on an infinitely long tape.
Now, an immutable tape without overwrites is at best a finite automaton, but giving it the ability to replace tokens — which we have and can do — gives us an interpreted tape that — in a very handwavey manner and if you really squeeze your eyes — resembles a Turing machine.
So backtracking could be simulated by an LLM, no?
Is there any “difference” under these conditions up to what an LLM can approximate?
I'm genuinely curious whether neuroscoence has shown that humans perform backtracking?
I am sure we can be taught to backtrack and many things may seem similar, but I just haven't heard of anything at the neuroscience level that backtracking is a fundamental capacity of biological neural networks. So to the extent that humans perform symbolic computations I'm not sure it's established that backtracking is necessary vs other strategies.
The head of a Turing machine is a Markov chain, even simpler even as it transitions completely predictably. But give it a tape and it can compute anything.
> avoidance, which we also discuss in this paper, necessitates putting a much higher premium on insight and understanding of the structural characteristics of the problems being investigated.
I wonder if the authors are aware of The Bitter Lesson
This article is accurate. That's why I'm investigating a bayesian symbolic lisp reasoner. It's incapable of hallucinating, it provides auditable traces which are actual programs and it kicks the crap out of LLMs at stuff like Arc-Agi, symbolic reasoning, logic programs, game playing, etc. I'm working on a paper where I show that the same model can break 80 on arc-agi, run the house by counting cards at blackjack, and solve complex mathematical word problems.
That article is weird. They seem obsessed with nuclear reactors.
Also, they misunderstand how floating point works.
As one learns at high school, the continuous derivative is the limit of the discrete version
as the displacement h is sent to zero. If our computers could afford infinite precision,
this statement would be equally good in practice as it is in continuum mathematics. But
no computer can afford infinite precision, in fact, the standard double-precision IEEE
representation of floating numbers offers an accuracy around the 16th digit, meaning that
numbers below 10−16 are basically treated as pure noise. This means that upon sending
the displacement h below machine precision, the discrete derivatives start to diverge from
the continuum value as roundoff errors then dominate the discretization errors.
Yes, differentiating data has a noise problem. This is where gradient followers sometimes get stuck.
A low pass filter can help by smoothing the data so the derivatives are less noisy. But is that relevant to LLMs? A big insight in machine learning optimization was that, in a high dimensional space, there's usually some dimension with a significant signal, which gets you out of local minima. Most machine learning is in high dimensional spaces but with low resolution data points.
Sauro, if you read this, please refrain from such low-content speculative statements:
“On a loose but telling note, this is still three decades short of the number
of neural connections in the human brain, 1015, and yet they consume some one hundred
million times more power (GWatts as compared to the very modest 20 Watts required by
our brains).”
No human brain could have time to read all the materials of a modern LLM training run even if they lived and read eight hours a day since humans first appeared over 300,000 years ago. More to the point, inference of an LLM is way more energy efficient than human inference (see the energy costs of a B200 decoding a 671B parameter model and estimate the energy needed to write the equivalent of a human book worth of information as part of a larger batch). The main reason for the large energy costs of inference is that we are serving hundreds of millions of people with the same model. No humans have this type of scaling capability.
There is a lot of focus in the comments on the authors' credentials and, apparently, their writing style. It is a pity, because I think their discussion of scaling is interesting, even if comparing LLMs to grid-based differential equation solvers might be unconventional (I haven't convinced myself whether the analogy is entirely apt/valid yet, but it could conceivably be).
23 comments
[ 3.1 ms ] story [ 51.9 ms ] threadGPT5 said it thinks it's fixable when I asked it:
>Marcus is right that LLMs alone are not the full story of reasoning. But the evidence so far suggests the gap can be bridged—either by scaling, better architectures, or hybrid neuro-symbolic approaches.
Unless you either claim that humans can't do logical reasoning, or claim humans exceed the Turing computable, then given you can trivially wire an LLM into a Turing complete system, this reasoning is illogical due to Turing equivalence.
And either of those two claims lack evidence.
There's a formal equivalence between Markov chains and literally any system. The entire world can be viewed as a Markov chain. This doesn't tell you anything of interest, just that if you expand state without bound you eventually get the Markov property.
Why can't an LLM do backtracking? Not only within its multiple layers but across token models as reasoning models already do.
You are a probabilistic generative model (If you object, all of quantum mechanics is). I guess that means you can't do any reasoning!
2. Agentic AI already does this in the way that you do it.
In essence, thinking of the computations performed as if logged on an infinitely long tape.
Now, an immutable tape without overwrites is at best a finite automaton, but giving it the ability to replace tokens — which we have and can do — gives us an interpreted tape that — in a very handwavey manner and if you really squeeze your eyes — resembles a Turing machine.
So backtracking could be simulated by an LLM, no?
Is there any “difference” under these conditions up to what an LLM can approximate?
I am sure we can be taught to backtrack and many things may seem similar, but I just haven't heard of anything at the neuroscience level that backtracking is a fundamental capacity of biological neural networks. So to the extent that humans perform symbolic computations I'm not sure it's established that backtracking is necessary vs other strategies.
He's been saying LLMs wouldn't scale since GPT-3 came out
And yet we all use them every day
Who cares about tracing? Prolog can't be multi-threaded on a GPU, why is that even in the conversation lol
I wonder if the authors are aware of The Bitter Lesson
As one learns at high school, the continuous derivative is the limit of the discrete version as the displacement h is sent to zero. If our computers could afford infinite precision, this statement would be equally good in practice as it is in continuum mathematics. But no computer can afford infinite precision, in fact, the standard double-precision IEEE representation of floating numbers offers an accuracy around the 16th digit, meaning that numbers below 10−16 are basically treated as pure noise. This means that upon sending the displacement h below machine precision, the discrete derivatives start to diverge from the continuum value as roundoff errors then dominate the discretization errors.
Yes, differentiating data has a noise problem. This is where gradient followers sometimes get stuck. A low pass filter can help by smoothing the data so the derivatives are less noisy. But is that relevant to LLMs? A big insight in machine learning optimization was that, in a high dimensional space, there's usually some dimension with a significant signal, which gets you out of local minima. Most machine learning is in high dimensional spaces but with low resolution data points.
“On a loose but telling note, this is still three decades short of the number of neural connections in the human brain, 1015, and yet they consume some one hundred million times more power (GWatts as compared to the very modest 20 Watts required by our brains).”
No human brain could have time to read all the materials of a modern LLM training run even if they lived and read eight hours a day since humans first appeared over 300,000 years ago. More to the point, inference of an LLM is way more energy efficient than human inference (see the energy costs of a B200 decoding a 671B parameter model and estimate the energy needed to write the equivalent of a human book worth of information as part of a larger batch). The main reason for the large energy costs of inference is that we are serving hundreds of millions of people with the same model. No humans have this type of scaling capability.