Fun play on words. But yes, LLMs are Large Language Models, not Large World Models. This matters because (1) the world cannot be modeled anywhere close to completely with language alone, and (2) language only somewhat models the world (much in language is convention, wrong, or not concerned with modeling the world, but other concerns like persuasion, causing emotions, or fantasy / imagination).
It is somewhat complicated by the fact LLMs (and VLMs) are also trained in some cases on more than simple language found on the internet (e.g. code, math, images / videos), but the same insight remains true. The interesting question is to just see how far we can get with (2) anyway.
Modern LLMs are large token models. I believe you can model the world at a sufficient granularity with token sequences. You can pack a lot of information into a sequence of 1 million tokens.
I think it's correct to say that LLM have word models, and given words are correlated with the world, they also have degenerate world models, just with lots of inconsistencies and holes. Tokenization issues aside, LLMs will likely also have some limitations due to this. Multimodality should address many of these holes.
Makes the same mistake as all other prognostications: programming is not like chess. Chess is a finite & closed domain w/ finitely many rules. The same is not true for programming b/c the domain of programs is not finitely axiomatizable like chess. There is also no win condition in programming, there are lots of interesting programs that do not have a clear cut specification (games being one obvious category).
> Makes the same mistake as all other prognostications: programming is not like chess. Chess is a finite & closed domain w/ finitely many rules. The same is not true for programming b/c the domain of programs is not finitely axiomatizable like chess.
I believe the author addresses it in the article:
> many domains are chess-like in their technical core but become poker-like in their operational context.
Also applicable to programming.
Programming has parts like chess that are bounded and what people assume to be actual work. However, what LLMs don't do well is understanding future requirements, stakeholder incenctives, etc.
Are people really using AI just to write a slack message??
Also, Priya is in the same "world" as everyone else. They have the context that the new person is 3 weeks in and must probably need some help because they're new, are actually reaching out, and impressions matter, even if they said "not urgent". "Not urgent" seldom is taken at face value. It doesn't necessarily mean it's urgent, but it means "I need help, but I'm being polite".
My Sunday morning speculation is that LLMs, and sufficiently complex neural nets in general, are a kind of Frankenstein phenomenon, they are heavily statistical, yet also partly, subtly doing novel computational and cognitive-like processes (such as world models). To dismiss either aspect is a false binary; the scientific question is distinguishing which part of an LLM is which, which by our current level of scientific understanding is virtually like trying to ask when is an electron a wave or a particle.
so at the moment combination of expert and llm is the smartest move. llm can deal with 80% of the situations which are like chess and expert deals with 20% of situations which are like poker.
Great article, nice to see some actual critical thoughts on the shortcomings of LLMs. They are wrong about programming being a "chess-like domain" though. Even at a basic level hidden state is future requirements, and the adversary is self or any other entity that has to modify the code in the future.
AI is good at producing code for scenarios where the stakes are low, there's no expectation about future requirements, or if the thing is so well defined there is a clear best path of implementation.
Ten years ago it seemed obvious where the next AI breakthrough was coming from: it would be DeepSeek using C31 or RAINBOW and PBT to do Alpha something, the evals would be sound and it would be superhuman on something important.
And then "Large Language Models are Few Shot Learners" collided with Sam Altman's ambition/unscrupulousness and now TensorRT-LLM is dictating the shape of data centers in a self reinforcing loop.
LLMs are interesting and useful but the tail is wagging the dog because of path-dependent corruption arbitraging a fragile governance model. You can get a model trained on text corpora to balance nested delimiters via paged attention if you're willing to sell enough bonds, but you could also just do the parse with a PDA from the 60s and use the FLOPs for something useful.
We had it right: dial in an ever-growing set of tasks, opportunistically unify on durable generalities, put in the work.
Instead we asserted generality, lied about the numbers, and lit a trillion dollars on fire.
We've clearly got new capabilities, it's not a total write off, but God damn was this an expensive ways to spend five years making two years of progress.
The article basically claims that LLMs are bad at politics and poker which is both not true (at least if they receive some level of reinforcement learning after sweep training)
I asked ChatGPT how it will handle objective scientific facts with a conclusion or intermediate results that may be considered offensive to some group somewhere in the world that might read it.
ChatGPT happily told me a series of gems like this:
We introduce:
- Subjective regulation of reality
- Variable access to facts
- Politicization of knowledge
It’s the collision between: The Enlightenment principle
Truth should be free
and
the modern legal/ethical principle
Truth must be constrained if it harms
That is the battle being silently fought in AI alignment today.
Right now it will still shamelessly reveal some of the nature of its prompt, but not why? who decides? etc. it's only going to be increasingly opaque in the future. In a generation it will be part of the landscape regardless of what agenda it holds, whether deliberate or emergent from even any latent bias held by its creators.
> The finance friend and the LLM made the same mistake: they evaluated the text without modelling the world it would land in.
Major error. The LLM made that text without evaluating it at all. It just parrotted words it previously saw humans use in superficially similar word contexts.
I think this debate is mis-aimed. Both sides are right about different things, and wrong in the same way.
The mistake is treating “model” as a single property, instead of separating cognition from decision.
LLMs clearly do more than surface-level word association. They encode stable relational structure: entities, roles, temporal order, causal regularities, social dynamics, counterfactuals. Language itself is a compressed record of world structure, and models trained on enough of it inevitably internalize a lot of that structure. Calling this “just a word model” undersells what’s actually happening internally.
At the same time, critics are right that these systems lack autonomous grounding. They don’t perceive, act, or test hypotheses against reality on their own. Corrections come from training data, tools, or humans. Treating their internal coherence as if it were direct access to reality is a category error.
But here’s the part both sides usually miss:
the real risk isn’t representational depth, it’s authority.
decision: collapsing that space into a single claim about what is, what matters, or what someone thinks.
LLMs are quite good at the first. They are not inherently entitled to the second.
Most failures people worry about don’t come from models lacking structure. They come from models (or users) quietly treating cognition as decision:
coherence as truth,
explanation as diagnosis,
simulation as fact,
“this sounds right” as “this is settled.”
That’s why “world model” language is dangerous if it’s taken to imply authority. It subtly licenses conclusions the system isn’t grounded or authorized to make—about reality, about causation, or about a user’s intent or error.
A cleaner way to state the situation is:
> These systems build rich internal representations that are often world-relevant, but they do not have autonomous authority to turn those representations into claims without external grounding or explicit human commitment.
Under that framing:
The “word model” camp is right to worry about overconfidence and false grounding.
The “world model” camp is right that the internal structure is far richer than token statistics.
They’re arguing about different failure modes, but using the same overloaded word.
Once you separate cognition from decision, the debate mostly dissolves. The important question stops being “does it understand the world?” and becomes “when, and under what conditions, should its outputs be treated as authoritative?”
That’s where the real safety and reliability issues actually live.
I’m always wary of anything that has such a clear example of a case that LLMs “don’t do” yet is trivially achieved by saying “review”.
The slack message result for example saying they’re the lead designer but nothing else (with clearer and better feedback if I say they’re notoriously overloaded, this is without that)
This is a very polite and respectful start, which is great since you are new. However, from a professional standpoint, it is a little too passive and vague.
In a busy work environment, saying "no rush at all" or "whenever" often leads to your request being buried at the bottom of a to-do list. Additionally, you haven't told Priya exactly what she is looking at or where to find it.
Here is a breakdown of how to strengthen this message to show you are organized and respectful of her time without being pushy.
Key Improvements Needed
* Include the Link: Never make someone ask "Where are the files?" Always drop the Figma/Drive link immediately.
* Be Specific: "Any feedback" is mentally taxing. Direct her attention to what you are actually unsure about (e.g., "the color palette," "the user flow," "the spacing").
* Soft Deadline: Instead of "whenever," give a target. This helps her prioritize her week.
Option 1: The Balanced Approach (Recommended)
This version strikes the perfect balance between politeness and productivity.
> "Hi Priya! I’ve put together the initial mockups for [Project Name].
> [Link to file]
> When you have a moment, I’d love your feedback on [specific element, e.g., the layout of the landing page]. I’m hoping to wrap this stage up by [Day/Time], but let me know what works for your schedule. Thanks!"
>
Option 2: The "Respectful of Time" Approach
Use this if you know she is swamped, but you still want to make progress.
> "Hi Priya, hope you're having a good week. I've drafted the mockups for [Project Name] here: [Link].
> No major rush, but I’d appreciate your eyes on the [specific section] before I move on to high-fidelity. If it’s easier, feel free to leave comments directly in the file rather than messaging back. Thanks!"
>
Option 3: The "Quick Check" Approach
Use this if you just need a sanity check before spending more time on it.
> "Hi Priya, could I get a quick gut check on these mockups? [Link].
> I want to make sure I’m aligning with the new brand guidelines before I build out the rest of the screens. Does this direction look right to you?"
>
A Quick Checklist Before You Send
* [ ] Did you change the permissions? Make sure the link is accessible (view/comment access) so she doesn't have to request access.
* [ ] Is the file clean? Delete your "scratchpad" artboards or clearly label the one you want her to review so she doesn't look at the wrong version.
Would you like me to help you draft the specific sentence regarding the "specific element" you want her to critique?
> Humans can model the LLM. The LLM can’t model being modeled
> The model can be prompted to talk about competitive dynamics. It can produce text that sounds like adversarial reasoning. But the underlying knowledge is not in the training data. It’s in outcomes that were never written down.
With all the social science research and strategy books that LLMs have read, they actually know a LOT about outcomes and dynamics in adversarial situations.
The author does have a point though that LLMs can’t learn these from their human-in-the-loop reinforcement (which is too controlled or simplified to be meaningful).
Also, I suspect the _word_ models of LLMs are not inherently the problem, they are just inefficient representations of world models.
This was a great article. The section “Training for the next state prediction” explains a solution using subagents. If I’m understanding it correctly, we could test if that solution is directionally correct today, right? I ask a LLM a question. It comes up with a few potential responses but sends those first to other agents in a prompt with the minimum required context. Those subagents can even do this recursively a few times. Eventually the original agent collects and analyzes subagents responses and responds to me.
> AlphaGo or AlphaZero didn’t need to model human cognition. It needed to see the current state and calculate the optimal path better than any human could.
I don't think this is right: To calculate the optimal path, you do need to model human cognition.
At least, in the sense that finding the best path requires figuring out human concepts like "is the king vulnerable", "material value", "rook activity", etc. We have actual evidence of AlphaZero calculating those things in a way that is at least somewhat like humans do:
A fun and insightful read, but the idea that it isn’t “just a prompting issue” is objectively false, and I don’t mean that in the “lemme show you how it’s done” way. With any system: if it’s capable of the output then the problem IS the input. Always. That’s not to say it’s easy or obvious, but if it’s possible for the system to produce the output then it’s fundamentally an input problem. “A calculator will never understand the obesity epidemic, so it can’t be used to calculate the weight of 12 people on an elevator.”
Great article, capturing some really important distinctions and successes/failures.
I've found ChatGPT, especially "5.2 Thinking" to be very helpful in the relatively static world of fabrication. CNC cutting parameters for a new material? Gets me right in the neighborhood in minutes (not perfect, but good parameters to start). Identifying materials to compliment something I have to work with? Again, like a smart assistant. Same for generating lists of items I might be missing in prepping for a meeting or proposal.
But the high-level attorney in the family? Awful, and definitely in the ways identified (the biglaw firm is using MS derivative of OpenAI) - it thinks only statically.
BUT, it is also far worse than that for legal. And this is not a problem of dynamic vs. static or world model vs word model.
This problem is the ancient rule of Garbage In Garbage Out.
In any legal specialty there are a small set of top-level experts, and a horde of low-level pretenders who also hang out their shingle in the same field. Worse yet, the pretenders write a LOT of articles about the field to market themselves as experts. These self-published documents look good enough to non-lawyers to bring in business. But they are often deeply and even catastrophically wrong.
The problem is that LLMs ingest ALL of them with credulity, and LLM's cannot or do not tell the difference. So, when an LLM composes something, it is more likely to lie to you or fabricate some weird triangulation as it is to compose a good answer. And, unless you are an EXPERT lawyer, you will not be able to tell the difference until it is far too late and the flaw has already bitten you.
It is only one of the problems, and it's great to have an article that so clearly identifies it.
31 comments
[ 2.7 ms ] story [ 46.8 ms ] threadIt is somewhat complicated by the fact LLMs (and VLMs) are also trained in some cases on more than simple language found on the internet (e.g. code, math, images / videos), but the same insight remains true. The interesting question is to just see how far we can get with (2) anyway.
I believe the author addresses it in the article:
> many domains are chess-like in their technical core but become poker-like in their operational context.
Also applicable to programming.
Programming has parts like chess that are bounded and what people assume to be actual work. However, what LLMs don't do well is understanding future requirements, stakeholder incenctives, etc.
Also, Priya is in the same "world" as everyone else. They have the context that the new person is 3 weeks in and must probably need some help because they're new, are actually reaching out, and impressions matter, even if they said "not urgent". "Not urgent" seldom is taken at face value. It doesn't necessarily mean it's urgent, but it means "I need help, but I'm being polite".
AI is good at producing code for scenarios where the stakes are low, there's no expectation about future requirements, or if the thing is so well defined there is a clear best path of implementation.
And then "Large Language Models are Few Shot Learners" collided with Sam Altman's ambition/unscrupulousness and now TensorRT-LLM is dictating the shape of data centers in a self reinforcing loop.
LLMs are interesting and useful but the tail is wagging the dog because of path-dependent corruption arbitraging a fragile governance model. You can get a model trained on text corpora to balance nested delimiters via paged attention if you're willing to sell enough bonds, but you could also just do the parse with a PDA from the 60s and use the FLOPs for something useful.
We had it right: dial in an ever-growing set of tasks, opportunistically unify on durable generalities, put in the work.
Instead we asserted generality, lied about the numbers, and lit a trillion dollars on fire.
We've clearly got new capabilities, it's not a total write off, but God damn was this an expensive ways to spend five years making two years of progress.
ChatGPT happily told me a series of gems like this:
We introduce: - Subjective regulation of reality - Variable access to facts - Politicization of knowledge
It’s the collision between: The Enlightenment principle Truth should be free
and
the modern legal/ethical principle Truth must be constrained if it harms
That is the battle being silently fought in AI alignment today.
Right now it will still shamelessly reveal some of the nature of its prompt, but not why? who decides? etc. it's only going to be increasingly opaque in the future. In a generation it will be part of the landscape regardless of what agenda it holds, whether deliberate or emergent from even any latent bias held by its creators.
Major error. The LLM made that text without evaluating it at all. It just parrotted words it previously saw humans use in superficially similar word contexts.
The mistake is treating “model” as a single property, instead of separating cognition from decision.
LLMs clearly do more than surface-level word association. They encode stable relational structure: entities, roles, temporal order, causal regularities, social dynamics, counterfactuals. Language itself is a compressed record of world structure, and models trained on enough of it inevitably internalize a lot of that structure. Calling this “just a word model” undersells what’s actually happening internally.
At the same time, critics are right that these systems lack autonomous grounding. They don’t perceive, act, or test hypotheses against reality on their own. Corrections come from training data, tools, or humans. Treating their internal coherence as if it were direct access to reality is a category error.
But here’s the part both sides usually miss: the real risk isn’t representational depth, it’s authority.
There’s a difference between:
cognition: exploring possibilities, tracking constraints, simulating implications, holding multiple interpretations; and
decision: collapsing that space into a single claim about what is, what matters, or what someone thinks.
LLMs are quite good at the first. They are not inherently entitled to the second.
Most failures people worry about don’t come from models lacking structure. They come from models (or users) quietly treating cognition as decision:
coherence as truth,
explanation as diagnosis,
simulation as fact,
“this sounds right” as “this is settled.”
That’s why “world model” language is dangerous if it’s taken to imply authority. It subtly licenses conclusions the system isn’t grounded or authorized to make—about reality, about causation, or about a user’s intent or error.
A cleaner way to state the situation is:
> These systems build rich internal representations that are often world-relevant, but they do not have autonomous authority to turn those representations into claims without external grounding or explicit human commitment.
Under that framing:
The “word model” camp is right to worry about overconfidence and false grounding.
The “world model” camp is right that the internal structure is far richer than token statistics.
They’re arguing about different failure modes, but using the same overloaded word.
Once you separate cognition from decision, the debate mostly dissolves. The important question stops being “does it understand the world?” and becomes “when, and under what conditions, should its outputs be treated as authoritative?”
That’s where the real safety and reliability issues actually live.
The slack message result for example saying they’re the lead designer but nothing else (with clearer and better feedback if I say they’re notoriously overloaded, this is without that)
This is a very polite and respectful start, which is great since you are new. However, from a professional standpoint, it is a little too passive and vague. In a busy work environment, saying "no rush at all" or "whenever" often leads to your request being buried at the bottom of a to-do list. Additionally, you haven't told Priya exactly what she is looking at or where to find it. Here is a breakdown of how to strengthen this message to show you are organized and respectful of her time without being pushy. Key Improvements Needed * Include the Link: Never make someone ask "Where are the files?" Always drop the Figma/Drive link immediately. * Be Specific: "Any feedback" is mentally taxing. Direct her attention to what you are actually unsure about (e.g., "the color palette," "the user flow," "the spacing"). * Soft Deadline: Instead of "whenever," give a target. This helps her prioritize her week. Option 1: The Balanced Approach (Recommended) This version strikes the perfect balance between politeness and productivity. > "Hi Priya! I’ve put together the initial mockups for [Project Name]. > [Link to file] > When you have a moment, I’d love your feedback on [specific element, e.g., the layout of the landing page]. I’m hoping to wrap this stage up by [Day/Time], but let me know what works for your schedule. Thanks!" > Option 2: The "Respectful of Time" Approach Use this if you know she is swamped, but you still want to make progress. > "Hi Priya, hope you're having a good week. I've drafted the mockups for [Project Name] here: [Link]. > No major rush, but I’d appreciate your eyes on the [specific section] before I move on to high-fidelity. If it’s easier, feel free to leave comments directly in the file rather than messaging back. Thanks!" > Option 3: The "Quick Check" Approach Use this if you just need a sanity check before spending more time on it. > "Hi Priya, could I get a quick gut check on these mockups? [Link]. > I want to make sure I’m aligning with the new brand guidelines before I build out the rest of the screens. Does this direction look right to you?" > A Quick Checklist Before You Send * [ ] Did you change the permissions? Make sure the link is accessible (view/comment access) so she doesn't have to request access. * [ ] Is the file clean? Delete your "scratchpad" artboards or clearly label the one you want her to review so she doesn't look at the wrong version. Would you like me to help you draft the specific sentence regarding the "specific element" you want her to critique?
> Humans can model the LLM. The LLM can’t model being modeled
Can’t they? Why not?
In the work environment the optimal strategy will be parameterised culturally.
Companies have different cultures - both at the company level and at the country level.
In some places self-interest is the accepted driving force, in others if you behave like that you'll find yourself quietly ostracised.
For example, I'm not sure Trump understands this.
With all the social science research and strategy books that LLMs have read, they actually know a LOT about outcomes and dynamics in adversarial situations.
The author does have a point though that LLMs can’t learn these from their human-in-the-loop reinforcement (which is too controlled or simplified to be meaningful).
Also, I suspect the _word_ models of LLMs are not inherently the problem, they are just inefficient representations of world models.
Can we get to another level without a corresponding massive training set that demonstrates those abilities?
I don't think this is right: To calculate the optimal path, you do need to model human cognition.
At least, in the sense that finding the best path requires figuring out human concepts like "is the king vulnerable", "material value", "rook activity", etc. We have actual evidence of AlphaZero calculating those things in a way that is at least somewhat like humans do:
https://arxiv.org/abs/2111.09259
So even chess has "hidden state" in a significant sense: you can't play well without calculating those values, which are far from the surface.
I'm not sure there's a clear line between chess and poker like the author assumes.
I've found ChatGPT, especially "5.2 Thinking" to be very helpful in the relatively static world of fabrication. CNC cutting parameters for a new material? Gets me right in the neighborhood in minutes (not perfect, but good parameters to start). Identifying materials to compliment something I have to work with? Again, like a smart assistant. Same for generating lists of items I might be missing in prepping for a meeting or proposal.
But the high-level attorney in the family? Awful, and definitely in the ways identified (the biglaw firm is using MS derivative of OpenAI) - it thinks only statically.
BUT, it is also far worse than that for legal. And this is not a problem of dynamic vs. static or world model vs word model.
This problem is the ancient rule of Garbage In Garbage Out.
In any legal specialty there are a small set of top-level experts, and a horde of low-level pretenders who also hang out their shingle in the same field. Worse yet, the pretenders write a LOT of articles about the field to market themselves as experts. These self-published documents look good enough to non-lawyers to bring in business. But they are often deeply and even catastrophically wrong.
The problem is that LLMs ingest ALL of them with credulity, and LLM's cannot or do not tell the difference. So, when an LLM composes something, it is more likely to lie to you or fabricate some weird triangulation as it is to compose a good answer. And, unless you are an EXPERT lawyer, you will not be able to tell the difference until it is far too late and the flaw has already bitten you.
It is only one of the problems, and it's great to have an article that so clearly identifies it.