Fwiw nothing beats ‘implement the game logic in full (huge amounts of work) and with pruning on some heuristics look 50 moves ahead’. This is how chess engines work and how all good turn based game ai works.
I’ve tried throwing masses of game state data at latest models in pytorch. Unusable. It
Makes really dumb moves. In fact one big issue is that it often suggests invalid moves and the best way to avoid this is to implement the board game logic in full to validate it. At which point, why don’t i just do the above scan ahead X moves since i have to do the hard parts of manually building the world model anyway?
One area where current ai is helping is on the heuristics themselves for evaluating best moves when scanning ahead. You can input various game states and whether the player won the game or not in the end to train the values of the heuristics. You still need to implement the world model and look ahead to use those heuristics though! When you hear of neural networks being used for go or chess this is where they are used. You still need to build the world model and brute force scan ahead.
One path i do want to try more: In theory coding assistants should be able to read rulebooks and dynamically generate code to represent those rules. If you can do that part the rest should be easy. Ie. it could be possible to throw rulebooks at ai and it play the game. It would generate a world model from the rulebook via coding assistants and scan ahead more moves than humanly possible using that world model, evaluating to some heuristics that would need to be trained through trial and error.
Of course coding assistants aren’t at a point where you can throw rulebooks at them to generate an internal representation of game states. I should know. I just spent weeks building the game model even with a coding assistant.
You have a tiny, completely known, deterministic rule based 'world'. 'Reasoning' forwards for that is trivial.
Now try your approach for much more 'fuzzy', incomletely and ill defined environments, e.g. natural language production, and watch it go down in flames.
Different problems need different solutions. While current frontier llm's show surprising results in emergent shallow and linguistic reasoning, they are far away from deep abstract logical reasoning. A sota theorem prover otoh, can excel at that, but can still struggle to produce a coherent sentence.
I think most have always agreed that for certain tasks, an abstraction over which one can 'reason' is required. People differ in opinion over wether this faculty is to be 'crafted' in or wether it is possible to have it emerge implicitly and more robust from observations and interactions.
> Fwiw nothing beats ‘implement the game logic in full (huge amounts of work) and with pruning on some heuristics look 50 moves ahead’. This is how chess engines work and how all good turn based game ai works.
For board games this is mostly true. For turn based games in general, it is not. It's certainly not true to say "all good turn based game ai" works like this.
Turn based games where multiple "moves" are allowed per turn can very quickly have far too many branches to look ahead more than a very small number of turns. On board games you might have something like Warhammer, or Blood Bowl where there are many possible actions and order of actions within a turn matters.
For computer games you may Screeps [2] or the Lux multi-agent AI competitions [3] which both have multiple "units" per player, where each unit may have multiple possible actions. You can easily reach a combinatorial explosion where any attempt at modeling future states of the world fails and you have to fall back on pure heuristics.
I'm sure neural networks are a great tool here, but I don't know how the training would proceed effectively off "mere data"; too much of the data we have is incomplete, inaccurate, or outright fantasy or misinformation or out of the ordinary.
I could see this being the domain of fleets of robots, many different styles, compositions, materials, etc. Send ten robots in to survey a room - drones, crawlers, dogs, rollers, etc - they'll bang against things, knock things off shelves, illuminate corners, etc. The aggregate of their observations is the useful output, kinda like networked toddlers.
And yeah, unfortunately, sometimes this means you just need to send a swarm of robots to attack a city bus... or a bank... to "learn how things work." Or an internment camp. Don't get upset, guy, we're building a world model.
The end of westworld basically put forth that the only way we could stabilize the world is if we just destroyed it and moved it all to a parallel simulation. Since early attempts at world Modeling failed due to complexity of Outliers the only way ai could handle a world model was to just get rid of the real one.
People didn’t give the later seasons enough credit even if they didn’t rise tot he same dramatic effect as the first.
This is a very interesting article. The concept "run an experiment in your head and predict the outcome" is a capability that AIs must have to attain some kind of general intelligence. Anyway, read the article, it's great.
A footnote in the GPT-5 announcement was that you can now give OpenAI's API a context-free grammar that the LLM must follow. One way of thinking about this feature is that it's a user-defined world model. You could tell the model "the sky is" => "blue" for example.
Obviously you can't actually use this feature as a true world model. There's just too much stuff you have to codify, and basing such a system on tokens is inherently limiting.
The basic principle sounds like what we're looking for, though: a strict automata or rule set that steers the model's output reliably and provably. Perhaps a similar kind of thing that operates on neurons, rather than tokens? Hmm.
It's good to have this support in APIs but grammar constrained decoding has been around for quite a while, even before the contemporary LLM era (e.g. [1] is similar in spirit). Local vs global planning is a huge issue here though - if you enforce local constraints during decoding time, an LLM might be forced to make suboptimal token decisions. This could result in a "global" (i.e. all tokens) miss, where the probability of the constrained output is far lower than the probability of the optimal response (which may also conform to the grammar). Algorithms like beam search can alleviate this, but it's still difficult. This is one of the reasons that XML tags work better than JSON outputs - less constraints on "weird" tokens.
Why would this be? I'm probably missing something.
Don't these LLMs fundamentally work by outputting a vector of all possible tokens and strengths assigned to each, which is sampled via some form of sampler (that typically implements some softmax variant, and then picks a random output form that distribution), which now becomes the newest input token, repeat until some limit is hit, or an end of output token is selected?
I don't see why limiting that sampling to the set of valid tokens to fit a grammar should be harmful vs repeated generation until you get something that fits your grammar. (Assuming identical input to both processes.) This is especially the case if you maintain the relative probability of valid (per grammar) tokens in the restricted sampling. If one lets the relative probabilities change substantially, then I could see that giving worse results.
Now, I could certainly imagine blindsiding the LLM with output restrictions when it is expecting to be able to give a freeform response might give worse results than if one prompts it to give output in that format without restricting it. (Simply because forcing an output that is not natural and not a good fit for training can mean the LLM will struggle with creating good output.) I'd imagine the best results likely come from both textually prompting it to give output in your desired format, plus constraining the output to prevent it from accidentally going off the rails.
Oh, OpenAI finally added it? Structured generation has been available in things like llama.cpp and Instructor for a while, so I was wondering if they were going to get around to adding it.
In the examples I've seen, it's not something you can define an entire world model in, but you can sure constrain the immediate action space so the model does something sensible.
"You’re carrying around in your head a model of how the world works" (or so you thought) ... the real AI is in a) how fast you can realize it's changed and b) how fast you can adapt. This bit isn't being optimized, it's being dragged out.
> When researchers attempt(opens a new tab) to recover [something like] a coherent computational representation of an Othello game board they instead find [bags of heuristics]
Humans don't exactly have a full representation of board space in their head either. Notably, chess masters and amateurs can memorize completely random board positions as well as the other. I'd think neither could memorize 64 chess pieces in random positions on a board.
Some new ideas in world models are beginning to work. Using Gaussian splatting as a world model has had some recent success.[1] It's a representation that's somewhat tolerant of areas where there's not enough information. Some of the systems that generate video from images work this way.
A world model itself, in its particulars, isn't as important as the tacit understanding that the "world model" is necessarily incomplete and subordinate to the world itself, that there are sensory inputs from the world that would indicate you should adjust your world model, and the capacity and commitment to adjust that model in a way that maintains a level of coherence. With those things you don't need a complex model, you could start with a very simple but flexible model that would be adjusted over time by the system.
But I don't think we have a hint of a proposal for how to incorporate even the first part of that into our current systems.
A little bit disappointed that there was no mention of the Frame Problem [1], a major challenge with world models. The issue arises when you're building an AI agent with the ability to move through and act in the real world, updating its world model as it does so.
The challenge comes from the problem of finding a set of axioms that tell you how to make predictions about what changes a particular action will cause in the world. Naively, we might suppose that the laws of physics would be suitable axioms but this immediately turns out to be computationally intractable. So then we're stuck trying to find a set of heuristics, as alluded to in the article.
Without being a neuroscientist, I think it's likely that at least some of the axioms of our own world models (as human beings) are built into the structure of our brains, rather than being knowledge that we learn as we grow up. We know, for example, that our visual systems have a great deal of built-in assumptions about the way light works and how objects appear under different lighting conditions, a fact revealed to us by optical illusions such as the checker shadow illusion [2]. Building a complete set of heuristics such as this does not sound impossible, just somewhat obscure and unexplored as an engineering problem, and does not seem to be related whatsoever to currently popular means of building and training AI models.
I stumbled upon a lecture by Josh Tenenbaum (MIT) yesterday. Starting from minute 19 he talks about world models, and how we're nowhere near 'real AI'. This lecture was from 7 years ago, I wonder what a more recent take from him on this topic would be. https://youtu.be/TFyAEHk5asY?si=lZfjeF7t66FhkdSZ&t=1157
30 comments
[ 3.0 ms ] story [ 51.8 ms ] threadFwiw nothing beats ‘implement the game logic in full (huge amounts of work) and with pruning on some heuristics look 50 moves ahead’. This is how chess engines work and how all good turn based game ai works.
I’ve tried throwing masses of game state data at latest models in pytorch. Unusable. It Makes really dumb moves. In fact one big issue is that it often suggests invalid moves and the best way to avoid this is to implement the board game logic in full to validate it. At which point, why don’t i just do the above scan ahead X moves since i have to do the hard parts of manually building the world model anyway?
One area where current ai is helping is on the heuristics themselves for evaluating best moves when scanning ahead. You can input various game states and whether the player won the game or not in the end to train the values of the heuristics. You still need to implement the world model and look ahead to use those heuristics though! When you hear of neural networks being used for go or chess this is where they are used. You still need to build the world model and brute force scan ahead.
One path i do want to try more: In theory coding assistants should be able to read rulebooks and dynamically generate code to represent those rules. If you can do that part the rest should be easy. Ie. it could be possible to throw rulebooks at ai and it play the game. It would generate a world model from the rulebook via coding assistants and scan ahead more moves than humanly possible using that world model, evaluating to some heuristics that would need to be trained through trial and error.
Of course coding assistants aren’t at a point where you can throw rulebooks at them to generate an internal representation of game states. I should know. I just spent weeks building the game model even with a coding assistant.
You have a tiny, completely known, deterministic rule based 'world'. 'Reasoning' forwards for that is trivial.
Now try your approach for much more 'fuzzy', incomletely and ill defined environments, e.g. natural language production, and watch it go down in flames.
Different problems need different solutions. While current frontier llm's show surprising results in emergent shallow and linguistic reasoning, they are far away from deep abstract logical reasoning. A sota theorem prover otoh, can excel at that, but can still struggle to produce a coherent sentence.
I think most have always agreed that for certain tasks, an abstraction over which one can 'reason' is required. People differ in opinion over wether this faculty is to be 'crafted' in or wether it is possible to have it emerge implicitly and more robust from observations and interactions.
https://people.csail.mit.edu/brooks/papers/elephants.pdf
For board games this is mostly true. For turn based games in general, it is not. It's certainly not true to say "all good turn based game ai" works like this.
Turn based games where multiple "moves" are allowed per turn can very quickly have far too many branches to look ahead more than a very small number of turns. On board games you might have something like Warhammer, or Blood Bowl where there are many possible actions and order of actions within a turn matters.
For computer games you may Screeps [2] or the Lux multi-agent AI competitions [3] which both have multiple "units" per player, where each unit may have multiple possible actions. You can easily reach a combinatorial explosion where any attempt at modeling future states of the world fails and you have to fall back on pure heuristics.
[1]https://en.wikipedia.org/wiki/Blood_Bowl
[2]https://screeps.com/
[3]https://www.kaggle.com/competitions/lux-ai-season-2
I could see this being the domain of fleets of robots, many different styles, compositions, materials, etc. Send ten robots in to survey a room - drones, crawlers, dogs, rollers, etc - they'll bang against things, knock things off shelves, illuminate corners, etc. The aggregate of their observations is the useful output, kinda like networked toddlers.
And yeah, unfortunately, sometimes this means you just need to send a swarm of robots to attack a city bus... or a bank... to "learn how things work." Or an internment camp. Don't get upset, guy, we're building a world model.
Anybody wanna give me VC money to work on this?
People didn’t give the later seasons enough credit even if they didn’t rise tot he same dramatic effect as the first.
Evaluating AI's World Models (https://www.youtube.com/watch?v=hguIUmMsvA4)
Goes into details about several of the challenges discussed.
Obviously you can't actually use this feature as a true world model. There's just too much stuff you have to codify, and basing such a system on tokens is inherently limiting.
The basic principle sounds like what we're looking for, though: a strict automata or rule set that steers the model's output reliably and provably. Perhaps a similar kind of thing that operates on neurons, rather than tokens? Hmm.
[1] https://aclanthology.org/P17-2012/
Don't these LLMs fundamentally work by outputting a vector of all possible tokens and strengths assigned to each, which is sampled via some form of sampler (that typically implements some softmax variant, and then picks a random output form that distribution), which now becomes the newest input token, repeat until some limit is hit, or an end of output token is selected?
I don't see why limiting that sampling to the set of valid tokens to fit a grammar should be harmful vs repeated generation until you get something that fits your grammar. (Assuming identical input to both processes.) This is especially the case if you maintain the relative probability of valid (per grammar) tokens in the restricted sampling. If one lets the relative probabilities change substantially, then I could see that giving worse results.
Now, I could certainly imagine blindsiding the LLM with output restrictions when it is expecting to be able to give a freeform response might give worse results than if one prompts it to give output in that format without restricting it. (Simply because forcing an output that is not natural and not a good fit for training can mean the LLM will struggle with creating good output.) I'd imagine the best results likely come from both textually prompting it to give output in your desired format, plus constraining the output to prevent it from accidentally going off the rails.
In the examples I've seen, it's not something you can define an entire world model in, but you can sure constrain the immediate action space so the model does something sensible.
The worst part is that they namedrop many other tangentially related and/or outright fraudulent "ai experts" like Hinton, Bengio, and LeCun.
Humans don't exactly have a full representation of board space in their head either. Notably, chess masters and amateurs can memorize completely random board positions as well as the other. I'd think neither could memorize 64 chess pieces in random positions on a board.
Some new ideas in world models are beginning to work. Using Gaussian splatting as a world model has had some recent success.[1] It's a representation that's somewhat tolerant of areas where there's not enough information. Some of the systems that generate video from images work this way.
[1] https://katjaschwarz.github.io/ggs/
But I don't think we have a hint of a proposal for how to incorporate even the first part of that into our current systems.
The challenge comes from the problem of finding a set of axioms that tell you how to make predictions about what changes a particular action will cause in the world. Naively, we might suppose that the laws of physics would be suitable axioms but this immediately turns out to be computationally intractable. So then we're stuck trying to find a set of heuristics, as alluded to in the article.
Without being a neuroscientist, I think it's likely that at least some of the axioms of our own world models (as human beings) are built into the structure of our brains, rather than being knowledge that we learn as we grow up. We know, for example, that our visual systems have a great deal of built-in assumptions about the way light works and how objects appear under different lighting conditions, a fact revealed to us by optical illusions such as the checker shadow illusion [2]. Building a complete set of heuristics such as this does not sound impossible, just somewhat obscure and unexplored as an engineering problem, and does not seem to be related whatsoever to currently popular means of building and training AI models.
[1] https://plato.stanford.edu/entries/frame-problem/
[2] https://en.wikipedia.org/wiki/Checker_shadow_illusion
https://garymarcus.substack.com/p/how-o3-and-grok-4-accident...
While biological systems (or other physical agents) do need to model the world around them to be able to operate.