If I was smarter, I would have predicted that not only would everyone else figure out that world models are a critical step, but that as a direct consequence the term "world model" would lose all meaning. Maybe next time. That said, Le Cunn's concept in the blog post is the only one worthy of the title.
I always felt like one of reasons LLMs are so good is that they piggyback on the many years that have gone into developing language as an information representation/compression format. I don’t know if there’s anything similar a world model can take advantage of.
That being said there have been models which are pretty effective at other things that don’t use language, so maybe it’s a non issue.
With all due respect, AI is ultimately a capital game. World models aren’t where real B2B customer revenue comes from—at least compared to today’s LLMs; they’re mainly a better story for raising huge amounts of private capital. Hopefully they figure out how to build the next-gen AI architecture along the way.
I played with Marble yesterday, Fei-Fei/World Labs' new product.
It is the most impressed I've been with an AI experience since the first time I saw a model one-shot material code.
Sure, its an early product. The visual output reminds me a lot of early SDXL. But just look at what's happened to video in the last year and image in the last three. The same thing is going to happen here, and fast, and I see the vision for generative worlds for everything from gaming/media to education to RL/simulation.
The LLM grift is burned up, so this is the next thing. It has just enough new magic tricks to wow the VCs who don't really get what's going on here. I think this comment from the article says it all:
“Taking images and turning them into 3D environments using gaussian splats, depth and inpainting. Cool, but that’s a 3D GS pipeline, not a robot brain.”
Because they are smart enough to realize current LLM tech is nearing a dead end and cannot serve as a full AGI, even ignoring context and hallucination issues, without actual knowledge of the real world.
And the pendulum swings back toward representation. It is becoming clear that the LLM approach is not adequate to reach what John McCarthy called human-level intelligence:
Between us and human-level intelligence lie many problems. They can be summarized as that of succeeding in the "common-sense informatic situation". [1]
> It is becoming clear that the LLM approach is not adequate to reach what John McCarthy called human-level intelligence
Perhaps paradoxically, if/as this becomes a consensus view, I can be more excited about AI. I am an "AI skeptic" not in principle, but with respect to the current intertwined investment and hype cycles surrounding "AI".
Absent the overblown hype, I can become more interested in the real possibilities (both immediate, using existing ML methods; and the remote, theoretical capabilities follow from what I think about minds and computers in general) again.
I think when this blows over I can also feel freer to appreciate some of the genuinely cool tricks LLMs can perform.
Everytime I see LeCun talk about world models, I can’t help but think it is also just a tweak on the fundamentals of what is behind current LLM technology. In the end it’s still neural networks. To me, having to “teach” the model how physics works makes me think it can’t be true AGI either.
Danijar Hafner just left DeepMind. He's behind the Dreamer series of models which are IMO the most promising direction for world models anyone has come up with yet. I'm wondering where he's headed. Maybe he could end up at LeCun's startup?
In Dreamer 4 they are able to train an agent to play Minecraft with enough skill to obtain diamonds, without ever playing the game at all. Only by watching humans play. They first build a world model, then train the agent purely in scenarios imagined by the world model, requiring zero extra data or experience. Hopefully it's obvious how generating data from a world model might be useful for training agents in domains where we don't have datasets like the entire internet just sitting around ready-made for us to use.
LLMs are parameter based representations of linguistic representations of the world. Relative to robot predictive control problems, they are low dimensional and static. They are batch trained using supervised learning and are not designed to manage real time shifts in the external world or the reward space. They work because they operate in abstract, rule governed spaces like language and mathematics. They are ill suited to predictive control tasks. They are the IBM 360s of AI. Even so, they are astonishing achievements.
LeCun is right to say that continuous self supervised (hierarchical) learning is the next frontier, and that means we need world models. I'm not sure that JEPA is the right tool to get us past that frontier, but at the moment there are not a lot of alternatives on the table.
In "From Words to Worlds: Spatial Intelligence is AI’s Next Frontier" Li states directly "I’m not a philosopher", proceeds to make a philosophical argument that elevates visual perception as basis for evolution of intelligence.
I'm sure there are other valid reasons, but I think the most obvious one is that LLMs are not improving as fast as money asks for so we're moving to the next buzzword.
I think there is a lot of merit to this approach. Ultimately we live in a world guided by physics and macro-level perception driven by our senses and our own motor control. Of course newtonian physics is not the end all be all -- cell biology or quantum mechanics works on a very different level... but what is important here is that we know that human beings understand these things and make novel discoveries on these things using a thinking apparatus that was pre-trained on large scale newtonian physics. I've always found that even in advanced mathematics my mind always uses low level geometric analogies. So the "embeddings" or priors that can be obtained are probably much better than what can be done through text correlation as with LLMs. It's very different to learn the word bounce through observation of a physical model of a ball bouncing vs. seeing what other words it co-occurs with.
I’m trying to understand the conversation around “world models.” Why is Tesla’s FSD rarely mentioned in these discussions? Their system perceives, reasons, and acts in the physical world, and they train it using large-scale simulation/digital-twin environments. In what sense does FSD not count as a world model—or does it, and I’m missing something?
25 comments
[ 3.1 ms ] story [ 36.8 ms ] threadThat being said there have been models which are pretty effective at other things that don’t use language, so maybe it’s a non issue.
But only some groups have the ability to systematically encode language as writing.
Writing is a technological marvel.
It is the most impressed I've been with an AI experience since the first time I saw a model one-shot material code.
Sure, its an early product. The visual output reminds me a lot of early SDXL. But just look at what's happened to video in the last year and image in the last three. The same thing is going to happen here, and fast, and I see the vision for generative worlds for everything from gaming/media to education to RL/simulation.
“Taking images and turning them into 3D environments using gaussian splats, depth and inpainting. Cool, but that’s a 3D GS pipeline, not a robot brain.”
Between us and human-level intelligence lie many problems. They can be summarized as that of succeeding in the "common-sense informatic situation". [1]
And the search continues...
[1] https://www-formal.stanford.edu/jmc/human.pdf
Perhaps paradoxically, if/as this becomes a consensus view, I can be more excited about AI. I am an "AI skeptic" not in principle, but with respect to the current intertwined investment and hype cycles surrounding "AI".
Absent the overblown hype, I can become more interested in the real possibilities (both immediate, using existing ML methods; and the remote, theoretical capabilities follow from what I think about minds and computers in general) again.
I think when this blows over I can also feel freer to appreciate some of the genuinely cool tricks LLMs can perform.
In Dreamer 4 they are able to train an agent to play Minecraft with enough skill to obtain diamonds, without ever playing the game at all. Only by watching humans play. They first build a world model, then train the agent purely in scenarios imagined by the world model, requiring zero extra data or experience. Hopefully it's obvious how generating data from a world model might be useful for training agents in domains where we don't have datasets like the entire internet just sitting around ready-made for us to use.
https://danijar.com/project/dreamer4/
LeCun is right to say that continuous self supervised (hierarchical) learning is the next frontier, and that means we need world models. I'm not sure that JEPA is the right tool to get us past that frontier, but at the moment there are not a lot of alternatives on the table.