Generative world models seem to be doing ok. Dreamer V4 looks promising. I’m not 100% sold on the necessity of EBMs.
Also I’m skeptical that self-supervised learning is sufficient for human level learning. Some of our ability is innate. I don’t believe it’s possible for statistical methods to learn language from raw audiovisual data the way children can.
Human DNA has under 1GB of information content in it. Most of which isn't even used in the brain. And the brain doesn't have a mechanism to read data out from the DNA efficiently.
This puts a severe limit on how much "innate knowledge" a human can possibly have.
Sure, human brain has a strong inductive bias. It also has a developmental plan, and it follows that plan. It guides its own learning, and ends up being better at self-supervised learning than even the very best of our AIs. But that guidance, that sequencing and that bias must all be created by the rules encoded in the DNA, and there's only this much data in the DNA.
It's quite possible that the human brain has a bunch of simple and clever learning tricks that, if we pried out and applied to our AIs, would give us x100 the learning rate and x1000 the sample efficiency. Or it could be that a single neuron in the human brain is worth 10000 neurons in an artificial neural network, and thus, the biggest part of the "secret" of human brain is just that it's hilariously overparameterized.
You’d have to explain where that innate knowledge is stored though. The entire human genome is less than a GB if I remember correctly. Some of that being allocated to ”priors” for neural circuit development seems reasonable, but it can’t be very detailed across everything a brain does. The rest of the body needs some bytes too.
This seems like the same exact talk LeCun has been giving for years, basically pushing JEPA, world models, and attacking contemporary LLMs. Maybe he’s right but it also seems like he’s wrong in terms of timing or impact. LLMs have been going strong for longer than he expected, and providing more value than expected.
Agree with LeCun that current ai doesn’t exhibit anything close to actual intelligence.
I think the solution lies into cracking the core algorithms used by nature to build the brain. Too bad it’s such an inscrutable hairball of analog spaghetti code.
LeCun has been giving the same talk with literally the exact same slides for the past 3 years. JEPA still hasn't delivered despite FAIR's substantial backing.
> Some people were technical, but they didn't do technical work for many months, or longer, and now are no longer technical, they fell behind, but still think they are.
I think that LeCun has correctly identified that LLM is only one type of intelligence and that AGI/AMI needs to combine multiple other types … hierarchical goal setting, attention/focus management, and so on.
Seems that he is able to garner support for his ideas and to make progress at the leading edge - yes a little bit hard to take the “I know better” style, but then many innovations are driven by narcissism.
12 comments
[ 2.9 ms ] story [ 27.0 ms ] threadAlso I’m skeptical that self-supervised learning is sufficient for human level learning. Some of our ability is innate. I don’t believe it’s possible for statistical methods to learn language from raw audiovisual data the way children can.
This puts a severe limit on how much "innate knowledge" a human can possibly have.
Sure, human brain has a strong inductive bias. It also has a developmental plan, and it follows that plan. It guides its own learning, and ends up being better at self-supervised learning than even the very best of our AIs. But that guidance, that sequencing and that bias must all be created by the rules encoded in the DNA, and there's only this much data in the DNA.
It's quite possible that the human brain has a bunch of simple and clever learning tricks that, if we pried out and applied to our AIs, would give us x100 the learning rate and x1000 the sample efficiency. Or it could be that a single neuron in the human brain is worth 10000 neurons in an artificial neural network, and thus, the biggest part of the "secret" of human brain is just that it's hilariously overparameterized.
I think the solution lies into cracking the core algorithms used by nature to build the brain. Too bad it’s such an inscrutable hairball of analog spaghetti code.
Humans are not intrinsically machines. Through the education system and so on, humans are taught to somewhat behave as such.
> Some people were technical, but they didn't do technical work for many months, or longer, and now are no longer technical, they fell behind, but still think they are.
Seems that he is able to garner support for his ideas and to make progress at the leading edge - yes a little bit hard to take the “I know better” style, but then many innovations are driven by narcissism.