Is a brain also inference? I know that an LLM works very different from the brain, but I wonder what makes a brain more capable of thinking. Is it the long term context? Is it a different type of neuron activation?
Not likely. Take with whatever grain of salt you'd like, but that was largely a property of development being academicized and subject to things like grant cycles, research topic fashionability trends, and institutional structure. It would be wrong to assume it's some baked in thing that's guaranteed to happen independent of how development looks.
We're past the point where there's a feasible argument that there is an AI winter coming.
The models work remarkably well for several classes of problem that seemed impossible a few years ago. They're not going away. There will still absolutely be a lot of ups and down and crazy stuff that happens in AI, but it won't be that AI almost completely stops being developed/funded for a decade or more. The biggest risk, I think, is regulatory capture; it's what Anthropic and OpenAI seem to be aiming for with their scaremongering about how capable and dangerous their models are. That'll put a damper on the industry for everyone except the companies that bribe the right people.
Left behind how? It's been transformers since 2016 and not much actual progress in basic architectures has happened 10 years later. I'm honestly struggling to see how you can be left behind in this field.
The article seems to define "smart" as being good at spatial awareness and navigating a body through 3D space and such. Thus, a mice is smarter than an LLM.
That's the first time in my life I hear this definition. Until now, the word "smart" has meant doing exactly the things LLMs do, and mice don't.
I guess it is a sign we are re-evaluating what makes humans special.
Was that ever solved? It seems that entire retort faded overnight, yet to my knowledge there was never any systematic analysis on cause or tokenizer change that fixed it. Maybe we just decided that this failure mode doesn't have any practical bearing given the existence of tool-use?
...I think it really is irrelevant, isn't it? The LLM gets words as tokens, not strings of letters. If you asked me how many of the letter s is in Mississippi, but said I'm not allowed to spell out the word in my head and count the letters, I don't think I could do it.
This isn't a great analogy, because part of the challenge would be preventing myself from picturing the spelling in my head. But my point is, the AI is not getting the words as letters. The correct solution is tool use.
LLMs can learn to do arithmetic (without tool use), and they can learn a mapping from tokens to the letter counts contained therein (you could imagine trivially training on synthetic data). So there doesn't seem to be any fundamental barrier.
Yann LeCun was saying 3 years ago that because token generation is auto-regressive, its mathematically impossible to generate a long stream of coherent tokens, because errors amplify exponentially.
and then models learned that they can back track and error correct
I think it was largely the introduction of tool calling that allowed models to mitigate the issue of errors amplifying exponentially since it allows the model to understand if what it generated is correct or has issues that it needs to address. This addresses the potential lack of or low quality of world model by being able to reference the current state of the world.
What argument, "a theory was wrong"? No, the inane central observation, the observation that a researcher was unable to predict a discovery before it was discovered, remains true despite the gratuitous insertion of a little bit of bullshit about AI learning.
I suppose it's additionally trying to imply something else, like "due to a pattern of researchers being unable to discover discoveries before they discover them, AGI is just around the corner".
For me, "smart" means doing things less based on instinct. Things humans can do but mice cannot, things mathematicians can do but normal people cannot, etc.
Considering the unit distance conjecture was disproved by OAI's model last month, I think maybe LLMs should count as "smart".
Besides "smart", the headline also conflates AI with LLMs. The real, non-clickbait title is "Yann LeCun, founder of AMI Labs, is developing a new AI system"
It is just so bizarre compared to my everyday experience also.
I never ask Opus or Fable a question and think "what a stupid response".
Quite the opposite. It has actually raised the bar of what I consider an intelligent response to my inquiry. So much so that most responses from humans on most subjects to most forms of inquiry seem stupid and not really thought out.
Everyone nowadays seems to only think of AI as LLMs or maybe also stable diffusion. People want to ban games with AI in them, when by definition every NPC is following some kind of AI algorithm.
They literally interview another person in it and mention a lot of other labs doing this kind of research including Google. Yes, he's the main starting interview but this is not really clickbait or a marketing piece.
> What's next is more AI spam-slop. I already noticed this on youtube.
I hope not but your observation on YouTube is spot-on. It's really frustrating. I've managed to keep good hygiene for my shorts feed. I practice zero-tolerance for braindead content; one strike is all it takes for the "never recommend" button.
But with the World Cup, the situation is just pandemonium. Football has always been breeding ground for low-effort content on the platform: unofficial highlights, cropped to death to avoid copyright, and always, for some reason, with a blaring background music. But now it's reached peak slop chaos: AI voiceovers, dubious anecdotes ("...and that kid, was Ronaldo"), and STILL the horribly blaring background music. The algorithm makes no distinction between quality and slop content of a topic. It's all just about the topic. So all it takes is for me to view a short related to football (even one of thoughtful commentary) for the slop to come in hordes.
At least I can now share in this forum's disdain for shorts. Kill that shit with fire man.
So we have that quote from the Oxford guy about explanations: "systems that can explain... You need models that can answer questions like: What matters? What causes what?", and then a mention of simulation of what the world looks like.
Fine, that describes theorizing.
But then a contradictory ending statement: "We're still going to need humans to figure out what questions to ask, what to build, what to create".
So that's moral theorizing. I don't think you can have one without the other. Then there's two more suggestions before the end of the article:
> smarter than us
> staff of assistants
Both of which are completely gratuitous assumptions. Why should its theories be better than established ones? Is it supposed to be a maverick hermit genius and come up with everything from first principles, or does it in fact participate in the existing world of ideas like a normal person? Then, being a normal person with moral theories, why would it take on the role of assistant rather than theorizing "I don't want to do that for you"?
40 comments
[ 4.7 ms ] story [ 59.9 ms ] threadThe models work remarkably well for several classes of problem that seemed impossible a few years ago. They're not going away. There will still absolutely be a lot of ups and down and crazy stuff that happens in AI, but it won't be that AI almost completely stops being developed/funded for a decade or more. The biggest risk, I think, is regulatory capture; it's what Anthropic and OpenAI seem to be aiming for with their scaremongering about how capable and dangerous their models are. That'll put a damper on the industry for everyone except the companies that bribe the right people.
That's the first time in my life I hear this definition. Until now, the word "smart" has meant doing exactly the things LLMs do, and mice don't.
I guess it is a sign we are re-evaluating what makes humans special.
Always has been: https://en.wikipedia.org/wiki/AI_effect
Tangentially: https://en.wikipedia.org/wiki/Moravec%27s_paradox
This isn't a great analogy, because part of the challenge would be preventing myself from picturing the spelling in my head. But my point is, the AI is not getting the words as letters. The correct solution is tool use.
and then models learned that they can back track and error correct
so much for "mathematically impossible..."
I very commonly see someone make some small mistake and end up going in the wrong direction, “accumulating stupid” as they go, sometimes for years.
You mean "Human developers learned ...", yes? Or was there really an all AI-driven, self-improving aspect to this?
I doubt that this was AI self-improvement.
I suppose it's additionally trying to imply something else, like "due to a pattern of researchers being unable to discover discoveries before they discover them, AGI is just around the corner".
For me, "smart" means doing things less based on instinct. Things humans can do but mice cannot, things mathematicians can do but normal people cannot, etc.
Considering the unit distance conjecture was disproved by OAI's model last month, I think maybe LLMs should count as "smart".
I never ask Opus or Fable a question and think "what a stupid response".
Quite the opposite. It has actually raised the bar of what I consider an intelligent response to my inquiry. So much so that most responses from humans on most subjects to most forms of inquiry seem stupid and not really thought out.
I hope not but your observation on YouTube is spot-on. It's really frustrating. I've managed to keep good hygiene for my shorts feed. I practice zero-tolerance for braindead content; one strike is all it takes for the "never recommend" button.
But with the World Cup, the situation is just pandemonium. Football has always been breeding ground for low-effort content on the platform: unofficial highlights, cropped to death to avoid copyright, and always, for some reason, with a blaring background music. But now it's reached peak slop chaos: AI voiceovers, dubious anecdotes ("...and that kid, was Ronaldo"), and STILL the horribly blaring background music. The algorithm makes no distinction between quality and slop content of a topic. It's all just about the topic. So all it takes is for me to view a short related to football (even one of thoughtful commentary) for the slop to come in hordes.
At least I can now share in this forum's disdain for shorts. Kill that shit with fire man.
Fine, that describes theorizing.
But then a contradictory ending statement: "We're still going to need humans to figure out what questions to ask, what to build, what to create".
So that's moral theorizing. I don't think you can have one without the other. Then there's two more suggestions before the end of the article:
> smarter than us
> staff of assistants
Both of which are completely gratuitous assumptions. Why should its theories be better than established ones? Is it supposed to be a maverick hermit genius and come up with everything from first principles, or does it in fact participate in the existing world of ideas like a normal person? Then, being a normal person with moral theories, why would it take on the role of assistant rather than theorizing "I don't want to do that for you"?