While I think there's obvious merit to their skepticism over the race towards agi, Sutskever's goal doesn't seem practical to me. As Dwarkesh also said, we reach to a safe and eventually perfect system by deploying it in public and iterating over it until optimal convergence dictated by users in a free market. Hence, I trust that Google, OpenAI or Anthropic will reach there, not SSI.
Some have been saying this for years now, but the consensus in the AI community and SV has been visibly shifting in the recent months.
Social contagion is astonishingly potent around ideas like this one, and this probably explains why the zeitgeist seems to be immovable for a time and then suddenly change.
All the frontier houses know this too. They also know it will be extremely difficult to raise more capital if their pitch is "we need to go back to research, which might return nothing at all."
Ilya did also acknowledge that these houses will still generate gobs of revenue, despite being at a dead end, so I'm not sure what the criticism is, exactly.
Everyone knows another breakthrough is required for agi to arrive; sama explicitly said this. Do you wait and sit on your hands until that breakthrough arrives? Or make a lot of money while skating to where the puck will be?
The frenzy around AI is to do with growth fueled cocaine capitalism seeking 'more' where rational minds can see that we don't have that much more runway left with our current mode of operation.
While the tech is useful, the mass amounts of money being shoveled into AI has more to do with the ever escaping mirage of a promised land where there will be an infinite amount of 'more'. For some people that means post scarcity, for others it means a world dominating AGI that achieves escape velocity against the current gridlock of geopolitics, for still others it means ejecting the pesky labour class and replacing all labour needs with AI and robots. Varied needs, but all perceived as urgent and inescapable by their vested interests.
I am somewhat relieved that we're not headed into the singularity just yet, I see it as way too risky given the current balance of power and stability across the planet. The outcome of ever accelerating tech progress at the expense of all other dimensions of wellbeing is not good for the majority of life here.
> Models look god-tier on paper:
> they pass exams
> solve benchmark coding tasks
> reach crazy scores on reasoning evals
Models don't look "god-tier" from benchmarks. Surely an 80% is not godlike. I would really like more human comparisons for these benchmarks to get a good idea of what an 80% means though.
I would not say that any model shows a "crazy" score on ARC-AGI.
I broadly have seen incremental improvements in benchmarks since 2020, mostly at a level I would believe to be below average human reasoning, but above average human knowledge. No one would call GPT-3 godlike and it is quite similar to modern models in benchmarks; it is not a difference of like 1% vs 90%. I think most people would consider gpt-3 to be closer to opus 4.5 than opus 4.5 is to a human.
I thought Ilya said we have more companies than we have ideas. He also noted that our current are resulting in models which are very good at benchmarks but have some problems with generalization (and gave a theory as to why).
But I don't recall him actually saying that the current ideas won't lead to AGI.
A problem is that the bulk of the people behind these labs are people that were conditioned from an early age to achieve high scores in standardized tests and conflate that with intelligence. Then apply that mentality to their models resulting in these leaderboards that nobody cares about.
This may not be AGI, but I think LLMs as is, with no other innovation, are capably enough for gigantic labor replacement with the right scaffolding. Even humans need a lot of scaffolding at scale (e.g. sales reps use CRMs even though they are generally intelligent). LLMs solve a “fuzzy input” problem that traditional software struggles with. I’m guessing something like 80% of current white collar jobs can be automated with LLMs plus scaffolding.
Didn’t we just see big pretraining gains from Google and likely Anthropic?
I like Dario’s view on this, we’ve seen this story before with deep learning. Then we progressively got better regularization, initialization, and activations.
I’m sure this will follow the same suit, the graph of improvement is still linear up and to the right
Say we discover a new architecture breakthrough like Yann LeCun's proposed JEPA. Won't scaling laws apply to it anyway?
Suppose training is so efficient that you can train state of the art AGI on a few GPUs. If it's better than current LLMs, there will be more demand/inference, which will require more GPUs and we are back at the same "add more gpus".
I find it hard to believe that we, as a humanity, will hit the wall of "we don't need more compute", no matter what the algorithms are.
I'm really annoyed by this. Not because I think Ilya is wrong, but because I think he is right. Because for years I think his current statement is right but for the same years I think his previous statement was wrong.
My stance hasn't changed, his has.
There's a big problem in that we reward those who hype, not merit. When the "era of scaling" happened there was a split. Those that claimed "Scale is all you need" and those that claimed "Scale is not enough". The former won, and I even seem to remember a bunch of people with T-shirts at NeurIPS with "scale is all you need" around that time.
So then, why are we again rewarding those same people when they change tunes? Their bet lost, sorry. I'm happy we tried scale and I'm glad we made progress, but at the same time many of us have been working outside the SIAYN paradigm and we struggled to get papers through review[0]. Scaling efforts led to lots of publications and citations, but you got far less by working outside that domain. And FFS, the reason most of you know Gary Marcus is because he was a vocal opposition to SIAYN and had enough initial clout. So as this tune is changing does the money shift towards us? Of course not.
I don't care about being vindicated, I care about trying to do research[1]. I don't care about the money, I care about trying to make AGI. Even Sutton has said that the Bitter Lesson was not about SIAYN!
So why I'm annoyed is that it seems we're going to let those who made big claims and fell short rather than those who correctly predicted the result. Why do we reward those who chase hype more than we reward those who got it right?
[0] a common criticism being "but does it work at scale?" or "needs more experiments". While these critiques/questions are legitimate they are out of place. Let us publish the small scale results first so that we can evidence our requests for more scale. Do you expect us to start at large scale first?
[1] I'm anonymous here, I don't care about the internet points. For the sake of this comment I might as well be any one of those Debbie Downers who pushed back against SIAYN and talked about the limits and how we shouldn't put all our eggs in one basket. There's thousands of us
23 comments
[ 2.8 ms ] story [ 40.7 ms ] threadSocial contagion is astonishingly potent around ideas like this one, and this probably explains why the zeitgeist seems to be immovable for a time and then suddenly change.
What high quality data sources are not already tapped?
Where does the next 1000x flops come from?
Ilya did also acknowledge that these houses will still generate gobs of revenue, despite being at a dead end, so I'm not sure what the criticism is, exactly.
Everyone knows another breakthrough is required for agi to arrive; sama explicitly said this. Do you wait and sit on your hands until that breakthrough arrives? Or make a lot of money while skating to where the puck will be?
While the tech is useful, the mass amounts of money being shoveled into AI has more to do with the ever escaping mirage of a promised land where there will be an infinite amount of 'more'. For some people that means post scarcity, for others it means a world dominating AGI that achieves escape velocity against the current gridlock of geopolitics, for still others it means ejecting the pesky labour class and replacing all labour needs with AI and robots. Varied needs, but all perceived as urgent and inescapable by their vested interests.
I am somewhat relieved that we're not headed into the singularity just yet, I see it as way too risky given the current balance of power and stability across the planet. The outcome of ever accelerating tech progress at the expense of all other dimensions of wellbeing is not good for the majority of life here.
every LLM easily misaligned, "deceived to deceive" and whatnot and they want to focus on adding MORE ATTACK SURFACE???
and throw more CPU at it?
This is glorious.
time to invest in the pen & paper industry!
I would not say that any model shows a "crazy" score on ARC-AGI.
I broadly have seen incremental improvements in benchmarks since 2020, mostly at a level I would believe to be below average human reasoning, but above average human knowledge. No one would call GPT-3 godlike and it is quite similar to modern models in benchmarks; it is not a difference of like 1% vs 90%. I think most people would consider gpt-3 to be closer to opus 4.5 than opus 4.5 is to a human.
But I don't recall him actually saying that the current ideas won't lead to AGI.
> The industry is already operating at insane scale.
Sounds a lot like "640K ought to be enough for anybody", or "the market can stay irrational longer than you can stay solvent".
I don't doubt this person knows how things should go but I also don't doubt this will get bigger before it gets smaller.
I like Dario’s view on this, we’ve seen this story before with deep learning. Then we progressively got better regularization, initialization, and activations.
I’m sure this will follow the same suit, the graph of improvement is still linear up and to the right
Watch the original Sutskever interview: https://www.youtube.com/watch?v=aR20FWCCjAs
And LeCun: https://www.youtube.com/watch?v=4__gg83s_Do
Suppose training is so efficient that you can train state of the art AGI on a few GPUs. If it's better than current LLMs, there will be more demand/inference, which will require more GPUs and we are back at the same "add more gpus".
I find it hard to believe that we, as a humanity, will hit the wall of "we don't need more compute", no matter what the algorithms are.
Earlier:
Ilya Sutskever: We're moving from the age of scaling to the age of research
https://news.ycombinator.com/item?id=46048125
And one of the recent LeCun discussions:
https://news.ycombinator.com/item?id=45897271
This does not mean he's not an accomplished and very talented researcher.
LeCun was sacked from Meta.
Not sure if it's wise to listen to their advice ...
My stance hasn't changed, his has.
There's a big problem in that we reward those who hype, not merit. When the "era of scaling" happened there was a split. Those that claimed "Scale is all you need" and those that claimed "Scale is not enough". The former won, and I even seem to remember a bunch of people with T-shirts at NeurIPS with "scale is all you need" around that time.
So then, why are we again rewarding those same people when they change tunes? Their bet lost, sorry. I'm happy we tried scale and I'm glad we made progress, but at the same time many of us have been working outside the SIAYN paradigm and we struggled to get papers through review[0]. Scaling efforts led to lots of publications and citations, but you got far less by working outside that domain. And FFS, the reason most of you know Gary Marcus is because he was a vocal opposition to SIAYN and had enough initial clout. So as this tune is changing does the money shift towards us? Of course not.
I don't care about being vindicated, I care about trying to do research[1]. I don't care about the money, I care about trying to make AGI. Even Sutton has said that the Bitter Lesson was not about SIAYN!
So why I'm annoyed is that it seems we're going to let those who made big claims and fell short rather than those who correctly predicted the result. Why do we reward those who chase hype more than we reward those who got it right?
[0] a common criticism being "but does it work at scale?" or "needs more experiments". While these critiques/questions are legitimate they are out of place. Let us publish the small scale results first so that we can evidence our requests for more scale. Do you expect us to start at large scale first?
[1] I'm anonymous here, I don't care about the internet points. For the sake of this comment I might as well be any one of those Debbie Downers who pushed back against SIAYN and talked about the limits and how we shouldn't put all our eggs in one basket. There's thousands of us