And self sustained nuclear fusion is 20 years away, perpetually. On which evidence can he affirm a timeline for AGI when we can barely define intelligence?
AGI is already here if you shift some goal posts :)
From skimming the conversation it seems to mostly revolve around LLMs (transformer models) which is probably not going to be the way we obtain AGI to begin with, frankly it is too simple to be AGI, but the reason why there's so much hype is because it is simple to begin with so really I don't know.
It's funny how there's such a pervasive cynicism about AI in the developer community, yet everyone is still excited about vibe coding. Strange times...
AI has now been revealed to the masses. When AGI arrives most people will barely notice. It will just feel like slightly better LLMs to them. They will have already cemented notions of how it works and how it affects their lives.
"The question of whether a computer can think is no more interesting than the question of whether a submarine can swim." - Edsger Dijkstra
The debate about AGI is interesting from a philosophical perspective, but from a practical perspective AI doesn't need to get anywhere close to AGI to turn the world upside down.
Am I dating myself by thinking Kurzweil is still relevant?
2029: Human-level AI
2045: The Singularity - machine intelligence 1 billion times more powerful than all human intelligence
Based on exponential growth in computing. He predicts we'll merge with AI to transcend biological limits.
His track record is mixed, but 2029 looks more credible post-GPT-5. The 2045 claim remains highly speculative.
> He predicts we'll merge with AI to transcend biological limits.
The merge with a machine 1 million times more intelligent than us is the same as letting AI use our bodies. I'd rather live in cave. Iirc, the 7th episode of Black Mirror starts with this plot line.
Following the comments here, yes: AGI is the new Cold Fusion.
However, don't let the bandwagon ( from either side ) cloud your judgment. Even warm fusion or any fusion at all is still very useful and it's here to stay.
This whole AGI and "the future" thing is mostly a VC/Banks and shovel sellers problem. A problem that has become ours too because the ridiculous amounts of money "invested", so even warm fusion is not enough from an investment vs expectations perspective.
They are already playing musical money chairs, unfortunately we already know who's going to pay for all of this "exuberance" in the end.
I hope this whole thing crashes and burns as soon as possible, not because I don't "believe" in AI, but because people have been absolutely stupid about it. The workplace has been unbearable with all this stupidity and amounts of fake "courage" about every single problem and the usual judgment of the value of work and knowledge your run-of-the-mill dipshit manager has now.
I have massive respect for Andrej, my first encounter with "him" was following his tutorials/notes when he was a grad student/tutor for AI/ML.
I was a lot disappointed when he went to work for Tesla, and I think that he had some achievement there, butnot nearly the impact I believe he potentially has.
His switch (back?) to OpenAI was, in my mind, much more in keeping with where his spirit really lies.
So, with that in mind, maybe I've drunk too much kool aid, maybe not. But I'm in agreement with him, the LLMs are not AGI, they're bloody good natural language processors, but they're still regurgitating rather than creating.
Essentially that's what humans do, we're all repeating what our education/upbringing told us worked for our lives.
But we all recognise that what we call "smart" is people recognising/inventing ways to do things that did not exist before. In some cases its about applying a known methodset to a new problem, in others its about using a substance/method in a way that other substances/methodsets are used, but the different substance/methodset produces something interesting (think, oh instead of boiling food in water, we can boil food in animal fats... frying)
AI/LLMs cannot do this, not at all. That spark of creativity is agonisingly close, but, like all 80/20 problems, is likely still a while away.
The timeline (10 years) - it was the early 2010s (over 10 years ago now) that the idea of backward propagation, after a long AI winter, finally came of age. It (the idea) had been floating about since at least the 1970s. And that ushered in the start of our current revolution, that and "Deep Learning" (albeit with at least another AI winter spanning the last 4 or 5 years until LLMs arrived)
So, given that timeline, and the restraints in the currrent technology, I think that Andrej is on the right track, and it will be interesting to see where we are in ten years time.
You can make this bet functional if you really believe it, which you of course really don't. If you actually do then I can introduce you to some people happy to take your money in perpetuity.
>I would bet all of my assets of my life that AGI will not be seen in the lifetime of anyone reading this message right now.
That includes anyone reading this message long after the lives of those reading it on its post date have ended.
By almost any definition available during the 90s GPT-5 Thinking/Pro would pretty much qualify. The idea that we are somehow not going to make any progress for the next century seems absurd. Do you have any actual justification for why you believe this? Every lab is saying they see a clear path to improving capabilities and theres been nothing shown by any research I'm aware of to justify doubting that.
genuinely curious to hear your reasoning for why this is the case. i'm always somewhere between bemused and annoyed opening the daily HN thread about AGI and seeing everyone's totally unfounded confidence in their predictions.
my position is I have no idea what is going to happen.
It's about the same as betting all life savings on nuclear war not breaking out in our lifetime. If AI gets created, we are toast and those assets won't be worth anything.
How certain are you of this really? I'd take this bet with you.
You're saying that we won't achieve AGI in ~80 years, or roughly 2100, equivalent to the time since the end WW2.
To quote Shane Legg from 2009:
"It looks like we’re heading towards 10^20 FLOPS before 2030, even if things slow down a bit from 2020 onwards. That’s just plain nuts. Let me try to explain just how nuts: 10^20 is about the number of neurons in all human brains combined. It is also about the estimated number of grains of sand on all the beaches in the world. That’s a truly insane number of calculations in 1 second."
Are humans really so incompetent that we can't replicate what nature produced through evolutionary optimization with more compute than in EVERY human brain?
I agree with you, and I think that's where Polymarket or similar could be used to see if these people would put your money where their mouth is (my guess is that most won't).
But first we would need a precise definition of AGI. They may be able to come with a definition that makes the bet winnable for them.
I'd bet the other way because I think Moore's law like advances in compute will make things much easier for researchers.
Like I was watching Hinton explain LLMs to Jon Stewart and they were saying they came up with the algorithm in 1986 but then it didn't really work for the decades until now because the hardware wasn't up to it (https://youtu.be/jrK3PsD3APk?t=1899)
If things were 1000x faster you could semi randomly try all sorts of arrangements of neural nets to see which think better.
Agreed. But I'd also be willing to bet big, that the cycle of "new AI breakthrough is made, AI bubble ensues and hypesters claim AGI is just around the corner for several years, bubble bursts, all quiet on the AI front for a decade or two" continues beyond the lifetime of anyone reading this message right now.
"When you get a demo and something works 90% of the time, that’s just the first nine. Then you need the second nine, a third nine, a fourth nine, a fifth nine. While I was at Tesla for five years or so, we went through maybe three nines or two nines. I don’t know what it is, but multiple nines of iteration. There are still more nines to go.
Even without AGI, current LLMs will change society in ways we can't yet imagine. And this is both good and bad. Current LLMs are just a different type of automation, not mechanical like control systems and robots, but intellectual. They don't have to be able to think independently, but as long as they automate some white-collar tasks, they will change how the rest of society works. The simple transistor is just a small electronic component that is a better version of a tube, and yet it changed everything in a few decades. How will the world change because of LLMs? I have no idea, but I know it doesn't have to be AGI to cause a lot of upheaval.
>What takes the long amount of time and the way to think about it is that it’s a march of nines. Every single nine is a constant amount of work. Every single nine is the same amount of work. When you get a demo and something works 90% of the time, that’s just the first nine. Then you need the second nine, a third nine, a fourth nine, a fifth nine. While I was at Tesla for five years or so, we went through maybe three nines or two nines. I don’t know what it is, but multiple nines of iteration. There are still more nines to go.
I think this is an important way of understanding AI progress. Capability improvements often look exponential on a particular fixed benchmark, but the difficulty of the next step up is also often exponential, and so you get net linear improvement with a wider perspective.
if it works 90% of the time that means it fails 10% of the time, to get to 1% failure rate is a 10x improvement and from 1% failure rate to a 0.1% failure rate is also a 10x improvement
First time being hearing it be called "march of nines", did Tesla make the term, I thought it was an Amazon thing
I have a very surface level understanding of AI, and yet this always seemed obvious to me. It's almost a fundamental law of the universe that complexity of any kind has a long tail. So you can get AI to faithfully replicate 90% of a particular domain skill. That's phenomenal, and by itself can yield value for companies. But the journey from 90%-100% is going to be a very difficult march.
The thing about this, though - cars have been built before. We understand what's necessary to get those 9s. I'm sure there were some new problems that had to be solved along the way, but fundamentally, "build good car" is known to be achievable, so the process of "adding 9s" there makes sense.
But this method of AI is still pretty new, and we don't know it's upper limits. It may be that there are no more 9s to add, or that any more 9s cost prohibitively more. We might be effectively stuck at 91.25626726...% forever.
Not to be a doomer, but I DO think that anyone who is significantly invested in AI really has to have a plan in case that ends up being true. We can't just keep on saying "they'll get there some day" and acting as if it's true. (I mean you can, just not without consequences.)
FWIW, Karpathy literally says, multiple times, that he thinks we never left the exponential - that all human progress over last 4+ centuries averages out to that smooth ~2% growth rate exponential curve, that electricity and computing and AI are just ways we keep it going, and we'll continue on that curve for the time being.
It's the major point of contention between him and the host (who thinks growth rate will increase).
The thing is, the example of the "march of nines" is self-driving cars. These deal with roads and roads are interface between the chaos of the overall world and a system that has quite well-defined rules.
I can imagine other task on a human/rules-based "frontier" would have a similar quality. But I think there are others that are going to be inaccessible entirely "until AGI" (or something). Humanoid robots moving freely in human society would an example I think.
I think the point Andrej was making here is that in some areas, such as self driving, the cost of failure is extremely high (maybe death), so 99.9% reliable doesn't cut it, and therefore doesn't mean you are almost done, or have done 99.9% of the work. It's "The last 10% is 90% of the work" recursively applied.
He was also pointing out that the same high cost of failure consideration applies to many software systems (depending on what they are doing/controlling). We may already be at the level where AI coding agents are adequate for some less critical applications, but yet far away from them being a general developer replacement. I see software development as something that uses closer to 100% of your brain than 10% - we may well not see AI coding agents approach human reliability levels until we have human level AGI.
The AI snake oil salesmen/CEOs like to throw out competitive coding or math olympiad benchmarks as if they are somehow indicative of the readiness of AI for other tasks, but reliability matters. Nobody dies or loses millions of dollars if you get a math problem wrong.
211 comments
[ 5.1 ms ] story [ 107 ms ] threadAnd real AI is probably like fusion. Its always 10 years away.
From skimming the conversation it seems to mostly revolve around LLMs (transformer models) which is probably not going to be the way we obtain AGI to begin with, frankly it is too simple to be AGI, but the reason why there's so much hype is because it is simple to begin with so really I don't know.
But nothing will make grifters richer than promising it's right around the corner.
AI has now been revealed to the masses. When AGI arrives most people will barely notice. It will just feel like slightly better LLMs to them. They will have already cemented notions of how it works and how it affects their lives.
The debate about AGI is interesting from a philosophical perspective, but from a practical perspective AI doesn't need to get anywhere close to AGI to turn the world upside down.
2029: Human-level AI
2045: The Singularity - machine intelligence 1 billion times more powerful than all human intelligence
Based on exponential growth in computing. He predicts we'll merge with AI to transcend biological limits. His track record is mixed, but 2029 looks more credible post-GPT-5. The 2045 claim remains highly speculative.
The merge with a machine 1 million times more intelligent than us is the same as letting AI use our bodies. I'd rather live in cave. Iirc, the 7th episode of Black Mirror starts with this plot line.
However, don't let the bandwagon ( from either side ) cloud your judgment. Even warm fusion or any fusion at all is still very useful and it's here to stay.
This whole AGI and "the future" thing is mostly a VC/Banks and shovel sellers problem. A problem that has become ours too because the ridiculous amounts of money "invested", so even warm fusion is not enough from an investment vs expectations perspective.
They are already playing musical money chairs, unfortunately we already know who's going to pay for all of this "exuberance" in the end.
I hope this whole thing crashes and burns as soon as possible, not because I don't "believe" in AI, but because people have been absolutely stupid about it. The workplace has been unbearable with all this stupidity and amounts of fake "courage" about every single problem and the usual judgment of the value of work and knowledge your run-of-the-mill dipshit manager has now.
I was a lot disappointed when he went to work for Tesla, and I think that he had some achievement there, butnot nearly the impact I believe he potentially has.
His switch (back?) to OpenAI was, in my mind, much more in keeping with where his spirit really lies.
So, with that in mind, maybe I've drunk too much kool aid, maybe not. But I'm in agreement with him, the LLMs are not AGI, they're bloody good natural language processors, but they're still regurgitating rather than creating.
Essentially that's what humans do, we're all repeating what our education/upbringing told us worked for our lives.
But we all recognise that what we call "smart" is people recognising/inventing ways to do things that did not exist before. In some cases its about applying a known methodset to a new problem, in others its about using a substance/method in a way that other substances/methodsets are used, but the different substance/methodset produces something interesting (think, oh instead of boiling food in water, we can boil food in animal fats... frying)
AI/LLMs cannot do this, not at all. That spark of creativity is agonisingly close, but, like all 80/20 problems, is likely still a while away.
The timeline (10 years) - it was the early 2010s (over 10 years ago now) that the idea of backward propagation, after a long AI winter, finally came of age. It (the idea) had been floating about since at least the 1970s. And that ushered in the start of our current revolution, that and "Deep Learning" (albeit with at least another AI winter spanning the last 4 or 5 years until LLMs arrived)
So, given that timeline, and the restraints in the currrent technology, I think that Andrej is on the right track, and it will be interesting to see where we are in ten years time.
That includes anyone reading this message long after the lives of those reading it on its post date have ended.
Which of course raises the interesting question of how I can make good on this bet.
Example: better than average human across many thinking tasks is done.
By almost any definition available during the 90s GPT-5 Thinking/Pro would pretty much qualify. The idea that we are somehow not going to make any progress for the next century seems absurd. Do you have any actual justification for why you believe this? Every lab is saying they see a clear path to improving capabilities and theres been nothing shown by any research I'm aware of to justify doubting that.
my position is I have no idea what is going to happen.
If you're really serious about it put the money into a prediction market. Poly market has multiple AGI bets.
You're saying that we won't achieve AGI in ~80 years, or roughly 2100, equivalent to the time since the end WW2.
To quote Shane Legg from 2009:
"It looks like we’re heading towards 10^20 FLOPS before 2030, even if things slow down a bit from 2020 onwards. That’s just plain nuts. Let me try to explain just how nuts: 10^20 is about the number of neurons in all human brains combined. It is also about the estimated number of grains of sand on all the beaches in the world. That’s a truly insane number of calculations in 1 second."
Are humans really so incompetent that we can't replicate what nature produced through evolutionary optimization with more compute than in EVERY human brain?
I agree with you, and I think that's where Polymarket or similar could be used to see if these people would put your money where their mouth is (my guess is that most won't).
But first we would need a precise definition of AGI. They may be able to come with a definition that makes the bet winnable for them.
Like I was watching Hinton explain LLMs to Jon Stewart and they were saying they came up with the algorithm in 1986 but then it didn't really work for the decades until now because the hardware wasn't up to it (https://youtu.be/jrK3PsD3APk?t=1899)
If things were 1000x faster you could semi randomly try all sorts of arrangements of neural nets to see which think better.
jb1991 says >"Which of course raises the interesting question of how I can make good on this bet."<
Have children...
"When you get a demo and something works 90% of the time, that’s just the first nine. Then you need the second nine, a third nine, a fourth nine, a fifth nine. While I was at Tesla for five years or so, we went through maybe three nines or two nines. I don’t know what it is, but multiple nines of iteration. There are still more nines to go.
That’s why these things take so long."
I think this is an important way of understanding AI progress. Capability improvements often look exponential on a particular fixed benchmark, but the difficulty of the next step up is also often exponential, and so you get net linear improvement with a wider perspective.
First time being hearing it be called "march of nines", did Tesla make the term, I thought it was an Amazon thing
But this method of AI is still pretty new, and we don't know it's upper limits. It may be that there are no more 9s to add, or that any more 9s cost prohibitively more. We might be effectively stuck at 91.25626726...% forever.
Not to be a doomer, but I DO think that anyone who is significantly invested in AI really has to have a plan in case that ends up being true. We can't just keep on saying "they'll get there some day" and acting as if it's true. (I mean you can, just not without consequences.)
It's the major point of contention between him and the host (who thinks growth rate will increase).
I can imagine other task on a human/rules-based "frontier" would have a similar quality. But I think there are others that are going to be inaccessible entirely "until AGI" (or something). Humanoid robots moving freely in human society would an example I think.
He was also pointing out that the same high cost of failure consideration applies to many software systems (depending on what they are doing/controlling). We may already be at the level where AI coding agents are adequate for some less critical applications, but yet far away from them being a general developer replacement. I see software development as something that uses closer to 100% of your brain than 10% - we may well not see AI coding agents approach human reliability levels until we have human level AGI.
The AI snake oil salesmen/CEOs like to throw out competitive coding or math olympiad benchmarks as if they are somehow indicative of the readiness of AI for other tasks, but reliability matters. Nobody dies or loses millions of dollars if you get a math problem wrong.