So this is some kind of attempt to make a Drake Equation [1] for AGI? That's more useful as a thought experiment than something claimed with scientific precision.
Mine came out to 0%, which is also personally where I'd put it.
But I think it's a question wrongly put.
We don't need AGI for current tools and their descendants to be extremely disruptive. For the longest time chess programs only mimicked one aspect of the way humans play the game (calculation) but they did it at such a superhuman level we were unable to beat them.
LLMs, computer vision tools, etc. don't need to be AGI to be able to displace large amounts of human jobs.
Having the discussion about if and in what ways we can prepare for the innovation that seems inevitable in that space seems like a much more prudent discussion than speculating about whether AGI will exist or not.
Don’t we all known someone who is very widely read, but makes the occasional error or mistake? Gets confused about something, or is asked about a discipline they don’t understand or haven’t researched very deeply? Maybe they misremember a date here or there, but are otherwise fairly intelligent? Maybe they work a data entry job making a middle class salary.
This is basically where ChatGPT is at. It’s a very widely read person with an excellent memory and quick mind. It’s probably smarter than the average human (certainly in breadth, often in depth), not to mention a comparison of it to the average human globally.
ChatGPT is smarter than the average human already. It doesn’t need agency or a soul to do so. We already have AGI, we just keep moving the goal posts.
The architecture is not AGI. Whether AGI can be acheived within it is perhaps an open question, but that the architecture itself does not constitute AGI is pretty clear.
2. General, meaning they are able transfer knowledge to solve arbitrary problems outside of their training domain; and
3. Intelligent in the sense of being able efficiently find efficient solutions to problems which exploit structure of the problem domain.
Artificial General Intelligence: A.G.I.
If you think AGI should mean something else, then that's because the goal posts have moved since the term was defined by GOFAI AI researchers some 2 or 3 decades ago. Some people in the 90's and 00's thought that merely having an AGI system (like ChatGPT) would result in runaway self-improvement leading to god-like singular powers that take over the world. Now some people have taken "AGI" to mean this fictional (and impossible) thing. That's a confusion on their part.
In general I agree with you, but I think we need to figure out how to do really long context lengths before that's really true. A sci fi AI with a real identity would be able to relate to everything that happened in its "life", not just the last 4k tokens.
Title isn't quite right... The article is estimating the probability of widespread/consumer accessible/cost effective AGI. But AGI can be transformative without being widespread.
Replacing all human functions is not nessecary for transformation. A single superhumam AI could cost 1e6 that of a single human, but still do transformative work if it's 1.5x smarter than us.
Edit: nevermind... This was an intentional definition of "transformative" made by the authors. Seems wrong to me, but they were up front about it. Not impressed though. Anyone can drum attention by using a non-standard definition of something. But of a cheap trick imo, but that's academia these days I guess... For reference, my definition of transformative would be "responsible for large scale changes to society". In other words, it doesn't need to happen at scale for impact to happen at scale. Replication of a solution is cheap once it is discovered.
It appears like a very poor analysis overall. These closing arguments suggest lack of deep thinking: “Consider Alphabet, the parent company of Google DeepMind. Alphabet is not constrained by financial capital: as of March 2023, Alphabet had $115B of cash on hand.202 Alphabet is also not constrained by human capital: as of March 2023, Alphabet had 191,000 employees.203 If Alphabet wanted to, it could easily hire more AI researchers and allocate more software engineers to AI research infrastructure. But it chooses not to. Why? This deliberate choice presumably implies that Alphabet management doesn’t believe there is enough ROI from hiring more smart people to accelerate progress on AGI research. To us, this suggests that Alphabet management is bearish on the likelihood of transformative AGI by 2043. We also see this as a stronger signal than public markets, as Alphabet management is likely much more informed about AGI progress than the average stock trader.
Another implicit signal of low likelihood of transformative AGI is the comparatively low valuation of AGI companies. Consider Anthropic, which reportedly raised money in early 2023 at a valuation of ~$4B. $4B is ~0.004% of world product each year. If transformative AGI is soon able to do most human work, we expect it to be worth multiples of today’s world product. Supposing transformative AGI is worth $100T/yr, and Anthropic has a 10% chance of capturing 10% of the value for an expected duration of 10 years, that would imply a valuation of $10T (before time discounting). $4B is less than 1/1,000th of $10T, which seems to imply that Anthropic and its investors do not collectively believe that Anthropic has a chance of inventing transformative AGI by 2043 and capturing its value.”
AI can be disruptive and innovative without being AGI.
I think AGI discussion is more useful for financing academic projects.
Lower cost of gpt and whisper are good enough for business and dictatorships. Infinite drones with the current human recognition capability can do the leg work (pun), no universal AGI soldier needed. Just current AI and batteries!
Let's say we get AGI that costs billions to run, can't do some simple things well, don't have good cheap robots to use, but discovers a way to lengthen the human life significantly and cure most illnesses. Pretty transformative if you ask me.
It's a fairy tale that only serves to play for time. Fewer and fewer control more and more, and they will not suddenly play nice once they no longer need the consent of workers and/or the public, or can force people without their consent. That never passed the smell test.
Life, as a whole, already is immortal. Immortal individuals will at best kick away the ladder, and stop the stream of life. So they can live forever, and nobody else gets born. Only the people who are the worst at living, and the least wise, would even want this. I noticed this as early as the late 90s... the people who were excited about immortality exuded all sorts of things, but decidedly not life. That hasn't changed.
It's like power, those who want it the most are the ones who should under no circumstances be allowed to have it.
This must be one of the stupidest article I have ever read.
The first page only shows they have absolutely no clue how statistics works. Of course if you consider every events in the universe completely independent the probability of any large combination of those will be very low. But it is of course not the case with all the conditions they describe, those are highly correlated and P(A|B)P(B)>P(A)P(B) for all of those.
Impossible to predict future if future depends on unpredictable factors.
I can't shake the feeling that recent developments like LLM's, ChatGPT & co are barking up the wrong tree. Or are missing a key piece of the puzzle. Or that vastly simpler (computationally cheaper) constructs with similar capabilities could be found. That in hindsight (say, 20y from now) we'll say "see, it was really easy!". That missing piece(s) were small but essential. And (perhaps) non-obvious right now.
Ray Kurzweil's argument is well reasoned & very convincing imho. Computing power will get there or already is. And our understanding of the architecture & function of the human brain is a steadily-completing picture. Not to mention brains of smaller creatures. All it takes is time.
Doubtful about timeframe in above bet. And efficiencies of artificial vs. biological brains remains to be seen. But yes, AGI will be achieved. Likely sooner than later.
Look at the power draw of the human brains versus any of the hardware running these models. It’s an order of magnitude difference. Often multiple orders of magnitude.
Part of the reason is a massive focus on GPUs and digital techniques. Some things like math operations are quite fast and accurate to do in a discrete sense when that matters.
The human brain is not discrete though. Nor is it really fully analog.
Specialized hardware beyond GPUs for parallel digital calculations will likely be needed to implement architectures that get actually closer to human levels of intelligence.
Deep down the brain is 100% discrete: Neurons are either firing or not. To me, the brain's biggest mistery is how it goes from this to doing all the analogue stuff, and ends up with our capacity to deal with symbols.
Sorry, but this is untrue; action potentials in neurons have extremely complex interplay with each other, including residual “soft” periods and chemically-induced changes in how they fire. Neurons don’t just “fire” or “not fire”, they adaptively change the strength of their firing constantly, unpredictably and continuously.
Yes, and there's things like feedback / reflection, self-modifying like behaviour, lossy memory, emotional state, tiredness, aging, loads of drugs & hormones to influence the process, etc, etc.
Buuuttt... it's possible that few if any of those things are needed to capture the essence of a brains' functionality.
Maybe it's simply a matter of size. Maybe some configuration tweaks. Perhaps a different architecture.
"China has stated plainly it intends to reunify (invade) Taiwan. A majority of its population supports invasion. Its military is preparing to be ready for invasion"
My opinion on this may be worthless since I have worked in AI since 1982, and I am guilty of enjoying living through several AI hype cycles.
After seeing surprising advances in techniques for deep models, then attention+transformer models, I think there is a lot of progress still to be made with cooperating LLMs. I have a simple example of this in the last book I wrote.
I have no idea what new ideas will work, but I would be very surprised if every 4 or 5 years new and fresh ideas don’t occur to achieve really good reasoning, counterfactual reasoning, etc. Anyway, putting the chance of transformative AGI below 1% for the next 20 years seems wrong to me.
43 comments
[ 2.3 ms ] story [ 64.9 ms ] thread[1] https://en.wikipedia.org/wiki/Drake_equation
https://www.tedsanders.com/agi-forecaster/
But I think it's a question wrongly put.
We don't need AGI for current tools and their descendants to be extremely disruptive. For the longest time chess programs only mimicked one aspect of the way humans play the game (calculation) but they did it at such a superhuman level we were unable to beat them.
LLMs, computer vision tools, etc. don't need to be AGI to be able to displace large amounts of human jobs.
Having the discussion about if and in what ways we can prepare for the innovation that seems inevitable in that space seems like a much more prudent discussion than speculating about whether AGI will exist or not.
This is basically where ChatGPT is at. It’s a very widely read person with an excellent memory and quick mind. It’s probably smarter than the average human (certainly in breadth, often in depth), not to mention a comparison of it to the average human globally.
ChatGPT is smarter than the average human already. It doesn’t need agency or a soul to do so. We already have AGI, we just keep moving the goal posts.
1. Artificial, AKA man-made;
2. General, meaning they are able transfer knowledge to solve arbitrary problems outside of their training domain; and
3. Intelligent in the sense of being able efficiently find efficient solutions to problems which exploit structure of the problem domain.
Artificial General Intelligence: A.G.I.
If you think AGI should mean something else, then that's because the goal posts have moved since the term was defined by GOFAI AI researchers some 2 or 3 decades ago. Some people in the 90's and 00's thought that merely having an AGI system (like ChatGPT) would result in runaway self-improvement leading to god-like singular powers that take over the world. Now some people have taken "AGI" to mean this fictional (and impossible) thing. That's a confusion on their part.
Not enough significant digits to take seriously. Now if they had said 0.39785% they might have some credibility.
Replacing all human functions is not nessecary for transformation. A single superhumam AI could cost 1e6 that of a single human, but still do transformative work if it's 1.5x smarter than us.
Edit: nevermind... This was an intentional definition of "transformative" made by the authors. Seems wrong to me, but they were up front about it. Not impressed though. Anyone can drum attention by using a non-standard definition of something. But of a cheap trick imo, but that's academia these days I guess... For reference, my definition of transformative would be "responsible for large scale changes to society". In other words, it doesn't need to happen at scale for impact to happen at scale. Replication of a solution is cheap once it is discovered.
I don't think it needs that at all. It just needs to 'Code' 1.01 times better than the average programmer to enter a self improvement cycle.
Lower cost of gpt and whisper are good enough for business and dictatorships. Infinite drones with the current human recognition capability can do the leg work (pun), no universal AGI soldier needed. Just current AI and batteries!
https://pluralistic.net/2021/02/17/reverse-centaur/#reverse-...
> Pretty transformative if you ask me.
It's a fairy tale that only serves to play for time. Fewer and fewer control more and more, and they will not suddenly play nice once they no longer need the consent of workers and/or the public, or can force people without their consent. That never passed the smell test.
Life, as a whole, already is immortal. Immortal individuals will at best kick away the ladder, and stop the stream of life. So they can live forever, and nobody else gets born. Only the people who are the worst at living, and the least wise, would even want this. I noticed this as early as the late 90s... the people who were excited about immortality exuded all sorts of things, but decidedly not life. That hasn't changed.
It's like power, those who want it the most are the ones who should under no circumstances be allowed to have it.
Also it would finish hour already crumbling system
The first page only shows they have absolutely no clue how statistics works. Of course if you consider every events in the universe completely independent the probability of any large combination of those will be very low. But it is of course not the case with all the conditions they describe, those are highly correlated and P(A|B)P(B)>P(A)P(B) for all of those.
Maybe your reading comprehension is the problem.
I can't shake the feeling that recent developments like LLM's, ChatGPT & co are barking up the wrong tree. Or are missing a key piece of the puzzle. Or that vastly simpler (computationally cheaper) constructs with similar capabilities could be found. That in hindsight (say, 20y from now) we'll say "see, it was really easy!". That missing piece(s) were small but essential. And (perhaps) non-obvious right now.
See https://longbets.org/1/
Ray Kurzweil's argument is well reasoned & very convincing imho. Computing power will get there or already is. And our understanding of the architecture & function of the human brain is a steadily-completing picture. Not to mention brains of smaller creatures. All it takes is time.
Doubtful about timeframe in above bet. And efficiencies of artificial vs. biological brains remains to be seen. But yes, AGI will be achieved. Likely sooner than later.
Part of the reason is a massive focus on GPUs and digital techniques. Some things like math operations are quite fast and accurate to do in a discrete sense when that matters.
The human brain is not discrete though. Nor is it really fully analog.
Specialized hardware beyond GPUs for parallel digital calculations will likely be needed to implement architectures that get actually closer to human levels of intelligence.
Buuuttt... it's possible that few if any of those things are needed to capture the essence of a brains' functionality.
Maybe it's simply a matter of size. Maybe some configuration tweaks. Perhaps a different architecture.
We simply don't know - yet.
I'm pretty sure all those things being dismissed are the essence of a brain.
Am I reading a paper or listening to Rogan?
>it's Drake equation
After seeing surprising advances in techniques for deep models, then attention+transformer models, I think there is a lot of progress still to be made with cooperating LLMs. I have a simple example of this in the last book I wrote.
I have no idea what new ideas will work, but I would be very surprised if every 4 or 5 years new and fresh ideas don’t occur to achieve really good reasoning, counterfactual reasoning, etc. Anyway, putting the chance of transformative AGI below 1% for the next 20 years seems wrong to me.
https://arbital.com/p/multiple_stage_fallacy/
Both previous discussions contain multiple independent refutations of the core claim (the argument stacks quite a few errors on top of each other).