This is our future: short term growth in economy and stock markets followed by massive layoffs followed by economic crash followed by societal collapse.
If you think this is all BS and you will be always smarter than any machine, realize that raw computing power grows by 10^6 every ~25 years. AI seems to follow the same growth.
If you think you are safe, read what Geoffrey Hinton, one of the fathers of ML, said about 10-20% chance of AI leading to human extinction in 30 years. Or read about Sam Altman building his bunker in New Zealand while espousing AI utopia.
If this doesn't give you shivers, enjoy your blissful ignorance.
>raw computing power grows by 10^6 every ~25 years
That is really not true anymore unless there will be a new technology that can do much better than silicon electronics. Doesn't mean that were safe, but it's not that clear.
Edit: to expand, a single transistor is made of a pretty pedestrian number of atoms these days. The gate pitch of TSMC "3 nm" is about 40 nm (not all of which is transistor) and silicon atoms are about 0.2 nm apart. So you have, let's say, 30^3 / 0.2^3 ~= 3 million atoms in a transistor. There is no room for another 10^6 improvement because a transistor with just a few atoms probably wouldn't work, the thermal noise would dominate it. Paraphrasing the famous Feynman quote, there isn't much room at the botton anymore, now.
Your point is true but it is too narrow. A similar argument was bandied about regarding the CPU clock which hit a wall which was defeated with multi-cores and parallelization. The atom wall you are talking about is crushed with 3D semiconductors (hybrid bonding, wafer to wafer..., HBM), wafer scale CPUs like Cerebras, larger clusters, faster networking... Incidentally, massive parallelization suits AI just fine. Do you seriously think that computing power will stop growing? Ever?
I do think so. Every exponential growth is a sigmoid curve on a long enough timescale.
The end of Moore's Law is here (for now) even with multicores and everything else, by the way. That you can get more compute by throwing more resources at the problem doesn't really count. The big deal about Moore's Law was that it was free. The 3D stuff, clusters, etc, can increase the number of transistors... but it's gonna cost proportionally more in fabrication and energy when running.
> raw computing power grows by 10^6 every ~25 years
Good thing that the number of neurons in our brains are so many orders of magnitude larger than this that it doesn't matter.
I think what you're getting at is that in the next few years we're going to have AGI, which will eventually cause societal collapse. Under that assumption, while all of the AGI arguments are pretty poor, computing power increases over time is one of the weakest. It's going to take so much more than just computing power to get us close to AGI, it's going to take a lot more than LLMs, and it's possible to poison models with specially created data, so I'm not worried; I don't have shivers. What gives me more shivers is that there are people that seriously believe the hype to the degree that they believe the current technology, just a bit improved, will lead to the end of our current life as we know it.
Like 80% of the human brain is used for things that AI does not need (digestion, breathing, etc). So I think that automatically disqualifies your stance, but I am open-minded and would like to know what percent of the human brain you think is vital for AI to outperform us.
Even if you discount the role of the whole body as an integrated system, signaling and talking to itself, coordinated by the brain... you're still talking about 16 billion neurons in the cortex alone. That may sound like a number we can approach or exceed with modern tech, but neurons are not transistors.
Those 16 billion neurons in the cortex form over 140 TRILLION connections through synapes. Those are both computing elements as well as memory.
There's no comparison, not even close. As a bonus we don't each require a giant warehouse to house us, we don't require a vast supply of water, nor do we require a personal dedicated power plant.
Your last paragraph is the perfect rebuttal to GP. Computers don’t exist in a vacuum. They still require mineral and electrical inputs, repair, orchestration… sounds an awful lot like our bodies!
It’s always been possible to dream up some plausible doomsday scenarios.
I must have watched the movie Terminator 2 more than 50 times. I know the AI apocalypse very well.
Don’t worry about it. If the day comes, we’ll figure it out.
It’s not about computing power, it’s the inadequate algorithms, and ultimately the lack of scientific knowledge. Nobody has figured out how animal brains provide the plain old common sense you’d find in a fish or a frog, nor do we have any idea how to emulate that in a machine, even though in pure bits/second a 2025 supercomputer easily outperforms a frog. It’s not just robotics or 3D navigation: Veo2 still screws up object permanence, o1-pro still screws up distinguishing small quantities[1], and 99% of AI agents are simply infeasible because they have no clue what they’re doing beyond shallow pattern-matching.
[1] I imagine the same is true for o3. We simply don’t know how to make an AI that actually understands what numbers are in the way that toddlers and pigeons do.
> "We've recognized that we've reached the point as a technological civilization where the idea, there's huge abundance and huge economic value, but the idea that the way to distribute that value is for humans to produce economic labor, and this is where they feel their sense of self worth," he added. "Once that idea gets invalidated, we're all going to have to sit down and figure it out."
I suspect we'll just let those without the means to guarantee their well-being just die off then.
On the upside probably good for the environment & mitigating against climate change though.
> I don't think it will be a whole bunch longer than that when AI systems are better than humans at almost everything. Better than almost all humans at almost everything. And then eventually better than all humans at everything, even robotics."
> Amodei also spoke some about the potential implications of highly intelligent AI systems when these AI models can control advanced robotics.
"[If] we make good enough AI systems, they'll enable us to make better robots. And so when that happens, we will need to have a conversation..."
Good luck... I'm sure some altruistic individuals will "hack" things so you don't need to be worried.
If the power of _some_ humans depends on robots+AI, and there is a long track record of computers/vehicles being hacked, then there is economic incentive for competing humans to hack robots, including but not limited to competing AI systems.
For a fictional treatment of this topic, see Jonathan Nolan's Person of Interest, which begins with one supercomputer, but ends with competing supercomputers and their competing human owners in an endless arms race.
The incentives in this scenario are power, not altruism.
How is it a hassle? The machines can build and improve on themselves.
When there's very little economic reason to keep most people around. Then you just need enough force to keep people from taking what is yours and wait it out. No need to hunt anyone down indiscriminately.
Plus, nukes generally have been controlled by governments and mass use of them leaves little left for those in power and a whole lot of work for them.
Why guarantee the social safety nets? There's no benefit for those in power to keep them and we already let those in the U.S. without means die. Why should the case be different here?
Future generations will thank those asking this perennial question, for providing contemporary physical evidence of lessons long learned by earlier generations of humans with transient "power".
> Yeah, that's when they're still economically dependent
It goes deeper than that. Being surrounding by poor and suffering people sucks. Most people can’t block out that level of suffering and may even feel guilt.
No doubt there are some clinical psychopaths out there to whom that doesn’t apply but I think it’s broadly true.
It’s much easier to block out when people or poor but still sorta ok. So yes I think it’s going towards some sort of UBI. A low and grim UBI but UBI anyway
You can tell when the AI companies need funding, because they start to frame themselves in the most terrifying terms possible. OpenAI did this to perfection and it made them rich.
When you can't sell with the capabilities you have, and promising anything amazing would be a lie, just intimate that you're BASICALLY building Skynet and wait for the love.
This kind of hype is nutso. Look I use LLMs every day all day but they are assistants. They aren’t inventors or prognosticators. They cannot predict how the market will react to a product or idea. They cannot and will never be able to construct anything new outside of their training data. They’ll always need humans in the loop to come up with new interesting things and those humans will be the ones doing most of the actual work.
The people that push this stuff are crazy on their own fumes.
It sounds so transparently full of hype I really don’t know how people take them seriously.
I'm with you but I'm also thinking about how I didn't even see the capability we have no being as close as it was 10 years ago. In 2015 I would have guessed 2030 or beyond.
Humans can obviously invent, assist, prognosticate, and predict. Just because not every human may be able to do every one of those things doesn't invalidate the point that LLMs cannot do those things.
Try providing the LLM the necessary context and you'll be pleasantly surprised. The problem is that we haven't created proper 'inputs' (data pipelines) into LLMs. But that is going to change this year with agents, and an entirely new software paradigm will emerge from it.
This is how the scam works. Show off the potential of AI with the excuse that you just need tons of GPUs that cost billions to train to get to the so-called “AGI” whilst promising “efficiency” to VCs but paint an AI utopia to end users whilst you build your bunker for the incoming societal collapse.
No AI company has any sensible answer to what happens when the gradual displacement of knowledge workers accelerates and what exactly these “new jobs” are.
Without them specifying, then the goal of “AGI” was always to raise more money until they get there with the hard deadline of 2030. That is Anthropic’s plan and especially OpenAI’s plan.
Otherwise, why do you think OpenAI is going to release an agentic AI called “Operator” and who do you think they are trying to displace? Why are they (Sam, Zuck, etc) building their bunkers ahead of 2030?
I’m giving you 5 years ahead of their hard deadline of this displacement to prepare for their plan. [0]
Automation doesn’t mean life has zero cost. Even when it is extremely efficient.
And efficient machines earn for their owners. Not for the unemployed.
Also, efficiency increases the demand and price of resources. Including land.
This is especially true if there is extreme inequality, meaning a minority are benefiting from the savings and increased leverage from efficiency improvements. Increasing their lifestyles, but more significantly and endlessly, their need to compete harder with each other to maintain the value they have.
That self-reinforcing loop is already in full force. It’s just going to cycle faster.
We don’t tend to question that ever present reality in the wild, when we consider other species and individuals of other species.
For good, or ill, the human condition is about to change radically.
A perfect AI assistant is never coming. A near perfect one still has all the context switching downsides of unlimited free use of the mechanical turk API just faster.
Their problem really is in the present, no one should be buying their dangerously premium productivity boost plans when others will offer intelligence too cheap to meter. Such lack of faith in their own AI to solve future problems kind of exposes the doubt tree about their current path.
None of these Hype AI CEOs are willing to put their money where their mouth is, draw up a contract to sell all your hard assets for their projected cost to replace them in 2028. If so confident your AI can replace human labor for near-free by then what is the worry? Oh no, it won't build a yacht or a house by then... I know.
52 comments
[ 0.22 ms ] story [ 96.1 ms ] threadIf you think this is all BS and you will be always smarter than any machine, realize that raw computing power grows by 10^6 every ~25 years. AI seems to follow the same growth.
If you think you are safe, read what Geoffrey Hinton, one of the fathers of ML, said about 10-20% chance of AI leading to human extinction in 30 years. Or read about Sam Altman building his bunker in New Zealand while espousing AI utopia.
If this doesn't give you shivers, enjoy your blissful ignorance.
That is really not true anymore unless there will be a new technology that can do much better than silicon electronics. Doesn't mean that were safe, but it's not that clear.
Edit: to expand, a single transistor is made of a pretty pedestrian number of atoms these days. The gate pitch of TSMC "3 nm" is about 40 nm (not all of which is transistor) and silicon atoms are about 0.2 nm apart. So you have, let's say, 30^3 / 0.2^3 ~= 3 million atoms in a transistor. There is no room for another 10^6 improvement because a transistor with just a few atoms probably wouldn't work, the thermal noise would dominate it. Paraphrasing the famous Feynman quote, there isn't much room at the botton anymore, now.
The end of Moore's Law is here (for now) even with multicores and everything else, by the way. That you can get more compute by throwing more resources at the problem doesn't really count. The big deal about Moore's Law was that it was free. The 3D stuff, clusters, etc, can increase the number of transistors... but it's gonna cost proportionally more in fabrication and energy when running.
Good thing that the number of neurons in our brains are so many orders of magnitude larger than this that it doesn't matter.
I think what you're getting at is that in the next few years we're going to have AGI, which will eventually cause societal collapse. Under that assumption, while all of the AGI arguments are pretty poor, computing power increases over time is one of the weakest. It's going to take so much more than just computing power to get us close to AGI, it's going to take a lot more than LLMs, and it's possible to poison models with specially created data, so I'm not worried; I don't have shivers. What gives me more shivers is that there are people that seriously believe the hype to the degree that they believe the current technology, just a bit improved, will lead to the end of our current life as we know it.
Those 16 billion neurons in the cortex form over 140 TRILLION connections through synapes. Those are both computing elements as well as memory.
There's no comparison, not even close. As a bonus we don't each require a giant warehouse to house us, we don't require a vast supply of water, nor do we require a personal dedicated power plant.
It is hardly impossible there are remote white color workers at 30 right now who have to get a 6am-3pm/3pm-11pm/11pm-6am factory job at 50.
It is such a miserable thought I think the idea of societal collapse is more comforting.
I must have watched the movie Terminator 2 more than 50 times. I know the AI apocalypse very well. Don’t worry about it. If the day comes, we’ll figure it out.
For how many years has this been the trend?
[1] I imagine the same is true for o3. We simply don’t know how to make an AI that actually understands what numbers are in the way that toddlers and pigeons do.
I suspect we'll just let those without the means to guarantee their well-being just die off then.
On the upside probably good for the environment & mitigating against climate change though.
History has some entertaining responses to previous holders of such viewpoints.
Any recommendations on vendors of unhackable robots, a.k.a. computers?
> Amodei also spoke some about the potential implications of highly intelligent AI systems when these AI models can control advanced robotics.
"[If] we make good enough AI systems, they'll enable us to make better robots. And so when that happens, we will need to have a conversation..."
Good luck... I'm sure some altruistic individuals will "hack" things so you don't need to be worried.
For a fictional treatment of this topic, see Jonathan Nolan's Person of Interest, which begins with one supercomputer, but ends with competing supercomputers and their competing human owners in an endless arms race.
The incentives in this scenario are power, not altruism.
That's assuming there are competing humans left and economic gain to be had.
Humans have one or two incentives beyond economics.
> assuming there are competing humans left
Indeed, one possible scenario is zero humans left.
When there's very little economic reason to keep most people around. Then you just need enough force to keep people from taking what is yours and wait it out. No need to hunt anyone down indiscriminately.
Plus, nukes generally have been controlled by governments and mass use of them leaves little left for those in power and a whole lot of work for them.
Modern arsenals include a range of targetable weapons.
Hopefully the "strategies" of would-be predators are not based on AI hallucinations.
It does include this "... then magic happens" speculation:
Not particularly worried about anyone dying off. At least in mid to rich countries.
Bigger concern in those to me is what sort of existence it’ll be. State keeps you alive sure but I’d like to do a bit more than that
See that us insurance ceo getting shot. Lambo is more fun if you’re not at risk of getting shot
When the mass of people have no economic value to contribute to those in power, then what?
Will they just feed and house those people?
It goes deeper than that. Being surrounding by poor and suffering people sucks. Most people can’t block out that level of suffering and may even feel guilt.
No doubt there are some clinical psychopaths out there to whom that doesn’t apply but I think it’s broadly true.
It’s much easier to block out when people or poor but still sorta ok. So yes I think it’s going towards some sort of UBI. A low and grim UBI but UBI anyway
Homeless gathering tend to be moved to be out of sight.
We (in the U.S.) happily do not care for those whom cannot afford healthcare.
And those currently with their hands on the levers of society have already shown themselves to not act all that altruistically.
When you can't sell with the capabilities you have, and promising anything amazing would be a lie, just intimate that you're BASICALLY building Skynet and wait for the love.
The people that push this stuff are crazy on their own fumes.
It sounds so transparently full of hype I really don’t know how people take them seriously.
10 years on from that I don't think what we have now is so shocking.
The main problem is this waive of the hand "..and then AGI".
Who knows, maybe we even find AGI is not as useful as what it seems like it would be right now.
It mostly worries me the way "AGI" has taken on this mythic, modern quest for the holy grail/philosophers stone feel.
Can you?
No AI company has any sensible answer to what happens when the gradual displacement of knowledge workers accelerates and what exactly these “new jobs” are.
Without them specifying, then the goal of “AGI” was always to raise more money until they get there with the hard deadline of 2030. That is Anthropic’s plan and especially OpenAI’s plan.
Otherwise, why do you think OpenAI is going to release an agentic AI called “Operator” and who do you think they are trying to displace? Why are they (Sam, Zuck, etc) building their bunkers ahead of 2030?
I’m giving you 5 years ahead of their hard deadline of this displacement to prepare for their plan. [0]
[0] https://news.ycombinator.com/item?id=42563239
And efficient machines earn for their owners. Not for the unemployed.
Also, efficiency increases the demand and price of resources. Including land.
This is especially true if there is extreme inequality, meaning a minority are benefiting from the savings and increased leverage from efficiency improvements. Increasing their lifestyles, but more significantly and endlessly, their need to compete harder with each other to maintain the value they have.
That self-reinforcing loop is already in full force. It’s just going to cycle faster.
We don’t tend to question that ever present reality in the wild, when we consider other species and individuals of other species.
For good, or ill, the human condition is about to change radically.
Their problem really is in the present, no one should be buying their dangerously premium productivity boost plans when others will offer intelligence too cheap to meter. Such lack of faith in their own AI to solve future problems kind of exposes the doubt tree about their current path.
None of these Hype AI CEOs are willing to put their money where their mouth is, draw up a contract to sell all your hard assets for their projected cost to replace them in 2028. If so confident your AI can replace human labor for near-free by then what is the worry? Oh no, it won't build a yacht or a house by then... I know.