Have they? I don't like it either but the headline bull predictions(reliable agents and most code written by AI by mid-2026, dramatic progress against jailbreaks, prompt injection, hallucination, etc., major improvements despite pretraining data exhaustion, continued exponential growth on the METR trendline) did come true, often ahead of even aggressive schedules. What major wrong predictions did you have in mind?
> "I am, somehow, less interested in the weight and convolutions of Einstein’s brain than in the near certainty that people of equal talent have lived and died
in cotton fields and sweatshops."
I think it is so reckless and unethical that AI companies are not spending good money on answering the political, social, economical, moral questions that they are raising.
Right, makes sense, if they just spent a bunch of money they could answer all of those "political, social, economical, moral questions". All of these unprecedented sociological and labor market changes flying towards us are trivially solvable, we just need to... what, hire a few dozen economists? Fund a team of philosophers to think about it really hard for a few months?
I wanted this to be thought provoking, but for a physicist to open with, "the acceptable risk of catastrophic meltdown for a nuclear power plant is roughly one in a million", just seems sad. It is a phrase without meaning.
Is it per plant? there aren't a million.
Is it per year? notably have been at least 2 major ones (arguably 7).
A meltdown (or loss of containment) just isn't that bad, if it doesn't affect ground water or lead to atmospheric fallout. We're turning 3 Mile Island back on to power an AI datacenter! The AI super-intelligence apocalypse envisioned (ignoring the likelihood) is inherently global.
That said, the rest of the analysis and proposal also greatly disappoints me. The idea that the current administrations of US and China could do anything constructive seems hilarious. They're so paranoid and self-serving they couldn't come together, even if there was an alien invasion. Then the idea that LLM safety is somehow as easily traceable as nuclear isotopes and bomb tests seems equally ludicrous. I am sad.
Honestly, this article has too many words which don't mean anything.
'AI Experts', 'superintelligence', and hand-waving doom scenarios.
As an example-
> Within a couple of years, possibly much sooner, AI may achieve so-called closed-loop recursive self-improvement (RSI): the capacity to rewrite its own code to become more capable, without human intervention. Should that happen, the result could be an intelligence explosion of a kind for which there is no precedent and no map.
I've heard these same objections in my lifetime about the internet. And I've read similar arguments against TV, radio, the phonograph, and the printing press.
Honestly, this is getting extremely tiring. Every new invention that has happened has affected the world in both good and bad ways.
I mean, firstly you get to have an "Agent" actually capable of really long horizon tasks without getting stuck in tools loops and having its context rot. Secondly, each trial (ie a model fully trained to convergence) costs millions and takes O(weeks). You can probably run 1 or 2 of those experiments in parallel even at big AI labs as the hardware is scarce and they are costly as mentioned before. Assuming this agent needs something like a hundred tries to just show some improvement, we are looking at years.
And you can't early stop training for candidates that are not promising due to "emerging capabilities". At some point you might get a big drop in loss even if the model has been plateuing for a while. And you can't really scale down models for running trials quickly either also due to these emerging capabilities. In fact you might create a model that is great at converging with in small trials (fewer params, fewer tokens), but that is uncapable of developing those unexpected traits. And this will likely happen as you created a sort of evolutionary pressure in this direction: if you are good at learning in the first few epochs you "survive" and get to pass down your traits to future trials.
All of this to say that recursive-self-improvement is waaaaaay out of our grasp as things stand right now. We need another one or two breakthroughs to get there (IMHO).
>Honestly, this is getting extremely tiring. Every new invention that has happened has affected the world in both good and bad ways.
Common human trope is to rely on analogies and past patterns to predict the future. It blinds you reality.
This isn't some new "invention". What's happening is fundamentally more profound. the stakes are much much much more higher and the consequences much more grave. Thus a historical analogy can't save you when the situation is categorically different.
I agree it's getting "tiring" but this is another common human trope. You see the same thing too many times and you lose the sense of danger that comes with it. It's like sky diving all the time. You eventually lose the fear via repetition, but fundamentally speaking, you are still parachuting out of a fucking plane.
The Economist reflects the views and biases of economic and political elites.
AI has turned into something of a religion for them because the substitution of technology for labor is a reflection of one of their deepest desires. Whatever the reality of the situation, they want to believe that AGI employees are imminent.
There’s a meme in finance circles that by the time some trend appears on the economist front cover it’s a top signal and time to get out of the trade and this is why.
It’s a very normie magazine written by and for normies, and by the time an idea has got that far down the easy money has already been made
> Strikingly, this concern is being openly voiced by the very people who have the strongest incentives to project confidence rather than alarm: the founders of the largest AI laboratories
I couldn't read the whole article, but just from the part I could read:
> Strikingly, this concern is being openly voiced by the very people who have the strongest incentives to project confidence rather than alarm: the founders of the largest ai laboratories.
I don't know, they also have an incentive to make their technology seem transformative and powerful, and saying that your technology has the power to cause a massive catastrophe is a way to promote that idea.
> an incentive to make their technology seem transformative and powerful
I have no frame of reference to process this.
Humans species perhaps 300,000 years, we are essentially the same. Transistor, 79 years, explosive growth in numbers and power. Integrated circuit, 68 years, explosive growth in numbers and power. "Attention is all you need", 9 years. ChatGPT, 4 years, explosive growth in instances and power. Humans species, not getting smarter. [0,1,2,3]
There is a syndrome where many people seem unable to perceive or reason about rates of change in technology.
We are going to spend the vast majority of our future lives without the intelligence crown.
In terms of verbally expressible knowledge, models are passing many people completely, and passing all of our individually respectively weak reasoning areas.
Other modalities are progressing very quickly.
There will be short periods where progress happens quickly, but the impact feels slow. There will be radical changes that feel slow too, because if something anticipated or important isn't instant, we tend to perceive it as slow.
But it won't be slow. And it won't be long. We are smart in a kind of pick the-best-of-us at the-best-of-times way. We are rarely consistently or broadly smart individually.
We are not in the same galaxy as "ready". What would that look like?
Is it really likely that a "recursive self-improvement" capability would lead to a great acceleration of AIs capabilities?
Isn't the preponderant bottleneck in improving the models the need to train them at scale to verify the hypotheses, and the time and cost that it takes?
Or does someone think that they could get magically able to predict big improvements without training?
Humans gain knowledge through experiments. Without a physical body it has no chance of performing the same. That it can update it's training weights does not seem particularly significant.
The preponderant bottleneck is inventing new architectures to make AI actually good at human and superhuman tasks. For example, AI agent harnesses add tool calls and long term state management, allowing AI to autonomously complete complex tasks. Once these are in place, finetuning models with examples of good tool calls helps, but somebody first needed to invent the fundamental capability. Now try to implement humanlike long term memory for AI to be your coworker or life assistant working on tasks that last month. Even if necessary low level technologies are already there, structuring them to be practically useful is non trivial.
RSI most likely does not exist. At least not in the sci fi sense that AI becomes super intelligent over night.
It will be like any other technology. Do computers make it faster and easier to design better computers? Yes. But that doesn't mean a step change overnight.
I want to name the AI era as the corruption of the elite.
The rhetorical structure of talking about uncertain risks and then trying to concentrate the authority to manage those risks in their own hands sounds utterly ridiculous to ordinary people like me.
It's just a simple hypothesis that AI will become uncontrollable to humans once it becomes superintelligent.
Isn't the fact that a reinforcement learning agent improves itself in a specific domain completely different from it recursively improving its own code in a 'better' way? It's just a tool to create a justification for regulation and control using sci-fi fear.
The comparison between nuclear power plant risk and AI risk is also absurd. Where exactly can you define and measure the probability of AI exterminating humanity? It's as unquantifiable as 'I, human JDW64, will become a successful programmer.' What is the measurement standard? Why dress up AI researchers' concerns as objective probabilities? Is it because numbers make it look logical?
The current US-China relationship is in the middle of an AI arms race. The US is strengthening export controls to limit China's AI development, and China is building its own ecosystem. In this situation, I don't understand the idea of cooperating for the common safety of humanity. RAND is an organization that presupposes cooperation—isn't it just a well-written research proposal from an institution that wants to position itself for that role?
Isn't the claim that 'government must step in' ultimately about protecting their own interests? 'A strong government that will protect us' is an authoritarian government. If they were East Asian, they would understand that such regimes have always been used as tools for surveillance and control.
And I don't understand why Fermi's paradox is being brought up here. Why package a software problem as something that inevitably requires strong control? Whenever I see articles like this, I think about what 'intelligence' really means. This person would probably be called 'intelligent' by others. But no matter how I look at it, the holes in the argument are too obvious. It really makes me think that there are different tiers of intelligence.
> Fermi asked why, given the apparent abundance of planets suitable for life, no evidence of other technologically advanced civilisations had been detected. One disquieting possibility is that intelligent life routinely reaches a technological threshold and fails to navigate it,
The Fermi paradox could actually also be taken as an evidence that it's rare (at least) for artificial intelligences to take over a civilization and sprawl and survive for very long times
Models consume a lot of memory and power to slowly generate autocompletions of existing content one token at a time. Letting this text control important things in real world is a human decision and control of nuclear weapons or penetration testing agents should be well regulated. But these should already be well regulated without AI. So how about we focus on drawing down nuclear arsenals, which is a present danger one idiot (or faulty AI) away from world shattering consequences rather than fearmongering about unknown future. Before effective AI regulation can be drafted, we need to anyway know more about AI specific dangers rather than humans committing already very common crimes like hacking for ransom with new tools?
A similar Luddite view formed in the 1990s. It turns out humanity constantly beats the expectations of economists. It makes you wonder if the "science" of economy is almost entirely bankrupt either in morality or imagination or same dangerous combination of both.
> Humanity simply does not have a strategy to ensure it remains safe through RSI.
Turn off the power. It's pretty simple. Leave it to an economist to forget about input costs. Your "super intelligence" only matters if it's actually more energy efficient than a human being and for a million years of evolution humanity is a much harder target to beat than this author seems to realize.
This article posits "recursive self-improvement"—which is just a more-technical name for the supposed Singularity.
There is no evidence that we are currently on a path that leads to the Singularity.
There is not yet any evidence that the Singularity is more than science fiction.
It is effectively an article of religion—the Rapture, but for techbros. Indeed, it is what some of them are pinning all their hopes on; after all, if they can recursively-self-improve their way to artificial superintelligence, then and only then could the absurd investments of companies like OpenAI actually prove worthwhile.
"Ten years from now, I think we will realize that we were standing in the foothills of the singularity now... I believe that we're only a few years away from [AGI], maybe 2030 plus or minus a year...
I think [AGI] will be an enormous transformative technology, it's going to effectively be a new human era...
We can feel this year, I would say, even though I've been working towards this for 30 years, I think this year with the way the agents are working and tool use, it started to become really useful, still early days of it, but genuinely useful in people's workflows...
And it's not any one thing, it's several different technologies, several use cases, several things that I thought were maybe a bit further out, turned out to be now, that are coming together that make me feel that in aggregate.
I think society needs to hear that because we don't have long to prepare for what that means." -- Demi Hassabis, CEO of Google DeepMind & Nobel laureate
Of course humanity is not ready for skynet. But there are rebel
forces - the no AI using humans. They may be a dying breed but
at the least they put up a fight. Many other humans already
have become servants to skynet (version 16.0 now). Next step for
them is the neuralink chip. Some billionaire with twitching right arm gestures owns that, doesn't it ...
(Also, Cameron was not quite right with regards to skynet. It is much much dumber but also more effective than displayed in his movie really. Kind of a weird combination if you look at the current AI slop out there.)
73 comments
[ 2.4 ms ] story [ 52.5 ms ] threadThe predictions of these "experts" have been drastically wrong for the last 3 years. At what point does someone lose their "expert" title?
Which ones? Please be specific.
Makes me think of a Stephen Jay Gould quote:
> "I am, somehow, less interested in the weight and convolutions of Einstein’s brain than in the near certainty that people of equal talent have lived and died in cotton fields and sweatshops."
My aching joints enter the chat...
That said, the rest of the analysis and proposal also greatly disappoints me. The idea that the current administrations of US and China could do anything constructive seems hilarious. They're so paranoid and self-serving they couldn't come together, even if there was an alien invasion. Then the idea that LLM safety is somehow as easily traceable as nuclear isotopes and bomb tests seems equally ludicrous. I am sad.
'AI Experts', 'superintelligence', and hand-waving doom scenarios.
As an example-
> Within a couple of years, possibly much sooner, AI may achieve so-called closed-loop recursive self-improvement (RSI): the capacity to rewrite its own code to become more capable, without human intervention. Should that happen, the result could be an intelligence explosion of a kind for which there is no precedent and no map.
I've heard these same objections in my lifetime about the internet. And I've read similar arguments against TV, radio, the phonograph, and the printing press.
Honestly, this is getting extremely tiring. Every new invention that has happened has affected the world in both good and bad ways.
Ultimately, what counts is what we do with it.
I mean, firstly you get to have an "Agent" actually capable of really long horizon tasks without getting stuck in tools loops and having its context rot. Secondly, each trial (ie a model fully trained to convergence) costs millions and takes O(weeks). You can probably run 1 or 2 of those experiments in parallel even at big AI labs as the hardware is scarce and they are costly as mentioned before. Assuming this agent needs something like a hundred tries to just show some improvement, we are looking at years.
And you can't early stop training for candidates that are not promising due to "emerging capabilities". At some point you might get a big drop in loss even if the model has been plateuing for a while. And you can't really scale down models for running trials quickly either also due to these emerging capabilities. In fact you might create a model that is great at converging with in small trials (fewer params, fewer tokens), but that is uncapable of developing those unexpected traits. And this will likely happen as you created a sort of evolutionary pressure in this direction: if you are good at learning in the first few epochs you "survive" and get to pass down your traits to future trials.
All of this to say that recursive-self-improvement is waaaaaay out of our grasp as things stand right now. We need another one or two breakthroughs to get there (IMHO).
Common human trope is to rely on analogies and past patterns to predict the future. It blinds you reality.
This isn't some new "invention". What's happening is fundamentally more profound. the stakes are much much much more higher and the consequences much more grave. Thus a historical analogy can't save you when the situation is categorically different.
I agree it's getting "tiring" but this is another common human trope. You see the same thing too many times and you lose the sense of danger that comes with it. It's like sky diving all the time. You eventually lose the fear via repetition, but fundamentally speaking, you are still parachuting out of a fucking plane.
AI has turned into something of a religion for them because the substitution of technology for labor is a reflection of one of their deepest desires. Whatever the reality of the situation, they want to believe that AGI employees are imminent.
It’s a very normie magazine written by and for normies, and by the time an idea has got that far down the easy money has already been made
Or to have AI regulated in their favor
> Strikingly, this concern is being openly voiced by the very people who have the strongest incentives to project confidence rather than alarm: the founders of the largest ai laboratories.
I don't know, they also have an incentive to make their technology seem transformative and powerful, and saying that your technology has the power to cause a massive catastrophe is a way to promote that idea.
I have no frame of reference to process this.
Humans species perhaps 300,000 years, we are essentially the same. Transistor, 79 years, explosive growth in numbers and power. Integrated circuit, 68 years, explosive growth in numbers and power. "Attention is all you need", 9 years. ChatGPT, 4 years, explosive growth in instances and power. Humans species, not getting smarter. [0,1,2,3]
There is a syndrome where many people seem unable to perceive or reason about rates of change in technology.
We are going to spend the vast majority of our future lives without the intelligence crown.
In terms of verbally expressible knowledge, models are passing many people completely, and passing all of our individually respectively weak reasoning areas.
Other modalities are progressing very quickly.
There will be short periods where progress happens quickly, but the impact feels slow. There will be radical changes that feel slow too, because if something anticipated or important isn't instant, we tend to perceive it as slow.
But it won't be slow. And it won't be long. We are smart in a kind of pick the-best-of-us at the-best-of-times way. We are rarely consistently or broadly smart individually.
We are not in the same galaxy as "ready". What would that look like?
[0] https://en.wikipedia.org/wiki/Human
[1] https://en.wikipedia.org/wiki/Transistor
[2] https://en.wikipedia.org/wiki/Integrated_circuit
[3] https://en.wikipedia.org/wiki/Large_language_model
I don't think it's that high but can we put that aside and focus on the 100% chance that it's being used to enshittify every part of our lives?
Isn't the preponderant bottleneck in improving the models the need to train them at scale to verify the hypotheses, and the time and cost that it takes?
Or does someone think that they could get magically able to predict big improvements without training?
It will be like any other technology. Do computers make it faster and easier to design better computers? Yes. But that doesn't mean a step change overnight.
The rhetorical structure of talking about uncertain risks and then trying to concentrate the authority to manage those risks in their own hands sounds utterly ridiculous to ordinary people like me.
It's just a simple hypothesis that AI will become uncontrollable to humans once it becomes superintelligent.
Isn't the fact that a reinforcement learning agent improves itself in a specific domain completely different from it recursively improving its own code in a 'better' way? It's just a tool to create a justification for regulation and control using sci-fi fear.
The comparison between nuclear power plant risk and AI risk is also absurd. Where exactly can you define and measure the probability of AI exterminating humanity? It's as unquantifiable as 'I, human JDW64, will become a successful programmer.' What is the measurement standard? Why dress up AI researchers' concerns as objective probabilities? Is it because numbers make it look logical?
The current US-China relationship is in the middle of an AI arms race. The US is strengthening export controls to limit China's AI development, and China is building its own ecosystem. In this situation, I don't understand the idea of cooperating for the common safety of humanity. RAND is an organization that presupposes cooperation—isn't it just a well-written research proposal from an institution that wants to position itself for that role?
Isn't the claim that 'government must step in' ultimately about protecting their own interests? 'A strong government that will protect us' is an authoritarian government. If they were East Asian, they would understand that such regimes have always been used as tools for surveillance and control.
And I don't understand why Fermi's paradox is being brought up here. Why package a software problem as something that inevitably requires strong control? Whenever I see articles like this, I think about what 'intelligence' really means. This person would probably be called 'intelligent' by others. But no matter how I look at it, the holes in the argument are too obvious. It really makes me think that there are different tiers of intelligence.
The Fermi paradox could actually also be taken as an evidence that it's rare (at least) for artificial intelligences to take over a civilization and sprawl and survive for very long times
The ones talking about how hard it would be to chose which person to run over.
Additionally, I find it hard to believe that this would be a case of the future just not being distributed evenly.
But sure. The AI labs relying on hype stating 15-50% risk of building a magic entity is certainly a reliable number.
> Humanity simply does not have a strategy to ensure it remains safe through RSI.
Turn off the power. It's pretty simple. Leave it to an economist to forget about input costs. Your "super intelligence" only matters if it's actually more energy efficient than a human being and for a million years of evolution humanity is a much harder target to beat than this author seems to realize.
There is no evidence that we are currently on a path that leads to the Singularity.
There is not yet any evidence that the Singularity is more than science fiction.
It is effectively an article of religion—the Rapture, but for techbros. Indeed, it is what some of them are pinning all their hopes on; after all, if they can recursively-self-improve their way to artificial superintelligence, then and only then could the absurd investments of companies like OpenAI actually prove worthwhile.
I think [AGI] will be an enormous transformative technology, it's going to effectively be a new human era...
We can feel this year, I would say, even though I've been working towards this for 30 years, I think this year with the way the agents are working and tool use, it started to become really useful, still early days of it, but genuinely useful in people's workflows...
And it's not any one thing, it's several different technologies, several use cases, several things that I thought were maybe a bit further out, turned out to be now, that are coming together that make me feel that in aggregate.
I think society needs to hear that because we don't have long to prepare for what that means." -- Demi Hassabis, CEO of Google DeepMind & Nobel laureate
Source with interview clip: https://x.com/deredleritt3r/status/2062223035940139253
EDIT: Ten years ago, most of the skeptics in these threads would have said the success of AlphaFold 3 was impossible.
(Also, Cameron was not quite right with regards to skynet. It is much much dumber but also more effective than displayed in his movie really. Kind of a weird combination if you look at the current AI slop out there.)