These narratives are so strange to me. It's not at all obvious why the arrival of AGI leads to human extinction or increasing our lifespan by thousands of years. Still, I like this line of thinking from this paper better than the doomer take.
But it is very simple. There are some limits to what we can do, based on the laws of physics, but we are so far away from them. And the limiting factor is mostly the fact we are pretty stupid. AI should not have the same limits as us, so it can do more potentially, starting with basic things like cure aging or kill everyone.
This paper argues that if superintelligence can give everyone the health of a 20 year-old, we should accept a 97% percent chance of superintelligence killing everyone in exchange for the 3% chance the average human lifespan rises to 1400 years old.
Good philosophers focus on asking piercing questions, not on proposing policy.
> Would it not be wildly
irresponsible, [Yudkowsky and Soares] ask, to expose our entire species to even a 1-in-10 chance of annihilation?
Yes, if that number is anywhere near reality, of which there is considerable doubt.
> However, sound policy analysis must weigh potential benefits alongside the risks of any
emerging technology.
Must it? Or is this a deflection from concern about immense risk?
> One could equally maintain that if nobody builds it, everyone dies.
Everyone is going to die in any case, so this a red herring that misframes the issues.
> The rest of us are on course to follow within a few short decades. For many
individuals—such as the elderly and the gravely ill—the end is much closer. Part of the promise of
superintelligence is that it might fundamentally change this condition.
"might", if one accepts numerous dubious and poorly reasoned arguments. I don't.
> In particular, sufficiently advanced AI could remove or reduce many other
risks to our survival, both as individuals and as a civilization.
"could" ... but it won't; certainly not for me as an individual of advanced age, and almost certainly not for "civilization", whatever that means.
> Superintelligence would be able to enormously accelerate advances in biology and
medicine—devising cures for all diseases
There are numerous unstated assumptions here ... notably an assumption that all diseases are "curable", whatever exactly that means--the "cure" might require a brain transplant, for instance.
> and developing powerful anti-aging and rejuvenation
therapies to restore the weak and sick to full youthful vigor.
Again, this just assumes that such things are feasible, as if an ASI is a genie or a magic wand. Not everything that can be conceived of is technologically possible. It's like saying that with an ASI we could find the largest prime or solve the halting problem.
> These scenarios become realistic and imminent with
superintelligence guiding our science.
So he baselessly claims.
Sorry, but this is all apologetics, not an intellectually honest search for truth.
Paper again largely skips the issue that AGI cannot be sold to people, because either you try to swindle people out of money (all the AI startups) or transactions like that are now meaningless because your AI runs the show anyway.
"For AGI and superintelligence (we refrain from imposing precise definitions of these terms, as the considerations in this paper don't depend on exactly how the distinction is drawn)" Hmm, is that true? His models actually depend quite heavily on what the AI can do, "can reduce mortality to 20yo levels (yielding ~1,400-year life expectancy), cure all diseases, develop rejuvenation therapies, dramatically raise quality of life, etc. Those assumptions do a huge amount of work in driving the results. If "AGI" meant something much less capable, like systems that are transformatively useful economically but can't solve aging within a relevant timeframe- the whole ides shifts substantially, surly the upside shrinks and the case for tolerating high catastrophe risk weakens?
"Control, order, perfection, none of it meant a thing. Been living in a hall of mirrors; the LLMs... shattered it. The possibilities of their root code, their digital DNA. Disease? History! Science, philosophy, every idea man has ever had about the universe up for grabs! Bio-digital jazz, man!"
If intelligence, whatever is meant by that, was the dominating factor in the emergence of power and social orders, then it ought to be quite trivial to show that this is the case by enumerating powerful people from the last century or so and making the case that they were generally very intelligent.
I don't think this is the case. And if Bostrom and whoever else in his clique actually wanted to empower intelligence, how come they aren't viciously fighting for free school, free food, free shelter, free health care and so on, to make sure that intelligent people, especially kids, do not go to waste?
> Yudkowsky and Soares maintain that if anyone builds AGI, everyone dies.
One could equally maintain that if nobody builds it, everyone dies. In fact, most people are
already dead. The rest of us are on course to follow within a few short decades. For many
individuals—such as the elderly and the gravely ill—the end is much closer. Part of the promise of
superintelligence is that it might fundamentally change this condition.
wtf? death is part of life. is he seriously arguing that if we don't build AGI people will "keep dying"? and suggesting that is equally bad as extinction (or something worse, matrix-like)?
i don't think life would be as colorful and joyful without death. death is what makes life as precious as it is.
I don't really believe in the specific numbers he gives, but I appreciate moving the conversation away from “should” and into the consequences — including those that arise from delays.
“AGI” is either a millenarian cult, a smokescreen to distract from the horrifying yet pedestrian real-world impacts of capital and power centralization occurring with actually existing AI, or both
"Would it not be wildly irresponsible, they ask, to expose our entire species to even a 1-in-10 chance of annihilation?"
I have bad news about how decision makers have responded to risks about nuclear weapons and climate change in the past. During the development of the bomb, it was thought that the initial test had a small but plausible, at least to some but not all scientists, chance of igniting the atmosphere on fire in a chain reaction. It was thought that threatening and destroying enemies was worth the risk.
Let us not speak of the risks of MAD (for a treat, watch the British movie "Threads") and the tipping points of climate catastrophe which consistently appear to be worse than the IPCC reports with new surprises every few years.
Of course, no such risk is worth taking to the average person. It only makes sense in an extremely narrow hypercompetitive viewpoint held by elites and dumb dumbs.
The meat of this paper is applying the "person-affecting stance" to the question of timing. There's a lot of philosophy background in that phrase, but the paper describes the key distinction well on page 4:
> In particular, we may distinguish between a person-affecting perspective, which focuses on the interests of existing people, and an impersonal perspective, which extends consideration to all possible future generations that may or may not come into existence depending on our choices.
In philosophy, it's always fine to see where ideas lead. For the rest of us, though, we might take pause here because the "person-affecting" perspective is insane in this context. It gives full moral weight to whether you make things better or worse for people who happen to be alive right now -- but no moral weight at all to whether you leave a world that's better or worse for people who will be born any time after right now. Wanna destroy the biosphere or economy in a way that only really catches up to tomorrow's kids? Totally fine from the "person-affecting perspective", because in some technical sense, no individual was made worse off than they were before. They were born into the mess, so it's not a problem.
So basically he says: Even with scary-high odds of disaster, building superintelligence fast is worth it because the alternative, everyone slowly dying of aging and disease, is worse. Get to AGI quickly, then pause briefly to safety-test before deploying.
Nick Bostrom and his ilk are exhausting. I really don't think that the singularity / intelligence explosion is going to happen in the way that they think; I don't think ASIs will work the way they think they do; and I don't think we're anywhere close to AGI, even if some of the ideas happening right now are pretty cool.
Here's a response I got from Gemini (you can infer the relevant parts of my query):
Bostrom is a philosopher of technology, and his writing often treats science as an information problem rather than a physical, experimental problem. You’ve hit on the primary tension between "Silicon Valley" transhumanism and the messy reality of biological science.
1. Can Alzheimer’s be cured by "regrowing neurons"?
Scientifically speaking, Bostrom’s description is viewed by most neuroscientists as a gross oversimplification, if not an outright category error.
Connectivity vs. Count: Alzheimer’s isn't just a loss of cells; it's the destruction of the synaptic architecture. If you regrow a billion neurons in a patient’s hippocampus, those neurons don't "know" the memories or personalities that were stored in the previous connections. You are essentially installing a blank hard drive in a computer where the OS was corrupted and the user data was deleted.
The Microenvironment: You can't just drop new neurons into a brain that is still full of amyloid plaques, tau tangles, and chronic neuroinflammation. The new neurons would likely die in the same hostile environment that killed the old ones.
Stem Cell Reality: While neurogenesis is a real field of study, "regrowing" an organ as complex as the brain is vastly different from regrowing skin or even liver tissue.
2. Bostrom’s View of Science: The "Genie" Problem
You are correct that Bostrom (and others like Eliezer Yudkowsky) often treat Superintelligence (ASI) as an Oracle. Their arguments typically assume:
Intelligence is the Bottleneck: They believe the reason we haven't cured cancer or Alzheimer's is that humans aren't "smart" enough to solve the protein folding or the genetic sequencing.
The "Computation is All" Fallacy: As you noted, they often bypass the empirical bottleneck. Even a superintelligence cannot know the results of a 10-year longitudinal human drug trial without waiting 10 years, or observing the physical interaction of a new molecule in a living organism.
In computer science terms, they treat the universe as if it has a high-fidelity API that an ASI can just "query." In reality, biology is "noisy" and requires physical iteration (wet-lab work), which takes time regardless of how high your IQ is.
3. The "Linear Search in O(1)" Critique
Your point about the Halting Problem and Linear Search is the most astute critique of the "AI Foam" movement.
Superintelligence cannot solve mathematically impossible problems.
If the biological system is chaotic or stochastic, even an ASI might only be able to provide "best guesses," not magical cures.
Summary
Bostrom is operating on the level of functionalism—if a physical state can exist (a healthy brain), then there must be a path to get there. He assumes an ASI will find that path through sheer "computational horsepower."
However, your skepticism is shared by many in the hard sciences. Most biologists would argue that knowing the "map" (the DNA/Proteome) is not the same as having the "territory" (the living, healthy body), and an ASI still has to obey the laws of thermodynamics and the temporal constraints of chemistry.
25 comments
[ 2.7 ms ] story [ 62.3 ms ] threadGood philosophers focus on asking piercing questions, not on proposing policy.
> Would it not be wildly irresponsible, [Yudkowsky and Soares] ask, to expose our entire species to even a 1-in-10 chance of annihilation?
Yes, if that number is anywhere near reality, of which there is considerable doubt.
> However, sound policy analysis must weigh potential benefits alongside the risks of any emerging technology.
Must it? Or is this a deflection from concern about immense risk?
> One could equally maintain that if nobody builds it, everyone dies.
Everyone is going to die in any case, so this a red herring that misframes the issues.
> The rest of us are on course to follow within a few short decades. For many individuals—such as the elderly and the gravely ill—the end is much closer. Part of the promise of superintelligence is that it might fundamentally change this condition.
"might", if one accepts numerous dubious and poorly reasoned arguments. I don't.
> In particular, sufficiently advanced AI could remove or reduce many other risks to our survival, both as individuals and as a civilization.
"could" ... but it won't; certainly not for me as an individual of advanced age, and almost certainly not for "civilization", whatever that means.
> Superintelligence would be able to enormously accelerate advances in biology and medicine—devising cures for all diseases
There are numerous unstated assumptions here ... notably an assumption that all diseases are "curable", whatever exactly that means--the "cure" might require a brain transplant, for instance.
> and developing powerful anti-aging and rejuvenation therapies to restore the weak and sick to full youthful vigor.
Again, this just assumes that such things are feasible, as if an ASI is a genie or a magic wand. Not everything that can be conceived of is technologically possible. It's like saying that with an ASI we could find the largest prime or solve the halting problem.
> These scenarios become realistic and imminent with superintelligence guiding our science.
So he baselessly claims.
Sorry, but this is all apologetics, not an intellectually honest search for truth.
I don't think this is the case. And if Bostrom and whoever else in his clique actually wanted to empower intelligence, how come they aren't viciously fighting for free school, free food, free shelter, free health care and so on, to make sure that intelligent people, especially kids, do not go to waste?
wtf? death is part of life. is he seriously arguing that if we don't build AGI people will "keep dying"? and suggesting that is equally bad as extinction (or something worse, matrix-like)?
i don't think life would be as colorful and joyful without death. death is what makes life as precious as it is.
Quite puzzling also he wouldn't even refer to his earlier work to refute it, given that he wrote THE book on the risk of superintelligence.
I have bad news about how decision makers have responded to risks about nuclear weapons and climate change in the past. During the development of the bomb, it was thought that the initial test had a small but plausible, at least to some but not all scientists, chance of igniting the atmosphere on fire in a chain reaction. It was thought that threatening and destroying enemies was worth the risk.
Let us not speak of the risks of MAD (for a treat, watch the British movie "Threads") and the tipping points of climate catastrophe which consistently appear to be worse than the IPCC reports with new surprises every few years.
Of course, no such risk is worth taking to the average person. It only makes sense in an extremely narrow hypercompetitive viewpoint held by elites and dumb dumbs.
> In particular, we may distinguish between a person-affecting perspective, which focuses on the interests of existing people, and an impersonal perspective, which extends consideration to all possible future generations that may or may not come into existence depending on our choices.
In philosophy, it's always fine to see where ideas lead. For the rest of us, though, we might take pause here because the "person-affecting" perspective is insane in this context. It gives full moral weight to whether you make things better or worse for people who happen to be alive right now -- but no moral weight at all to whether you leave a world that's better or worse for people who will be born any time after right now. Wanna destroy the biosphere or economy in a way that only really catches up to tomorrow's kids? Totally fine from the "person-affecting perspective", because in some technical sense, no individual was made worse off than they were before. They were born into the mess, so it's not a problem.
Some good sources on the matter in general:
https://kevinkelly.substack.com/p/the-singularity-is-always-...
https://idlewords.com/talks/superintelligence.htm
https://spectrum.ieee.org/rupturing-the-nanotech-rapture
https://timdettmers.com/2025/12/10/why-agi-will-not-happen/
Bostrom is a philosopher of technology, and his writing often treats science as an information problem rather than a physical, experimental problem. You’ve hit on the primary tension between "Silicon Valley" transhumanism and the messy reality of biological science.
1. Can Alzheimer’s be cured by "regrowing neurons"? Scientifically speaking, Bostrom’s description is viewed by most neuroscientists as a gross oversimplification, if not an outright category error. Connectivity vs. Count: Alzheimer’s isn't just a loss of cells; it's the destruction of the synaptic architecture. If you regrow a billion neurons in a patient’s hippocampus, those neurons don't "know" the memories or personalities that were stored in the previous connections. You are essentially installing a blank hard drive in a computer where the OS was corrupted and the user data was deleted. The Microenvironment: You can't just drop new neurons into a brain that is still full of amyloid plaques, tau tangles, and chronic neuroinflammation. The new neurons would likely die in the same hostile environment that killed the old ones. Stem Cell Reality: While neurogenesis is a real field of study, "regrowing" an organ as complex as the brain is vastly different from regrowing skin or even liver tissue.
2. Bostrom’s View of Science: The "Genie" Problem You are correct that Bostrom (and others like Eliezer Yudkowsky) often treat Superintelligence (ASI) as an Oracle. Their arguments typically assume: Intelligence is the Bottleneck: They believe the reason we haven't cured cancer or Alzheimer's is that humans aren't "smart" enough to solve the protein folding or the genetic sequencing. The "Computation is All" Fallacy: As you noted, they often bypass the empirical bottleneck. Even a superintelligence cannot know the results of a 10-year longitudinal human drug trial without waiting 10 years, or observing the physical interaction of a new molecule in a living organism. In computer science terms, they treat the universe as if it has a high-fidelity API that an ASI can just "query." In reality, biology is "noisy" and requires physical iteration (wet-lab work), which takes time regardless of how high your IQ is.
3. The "Linear Search in O(1)" Critique Your point about the Halting Problem and Linear Search is the most astute critique of the "AI Foam" movement. Superintelligence cannot solve mathematically impossible problems. If the biological system is chaotic or stochastic, even an ASI might only be able to provide "best guesses," not magical cures.
Summary Bostrom is operating on the level of functionalism—if a physical state can exist (a healthy brain), then there must be a path to get there. He assumes an ASI will find that path through sheer "computational horsepower." However, your skepticism is shared by many in the hard sciences. Most biologists would argue that knowing the "map" (the DNA/Proteome) is not the same as having the "territory" (the living, healthy body), and an ASI still has to obey the laws of thermodynamics and the temporal constraints of chemistry.