Most of modern history has been defined by our ability to outpace our problems through technological acceleration. This essay argues that, rather than an uncontrollable AI takeoff, we may be approaching physical, economic, and regulatory limits — a long plateau where progress slows.
TFA makes a compelling contrarian point we probably need to consider more than we do. As someone who's been in high tech for 40 years, progress in fundamental enabling drivers like semi scaling has slowed significantly since 2010 and the industry's projections don't foresee a return to the extraordinary rates we enjoyed from ~1970 to ~2010.
These aren't very convincing arguments; why don't aircraft factories of the 1930s or the ocean-going steamship dockyards of the 1830s count under "massive upfront investment and large and complex" and therefore predict progress stopping ages ago?
Asimov's story The Last Question ends with the Multivac machine having collected all the data in the universe and still not answering the question "how can entropy be reversed?", so it spends an immeasurable amount of time processing the data in all possible ways. The article argues that we might not get to "the singularity" because progress will stop, but even if we can't make better transistors, we can make more of them, and we can spend longer processing data with them. If what we're missing in an AGI is architectural it might only need insight and distributed computing, not future computers.
> "We built our optimism during a rare century when progress got cheaper as it got faster. That era may be over."
This effect of progress building on progress goes back a hundred years before that, and a hundred years before that. The first practical steam engine was weak, inefficient, and coal-hungry in the early 1700s and what made it 'practical' is that it pumped water out of coal mines. Coalmine owners could get more coal by buying a steam engine; the engine made its fuel cheaper and easier and more coal to sell. Probably this pattern goes back a lot before that because everything builds on everything, but this was a key industrial revolution turning point a long time before the article's claim. The era may be another two hundred years away from being over.
> "There are still areas of promise for step-function improvements: fusion, quantum computing, high-temperature superconductors. But scientific progress is not guaranteed to continue."
Opening with the recursively improving AGI and then having a section of "areas of promise for step-function improvements" and not mentioning any chance of an AGI breakthrough? Neuralink style cyborg interfaces, biological, genetic, health, anti-ageing, new materials or meta-materials, nanotechnology, distributed computing, vibe coding, no possible areas for step changes in any of those?
> "But the burden of proof lies with those claims. Based on what we know today, a plateau is inevitable. Within that plateau, we can only speculate:"
Based on what we know today there isn't "a" plateau, there are many, and they give way to newer things. Steam power plateaued, propellor aircraft plateaued, sailboat speed and size plateaued, cog and gear computer speed plateaued, then electro-mechanical computer speed, then valve computer speed, then discrete logic speed, then integrated circuit speed, then single core, then what, CPUs, then GPUs, then TPUs...
> "Are therapies for broad set of complex autoimmune diseases ahead of the plateau? Probably."
How many autoimmune diseases have been cured, ever? Where does this "Probably" come from - the burden of proof very much lies with that probably.
> "Will we have Earth-based space elevators before the plateau? Probably not."
We don't have a rope strong enough to hang 36km or a way to make one or a way to lift that much mass into geostationary orbit in one go. But if we could make a cable thicker in space, thinner at the ground, launch it in pieces and join it together, we might not be that far away from plausible space elevator. Like if Musk got a bee in his bonnet and opened his wallet wide, I wouldn't bet against SpaceX having a basic one by 2040. Or 2035. I probably would bet against 2028.
> "the cultural zeitgeist has centered on the idea of runaway progress: artificial intelligence recursively improving itself until we become subservient to it. far more plausible .. a world where the curve flattens and our rate of progress slows to a crawl."
> "our mission should remain the same: accelerate to the maximum extent possible."
I think you need to justify why hurrying to be AI servants should be our mission :-|
You're right, I should have tied that back to the opening.
The acceleration we've experienced has allowed us to "outrun" our problems. In earlier generations, that meant famine or disease. Today, it might be climate change. Tomorrow, it'll be something else entirely.
Technological progress has generally been the reason humanity should be optimistic against challenges: it gives us ever improving tools to solve our hardest problems faster than we succumb to them. Without it, that optimism becomes much harder to justify.
Even if there is a plateau we can't cross, if we believe we drive more benefit from technology than the problems it creates, it makes sense to extract as much progress as we can from the physics we have.
Explain to me how the, "Technological Singularity" isn't just Christian eschatology for dorks? As a Neon Genesis Evangelion fan this really gets me going but that's kind of why I ask.
"artificial intelligence recursively improving itself until we become subservient to it."
Hmm, the cultural zeitgeist is about LLMs.
Are LLMs improving anything (in the sense of optimization)? I think LLMs are enabling us to automate tasks which are tedious and don't really add value (eg, compliance tasks). And they are helping us create art, content, and ads. I'm not aware of LLMs optimizing systems, let alone themselves. But I'm not very tuned in to all the applications.
Re building a bigger colliders - it occurs to me that it would be difficult to find somewhere to put a 10,000 km collider, but that is roughly the circumference of the moon (which is already a vacuum, so no need for a pressure vessel), or build it in free space if that is not big enough.
The plateau you speak of is just inevitable human evolution. There will come a point where we will start to edit our genome and see what's capable of. Before we get there, though -
In time, AI and VR/AR will converge to allow us to evolve new ways to educate/entertain new generations, to distill knowledge in a faster and much more reliable way. We will experience societal upheavals before the "plateau". Our current world order will probably experience major changes.
AGI will probably be a long, long way ahead of us -- in its current state, LLMs are not going to spontaneously develop sentience. There is a massive scale (power, resources, space, etc) issue to contend with.
9 comments
[ 1.7 ms ] story [ 34.2 ms ] threadAsimov's story The Last Question ends with the Multivac machine having collected all the data in the universe and still not answering the question "how can entropy be reversed?", so it spends an immeasurable amount of time processing the data in all possible ways. The article argues that we might not get to "the singularity" because progress will stop, but even if we can't make better transistors, we can make more of them, and we can spend longer processing data with them. If what we're missing in an AGI is architectural it might only need insight and distributed computing, not future computers.
> "We built our optimism during a rare century when progress got cheaper as it got faster. That era may be over."
This effect of progress building on progress goes back a hundred years before that, and a hundred years before that. The first practical steam engine was weak, inefficient, and coal-hungry in the early 1700s and what made it 'practical' is that it pumped water out of coal mines. Coalmine owners could get more coal by buying a steam engine; the engine made its fuel cheaper and easier and more coal to sell. Probably this pattern goes back a lot before that because everything builds on everything, but this was a key industrial revolution turning point a long time before the article's claim. The era may be another two hundred years away from being over.
> "There are still areas of promise for step-function improvements: fusion, quantum computing, high-temperature superconductors. But scientific progress is not guaranteed to continue."
Opening with the recursively improving AGI and then having a section of "areas of promise for step-function improvements" and not mentioning any chance of an AGI breakthrough? Neuralink style cyborg interfaces, biological, genetic, health, anti-ageing, new materials or meta-materials, nanotechnology, distributed computing, vibe coding, no possible areas for step changes in any of those?
> "But the burden of proof lies with those claims. Based on what we know today, a plateau is inevitable. Within that plateau, we can only speculate:"
Based on what we know today there isn't "a" plateau, there are many, and they give way to newer things. Steam power plateaued, propellor aircraft plateaued, sailboat speed and size plateaued, cog and gear computer speed plateaued, then electro-mechanical computer speed, then valve computer speed, then discrete logic speed, then integrated circuit speed, then single core, then what, CPUs, then GPUs, then TPUs...
> "Are therapies for broad set of complex autoimmune diseases ahead of the plateau? Probably."
How many autoimmune diseases have been cured, ever? Where does this "Probably" come from - the burden of proof very much lies with that probably.
> "Will we have Earth-based space elevators before the plateau? Probably not."
We don't have a rope strong enough to hang 36km or a way to make one or a way to lift that much mass into geostationary orbit in one go. But if we could make a cable thicker in space, thinner at the ground, launch it in pieces and join it together, we might not be that far away from plausible space elevator. Like if Musk got a bee in his bonnet and opened his wallet wide, I wouldn't bet against SpaceX having a basic one by 2040. Or 2035. I probably would bet against 2028.
> "our mission should remain the same: accelerate to the maximum extent possible."
I think you need to justify why hurrying to be AI servants should be our mission :-|
The acceleration we've experienced has allowed us to "outrun" our problems. In earlier generations, that meant famine or disease. Today, it might be climate change. Tomorrow, it'll be something else entirely.
Technological progress has generally been the reason humanity should be optimistic against challenges: it gives us ever improving tools to solve our hardest problems faster than we succumb to them. Without it, that optimism becomes much harder to justify.
Even if there is a plateau we can't cross, if we believe we drive more benefit from technology than the problems it creates, it makes sense to extract as much progress as we can from the physics we have.
Hmm, the cultural zeitgeist is about LLMs.
Are LLMs improving anything (in the sense of optimization)? I think LLMs are enabling us to automate tasks which are tedious and don't really add value (eg, compliance tasks). And they are helping us create art, content, and ads. I'm not aware of LLMs optimizing systems, let alone themselves. But I'm not very tuned in to all the applications.
In time, AI and VR/AR will converge to allow us to evolve new ways to educate/entertain new generations, to distill knowledge in a faster and much more reliable way. We will experience societal upheavals before the "plateau". Our current world order will probably experience major changes.
AGI will probably be a long, long way ahead of us -- in its current state, LLMs are not going to spontaneously develop sentience. There is a massive scale (power, resources, space, etc) issue to contend with.