If carbon taxes are already a lethal policy for an political campaign, it's absurd to think that fears of ASI will create any real movement around pausing AI.
If there is any movement to pause AI development, it will come from the general public's dislike of these companies. Not from the AI safety angle.
I'm sure some people will have issue with my phrasing but, honest question:
Are there examples of where we have collective decided not to pursue knowledge? Successfully?
I guess nuclear weapons might be the best example though research doesn't seem have to actually "stopped" as much as gone underground and we still have country trying to climb that ladder.
But I don't know how relevant that is to LLMs/AI. It almost feels like pandora's box is open and our only option is continue to improve them. There is clearly value in what they do and while I can absolutely see the dangers, for example: authoritative governments and surveillance, I'm not convinced to throw the baby out with the bathwater.
All of technology back to the printing press (and probably before that) could also be said to make it easier for governments to oppress their citizens. Making laws (and enforcing them!) to prevent governments from doing these things feels like that route forward, not trying to stick our heads in the sand.
Perhaps I'm horribly naive, perhaps I just see the SciFi future I've spent my life reading and dreaming about on the horizon and I'm blinded by the reality, perhaps my ideals around "knowledge deserves to be free/accessible" are misguided. I don't know.
While technology has empowered governments, it’s also empowered the individual, and more importantly shifted the material dynamics to better align the incentives of governments with the people. Democracy followed material change, it didn’t precede it. Democracy came about because it was optimal for a power seeking government, not out of the kindness of their heart.
A resource extraction based economy sees people as slaves. The true source of power is the resource, people are just a means to an end, so you mistreat the people as much as you can get away with in pursuit of the resource while avoiding revolt.
With stable infrastructure, the government makes far more from an educated, rich population that it can tax and use the innovation from. It’s against its own quest for power to interfere too much in the prosperity of its citizens. The incentives are aligned.
Solving the AI problem isn’t about stopping the tech or making a bunch of brittle laws. It’s always been about alignment: aligning the large AGI-like entities that are the modern state, the modern economy, representative democracy, or AGI itself, with human prosperity
This is a just so story. The main issue today is the lack of democracy in the country and the use of technology to surveil and govern a restive population as the government has less and less legitimacy. The narrative you are telling is the heroic tale of computing and the internet c. 1990-2010.
Yasha Levine wrote about how this narrative was preceded by a forgotten one where MIT students protested because the computers were going to be linked to government databases and share data on anti-Vietnam war activists. Despite protestations, activists were correct and this happened, and now it happens at huge scale.
> Democracy came about because it was optimal for a power seeking government, not out of the kindness of their heart
It's not clear in this context what you actually mean by "government." You are assigning agency to something in a way that seems like a reification. While a bureaucracy can seem to have a life of its own, isn't it generally people who seek power?
> Are there examples of where we have collective decided not to pursue knowledge? Successfully?
Intrinsically, the knowledge humans choose not to pursue will not be much publicized. There's limited value in calling attention to it and it doesn't make for good entertainment. Plenty of examples provided by other comments nonetheless.
> Perhaps I'm horribly naive, perhaps I just see the SciFi future I've spent my life reading and dreaming about on the horizon and I'm blinded by the reality, perhaps my ideals around "knowledge deserves to be free/accessible" are misguided. I don't know.
I don't personally think there's intrinsic benefit in disseminating arbitrary knowledge. There's quite some difference between the printing press and nukes.
Perhaps the burning of the library at Alexandria would qualify. How intentional that was is somewhat in question, but the world certainly turned its back on the only collection of written knowledge and let it turn to ash.
Human embryo genetic modification has been effectively taboo if not banned until just recently since WWII and the aftermath of the Holocaust. I think some people in the US are proposing doing it now but I don't think anyone has tried it yet without repercussions.
I like your optimism and I think you will be vindicated. AI is democratic and AI talent is globally distributed. It will just take a while to get online. AI labs do not have a monopoly on human talent, and open source AI only empowers independent science and meritocracy.
On a funny note, I think their prompt was:
"Hey Fable. Please attribute every piece of scientific and economic progress to AI until 2040. And predict every major geopolitical event. Make no mistakes."
There is no rein on progress. You can buy electric motorcycles without speed governors. And you can drive them on public roads (subject to traffic laws) as long as you're properly licensed, registered, and insured. The distinction between electric "bikes" and motorcycles is mostly artificial.
It’s telling terrorists how to make bombs better apparently. Continuing to lower the barrier for that kind of stuff is clearly a negative in the “knowledge deserves to be free” world.
Biological weapons? Yes, there is research on defense, but no big arsenals of weapons etc.
My impression from the origin of the bioweapons convention is that collectively people decided that these things are too dangerous in various ways for any advantage that might be derived from them.
I can think of many examples that I won't name but you can imagine in biology/medical fields where certain lines of investigation are not performed due to ethical and legal repercussions.
People overestimate progress in physical world.
2035:
robot population will soon be larger than the human one
I'd bet that in most places 9 years is about the time needed to build a residential building. I think a good way to think about this is to think of this as producing a serial car. From pitching and capital acquisition to building a prototype to software, regulatory and then the final product which needs multiple factories and supply chains. Yes, of course robots sound cooler and there are compounding effects yada yada, but on the other side there are as many obstacles as things that accelerate this product (like capital acquisition and fearmongering of gov to bend regulatory stuff faster).
LLM adoption is 30% in the charts I saw googling for "ai adoption". An example of capibility: I have had Claude one shot an RL agent that learns connect four in 30 minutes. That's PhD level stuff.
LLMS are 4 years old and the companies that sell them 10x every year. What evidence can you cite? Could you convince a disinterested 3rd party you have anything other than cope? What facts about the world make you think this is anything other than the new (and probably temporary) normal?
What predictions about the technology are the authors making that you do not believe?
There is plenty falsifiable in this in ai-2027.com, and they have not gotten everything right. But some things they have: for example, the Pentagon has already invoked export controls to restrict the deployment of a frontier model. This level of government oversight wasn't predicted until 2027 in the original scenario.
Self improvement of agent-1 is not achieved. Sure, people in AI labs write python code with AI, but I doubt it resulted in 50% algorithmic effiecency in training. Writing python code never was the bottleneck, if it was, AI labs could hire more people to do it. And this is core of the prediction.
Nothing follows from this empty platitude though, right? It can't inform you choices or decisions? It's just a disempowering thought?
They are buying up all the RAM today. Do you think "this is fine because in 5 years post-crash I can buy some cheap RAM"? If everyone with money is betting differently, do you have some information they don't, or is the whole economy just slipping away from you?
You experience luxuries today, that no king 1000 years ago could afford. Instant access to communication, food, medicine for the right price of course.
The consumer economy was great while it lasted but it's over now. We have machines that do useful mechanical work (engines) and useful intellectual work (llm-computers). Capital will move productive work from people to machines(if we let them), and the only jobs left will be delivery driver and warehouse, and then those will be gone too.
Human population was exponential and now its flat, but that's a function of what exacly? It could go back down to 1 billion or less. When jobs demanded a person supply was ready to match it. When jobs dont demand a person? Go to a degrowth rally take the temperature (and average age and child-per-person ratio) to get a taste of the future shape of supply and demand in a pessimistic world of sentences that don't have subjects just vague plattitudes. Are they net shutting down grade schools or building them in your neck of the woods?
Whenever I encounter these people I'm reminded of the meme about the baby who has doubled his weight in the three months since birth. At that growth rate, he'll weigh trillions of pounds by age 10.
It seems to me
we’re already at the top of the S curve, not at the toe of an exponential curve. At least with LLMs. Better training data will make small improvements, better architecture will make it less compute intensive, and all these “hyper-scale” data centers will make it cheap and ubiquitous. But none of that is it getting exponentially more intelligent.
I completely agree, but I also think that an industry disrupting architecture tweak akin to the “Attention is all you need” is VERY possible to emerge at any moment.
It feels like the cognitive gaps on current LLMs are indeed structural, but also that if we solve that structural issue with a new or extended transformer type of architecture, we’ll be looking at a whole new ballgame.
I mean, basically we’re just looking at needing some type of new post training learning architecture. It’s very clear that extending context windows isn’t that. What’s needed is an honest to god, continuous learning and modification process.
People have been saying this since GPT-1. This idea that we can only squeeze a little bit more of intelligence out of LLMs isn't a new one. And thus far, it has always been wrong.
I found the AI 2027 paper to be overly optimistic, but not wholly fantastical. This paper feels wildly speculative, and relies on premises I am not confident even pass surface reasoning. Even under optimistic conditions, we are not going to see robots "capable of 95% of all cognitive and physical tasks" by 2035. Nor do I think a 74% unemployment rate is even remotely possible. Economic collapse would implode AI development long before those figures were plausible.
The "and physical" is the part I'm particularly skeptical of. Sure, drones are scary, but nobody's really solved getting a robot to deliver a package to your front porch in a civilian setting, and it seems unlikely to be solved quickly.
It has. For example mechanization of argiculture in places where it didnt coincide with a manufacturing boom (latin america, india, africa) resulted in shantytowns and long term unemployment.
If I took you back to 2020 and said in a little over 5 years there will basically be no human coders writing code anymore you'd almost certainly not believe me.
And similar things can be said about many technologies in recent history – cars replacing the horse, first flight to man on the moon, even the creation of early internet to its mass adoption.
You're talking generally a decade or 2 for society to completely change from the rapid advancement of a new technology.
I'm not saying I agree with the 2035 prediction, but it doesn't seem impossible to me, if AI can help us improve the pace that we're already developing disruptive robotics.
In 2010 the idea of self-driving cars and autonomous delivery drones seemed very sci-fi and a long way out. But today, just 15 years on, these things are increasingly starting to be rolled out.
If they dropped that 95% number to 50-60%, I think I'd probably lean towards agreeing. Not because it makes sense in my gut, but because the logical part of my brain knows exponential trends (if one exists) do things that we wouldn't instinctively predict. But even if you assume exponentials 95% does seem very high.
> "If I took you back to 2020 and said in a little over 5 years there will basically be no human coders writing code anymore you'd almost certainly not believe me."
It's 2026, one year after your predicted date, and that still hasn't happened though.
This is dangerously naive and misguided. They claim to want to avoid centralization of control but propose a world police state of AI regulation. Governments exerting this much control will only end in war and tyranny.
Did anyone else catch the logical inconsistency between Plan C and A?
Plan C:
> "... fewer and fewer humans are needed to conduct AI R&D, meaning that covert projects are easier and easier to pull off without detection."
Plan A:
> "... training AIs requires large numbers of AI chips. Most AI chips are in giant datacenters.50 AI datacenters are typically big enough to be visible from space, and power-hungry enough to require conspicuous infrastructure. New AI chips can only be manufactured at a handful of fabrication plants (fabs), located mostly in Taiwan, South Korea, the US, and China. The US and China negotiate with the countries that have a major role in the chip supply chain, and they require each major datacenter owner (and their upstream suppliers, including chip fabs) to publicly declare their major purchases and sales."
Plan A requires properties of AI training that Plan C requires do not exist.
This is by far the most realistic optimistic AI takeoff scenario I've seen, and more specifically it's the first one I've read that deals with both the AI alignment and power concentration issues in a sufficient way, even in a world where hard alignment is assumed (in this scenario the AIs are assumed to be misaligned until ~2038-39).
Bravo, and I hope it has the impact on the AI safety field it deserves to have.
... and in the meantime people are looking at their AI bills and realizing tokens aren't worth what they cost. The frontier is getting the cost down, not getting intelligence up. In a cage match between this guy and "Ed", Ed wins.
Sounds like another Chinese Op to me; Ensuring Chinese compliance would be incredible hard to enforce or to check.
Look, I am scared of where we are heading, but I cannot see how we can change the dilemma towards mutual cooperation unless, as humans tend to do, only react massively after something really bad happens.
NYT reported today that Russia and China are funding anti-datacenter and anti-ai hysteria on western social media.
Always easier to boost something already existing on social media than manufacture it themselves, then wildly blow it out of proportion to make it seem urgent and important.
138 comments
[ 3.1 ms ] story [ 125 ms ] threadhttps://www.astralcodexten.com/p/introducing-plan-a
They being the US and China and by agreement.
It would be ideal, but there’s far too much money on the table to overcome human nature.
So my hope is we hit some kind of limits naturally.. Wishful thinking?
If there is any movement to pause AI development, it will come from the general public's dislike of these companies. Not from the AI safety angle.
Are there examples of where we have collective decided not to pursue knowledge? Successfully?
I guess nuclear weapons might be the best example though research doesn't seem have to actually "stopped" as much as gone underground and we still have country trying to climb that ladder.
But I don't know how relevant that is to LLMs/AI. It almost feels like pandora's box is open and our only option is continue to improve them. There is clearly value in what they do and while I can absolutely see the dangers, for example: authoritative governments and surveillance, I'm not convinced to throw the baby out with the bathwater.
All of technology back to the printing press (and probably before that) could also be said to make it easier for governments to oppress their citizens. Making laws (and enforcing them!) to prevent governments from doing these things feels like that route forward, not trying to stick our heads in the sand.
Perhaps I'm horribly naive, perhaps I just see the SciFi future I've spent my life reading and dreaming about on the horizon and I'm blinded by the reality, perhaps my ideals around "knowledge deserves to be free/accessible" are misguided. I don't know.
A resource extraction based economy sees people as slaves. The true source of power is the resource, people are just a means to an end, so you mistreat the people as much as you can get away with in pursuit of the resource while avoiding revolt.
With stable infrastructure, the government makes far more from an educated, rich population that it can tax and use the innovation from. It’s against its own quest for power to interfere too much in the prosperity of its citizens. The incentives are aligned.
Solving the AI problem isn’t about stopping the tech or making a bunch of brittle laws. It’s always been about alignment: aligning the large AGI-like entities that are the modern state, the modern economy, representative democracy, or AGI itself, with human prosperity
Yasha Levine wrote about how this narrative was preceded by a forgotten one where MIT students protested because the computers were going to be linked to government databases and share data on anti-Vietnam war activists. Despite protestations, activists were correct and this happened, and now it happens at huge scale.
http://yashalevine.com/surveillance-valley
It's not clear in this context what you actually mean by "government." You are assigning agency to something in a way that seems like a reification. While a bureaucracy can seem to have a life of its own, isn't it generally people who seek power?
... recently, as in the last 10 years?
Studying human bio-diversity since WW2 is the most obvious example, though it hasn't been entirely successful.
Genomics is what finally broke the barrier, especially in the last decade or so.
Intrinsically, the knowledge humans choose not to pursue will not be much publicized. There's limited value in calling attention to it and it doesn't make for good entertainment. Plenty of examples provided by other comments nonetheless.
> Perhaps I'm horribly naive, perhaps I just see the SciFi future I've spent my life reading and dreaming about on the horizon and I'm blinded by the reality, perhaps my ideals around "knowledge deserves to be free/accessible" are misguided. I don't know.
I don't personally think there's intrinsic benefit in disseminating arbitrary knowledge. There's quite some difference between the printing press and nukes.
> Are there examples of where we have collective decided not to pursue knowledge? Successfully?
human GMO, some bioweopns, I'm sure theres a long list of awful stuff no one wants to exist.
On a funny note, I think their prompt was:
"Hey Fable. Please attribute every piece of scientific and economic progress to AI until 2040. And predict every major geopolitical event. Make no mistakes."
Isn’t that like all of the Middle Ages where we replaced knowledge with an alternate religious reality.
My impression from the origin of the bioweapons convention is that collectively people decided that these things are too dangerous in various ways for any advantage that might be derived from them.
I'd bet that in most places 9 years is about the time needed to build a residential building. I think a good way to think about this is to think of this as producing a serial car. From pitching and capital acquisition to building a prototype to software, regulatory and then the final product which needs multiple factories and supply chains. Yes, of course robots sound cooler and there are compounding effects yada yada, but on the other side there are as many obstacles as things that accelerate this product (like capital acquisition and fearmongering of gov to bend regulatory stuff faster).
LLMS are 4 years old and the companies that sell them 10x every year. What evidence can you cite? Could you convince a disinterested 3rd party you have anything other than cope? What facts about the world make you think this is anything other than the new (and probably temporary) normal?
There is plenty falsifiable in this in ai-2027.com, and they have not gotten everything right. But some things they have: for example, the Pentagon has already invoked export controls to restrict the deployment of a frontier model. This level of government oversight wasn't predicted until 2027 in the original scenario.
They are buying up all the RAM today. Do you think "this is fine because in 5 years post-crash I can buy some cheap RAM"? If everyone with money is betting differently, do you have some information they don't, or is the whole economy just slipping away from you?
You experience luxuries today, that no king 1000 years ago could afford. Instant access to communication, food, medicine for the right price of course.
The consumer economy was great while it lasted but it's over now. We have machines that do useful mechanical work (engines) and useful intellectual work (llm-computers). Capital will move productive work from people to machines(if we let them), and the only jobs left will be delivery driver and warehouse, and then those will be gone too.
Human population was exponential and now its flat, but that's a function of what exacly? It could go back down to 1 billion or less. When jobs demanded a person supply was ready to match it. When jobs dont demand a person? Go to a degrowth rally take the temperature (and average age and child-per-person ratio) to get a taste of the future shape of supply and demand in a pessimistic world of sentences that don't have subjects just vague plattitudes. Are they net shutting down grade schools or building them in your neck of the woods?
strongly, no. its just hard to distinguish them. for example, radioactive decay. cmon
It feels like the cognitive gaps on current LLMs are indeed structural, but also that if we solve that structural issue with a new or extended transformer type of architecture, we’ll be looking at a whole new ballgame.
I mean, basically we’re just looking at needing some type of new post training learning architecture. It’s very clear that extending context windows isn’t that. What’s needed is an honest to god, continuous learning and modification process.
250 years of constant automation has never produced large scale unemployment, despite obsoleting everyone's jobs several times over.
In the industrialized world, old professions disappear all the time, and are replaced with new ones. I saw somewhere that it's about 2% per year.
And similar things can be said about many technologies in recent history – cars replacing the horse, first flight to man on the moon, even the creation of early internet to its mass adoption.
You're talking generally a decade or 2 for society to completely change from the rapid advancement of a new technology.
I'm not saying I agree with the 2035 prediction, but it doesn't seem impossible to me, if AI can help us improve the pace that we're already developing disruptive robotics.
In 2010 the idea of self-driving cars and autonomous delivery drones seemed very sci-fi and a long way out. But today, just 15 years on, these things are increasingly starting to be rolled out.
If they dropped that 95% number to 50-60%, I think I'd probably lean towards agreeing. Not because it makes sense in my gut, but because the logical part of my brain knows exponential trends (if one exists) do things that we wouldn't instinctively predict. But even if you assume exponentials 95% does seem very high.
It's 2026, one year after your predicted date, and that still hasn't happened though.
elaborate. regulation -> war, how?
I am not sure where they believe that amount of capital could come from. It would require central bank level money printing never seen before.
https://ai-2040.com/supplements/compute-supplement
If we're producing 10x as much, why not print 10x as much money? The goods and services each dollar could buy would remain similar.
Plan C:
> "... fewer and fewer humans are needed to conduct AI R&D, meaning that covert projects are easier and easier to pull off without detection."
Plan A:
> "... training AIs requires large numbers of AI chips. Most AI chips are in giant datacenters.50 AI datacenters are typically big enough to be visible from space, and power-hungry enough to require conspicuous infrastructure. New AI chips can only be manufactured at a handful of fabrication plants (fabs), located mostly in Taiwan, South Korea, the US, and China. The US and China negotiate with the countries that have a major role in the chip supply chain, and they require each major datacenter owner (and their upstream suppliers, including chip fabs) to publicly declare their major purchases and sales."
Plan A requires properties of AI training that Plan C requires do not exist.
My early analysis of the analysis:
https://lifearchitect.substack.com/p/the-memo-special-editio...
Bravo, and I hope it has the impact on the AI safety field it deserves to have.
Look, I am scared of where we are heading, but I cannot see how we can change the dilemma towards mutual cooperation unless, as humans tend to do, only react massively after something really bad happens.
Always easier to boost something already existing on social media than manufacture it themselves, then wildly blow it out of proportion to make it seem urgent and important.