If carbon taxes are already a lethal policy for an political campaign, it's absurd to think that fears of ASI will create any real movement around pausing AI.
If there is any movement to pause AI development, it will come from the general public's dislike of these companies. Not from the AI safety angle.
If that is true and one cares about a moratorium on progress in the US then it seems like the number one way is to meet people where they are: so water use misinformation, degrowth, power supply constraints. That does place all the people who push for these things in a different light. They may well be attempting to do what the AI safety labs are ostensibly trying to do.
As an AI safetyist, one’s closest ally (in a distributed coordinated way) is the populist misinformer. Fascinating.
I don't think it's a gotcha at all, they openly said it and were pondering how that might help them. Misinformation plays more to sides that are wrong about more things, and I don't consider misinformation my ally in anything.
I'm sure some people will have issue with my phrasing but, honest question:
Are there examples of where we have collective decided not to pursue knowledge? Successfully?
I guess nuclear weapons might be the best example though research doesn't seem have to actually "stopped" as much as gone underground and we still have country trying to climb that ladder.
But I don't know how relevant that is to LLMs/AI. It almost feels like pandora's box is open and our only option is continue to improve them. There is clearly value in what they do and while I can absolutely see the dangers, for example: authoritative governments and surveillance, I'm not convinced to throw the baby out with the bathwater.
All of technology back to the printing press (and probably before that) could also be said to make it easier for governments to oppress their citizens. Making laws (and enforcing them!) to prevent governments from doing these things feels like that route forward, not trying to stick our heads in the sand.
Perhaps I'm horribly naive, perhaps I just see the SciFi future I've spent my life reading and dreaming about on the horizon and I'm blinded by the reality, perhaps my ideals around "knowledge deserves to be free/accessible" are misguided. I don't know.
Taking the approach with AI that we took with the atomic bomb would be catastrophic. If the only people who are allowed to use this technology are governments, intelligence agencies, and a select few anointed companies, then the risk of authoritarian misuse will skyrocket.
I worry that any attempt to limit their use and development will be abused and misdirected. We are already seeing people like Anthropic doing this, they are trying to use anti-AI sentiment to engage in regulatory capture. Go watch Dario’s speeches about how open weight models are dangerous and how they are “not really open”. Everyone can see that much of this “safety” conversation is ultimately just a tactic to shut potential competitors out of the market and establish a monopoly/duopoly.
I agree. I guess I should have said something like:
"Stopping" LLM research just means it will be in the hands of a few who can abuse it. I'd rather a state of M.A.D. but instead of a handful of countries/governments it's millions/billions of people with access to the models (open ideally). Again, perhaps horribly naive or misguided, I understand that bioterrorism could (is?) a real problem as well as more "mundane" things like building a bomb (nuclear or otherwise).
I just feel like limiting access to governments or "blessed" entities is even worse.
No, of course not. That would be an insane trust fall. Even relatively small advances in technology give a country world dominating power. Fun fact: India was militarily superior to Britain in the 1600s—a gunpowder empire with a million soldiers—but was taken over by it in the 1700s. Britain’s edge was small: lighter, more maneuverable cannons, standardized ammunition, better military and political organization. Not a first world country versus a third world country—more like the dynamic US versus a sclerotic EU. And that modest edge led to 200 years of colonization.
If we slow down on ASI voluntarily we’d be allowing a gap to open up that would make the difference between colonial europe and colonized Asia/Africa look trivial. It would be insane.
You overstate the advantages of technology. Mughal India was fragmented and on a sharp decline. The British used politics, finance, and treachery to divide and conquer what was remaining.
That’s a possibility, but not the only one. The two most realistic ones are: we race ahead and maintain our status, or we slow down and open ourselves up to colonization.
Worth noting that it is the Europeans and Americans that have been colonial. Asian peoples have, with the prominent exception of the Mongols and Japanese Empire, pretty much not done that. In particular, China shut down its exploration program.
This is a settler-colonial mindset that reflects all the bad things we did onto everyone else. Notably, it's a current US ally that is most guilty of this.
China is currently occupying Tibet, which never consented to the occupation and has invaded Vietnam 30 times. It ruled Vietnam for about 2 centuries starting about 600.
"Asian people have pretty much not done that except for two teeny tiny indiscretions that each killed more civilians than all of Europe's and America's colonial incursions combined."
> The alternative isn’t “everyone dying.” It’s us holding all the cards.
That's one outcome, certainly, but not the only one nor, I contend, the most likely one.
A most likely outcome of ASI is human extinction, because there's more paths to an ELE outcome for humans from ASI than there is for non-extinction level outcome.
Your outcome is only possible if:
1. ASI is never able to escape the confines it is placed in.
2. ASI is benevolent to humans.
3. ASI decides, in the spirit of its benevolence, that it should restrict its involvement in humans.
If all three of the above conditions are met, then sure, your outcome is possible. If not, humanity as we know it will end.
It is unlikely that those 3 conditions will all hold, though.
Yes, this one. Look at our governance; look at our coordination-at-scale; look at our collective problem solving. It's abysmal, beyond hope. If we have global scale problems, we are not capable of solving them effectively. We are literally not intelligent enough to handle the problems we are creating. Between rivers of garbage and CO2 levels and war, we have proven ourselves to be woefully unintelligent at the scale needed. If we are lucky, our thin window of survival depends on getting a hell of a lot smarter, real quick.
Consider this: All that hardware that's going into those datacentres right now? In 5 years or so it'll all be on the secondary market... an influx of cheaper compute like you've never seen.
lol, how the fuck is ASI going to solve any of those problems? we already know _how_ to solve them; the problem is that we don't want to, collectively speaking, because certain powerful, wealthy people would loose out if we did. ASI wouldn't change anything. unless you think... all of human society is going to restructure itself around unquestioning worship of the Machine God and would therefore present no resistance to its proposed solutions?
The fantastical belief is that ASI will be able to make things happen because it knows the right things to say to the right people at the right time in just the right way to make them do whatever it wants them to do.
Certain powerful wealthy people aren't omnipotent, them losing out isn't the only blocker to progress.
If you read the scenario, you'll see that the regulations are mostly about what people can do with giant compute clusters, and not about the ideas themselves. The ideas themselves are required to be totally transparent to the public.
As for historic precedents: Human cloning, human genome editing, and mirror life seem like one precedent; nuclear weapons and nuclear energy another; come to think of it I think drone delivery was strangled by regulations too...? Plan A isn't a proposal to never build superintelligence, it's a proposal to build it more cautiously and transparently.
Human cloning and genetic editing isn't pursued because we agreed it's unethical. It's not pursued because at the current level of advancements, it's pretty useless. It's easy to ban something that's not very useful. The things we can do there are pretty niche.
If we had a way to make genetic edits to make humans a lot smarter, a lot stronger and live a lot longer? Or a way to quick-grow human bodies to adulthood in a couple years? Capabilities that private actors or countries may want, ethics be damned? That would be closer to what we have with AI right now.
It's not useless. We know enough about DNA to be able to make better humans, but people get real squickly real fast when you talk about that. It has stalled because of that. If we go in and just check the DNA for Downs or Parkinson's, we can't have a conversation on aborting the foetus without religious beliefs coming into play. Designer children aren't a thing, despite the ability to edit DNA to do specific things. Hair and eye color are easy enough to go in and edit for. Humanity has decided to opt out of doing that for now.
Marginally better humans. And IVF paired with embryo selection is very competitive in that niche.
Now, we're getting better at predicting polygenic traits, and we're getting beefy multi-edit pipelines that might provide a meaningful advantage over embryo selection working in multiple animals. But as of yet, the advantage of genetic editing in humans over just doing aggressive IVF and dredging the embryos for desirable traits is minor.
Bit of a chicken and egg problem there. Can't advance the tech fast without actively using it, can't actively use the tech until it's advanced enough for the benefits to override ethical concerns. So it's getting there, but at a glacial pace.
Yes, there are examples of where we have collective decided not to pursue a particular technology tree.
Following the invention of recombinant DNA technology, scientists convened the Asilomar Conference in 1975. They established a voluntary self-moratorium on certain types of genetic engineering until strict laboratory containment protocols were created.
In the 1980s, bioethicists, theologians, and researchers established a hard ethical line between somatic editing (treating an existing patient's non-reproductive cells) and germline editing (altering future generations).
No one has performed the latter form of genetic engineering except for Chinese scientist He Jiankui in 2018. In response, Beijing heavily tightened its laws, classifying heritable gene editing as a high-risk medical technology subject to the penal code. He Jiankui was sentenced to three years in prison.
I completely can see why we'd want to, for ethical reasons, ban germline editing, and I want to be clear that I agree doing so cannot be done ethically, but there is a part of me that is wistful for what could have been. Same with things like CRISPR but it's probably just fun to dream and the reality would be a nightmare.
The decision to not go with the development of extremely large thermonuclear weapons might count - the US Sundial Project was supposed to be about 10 gigatons of TNT. Not the most practical weapons but once you get to a certain size delivery arguably stops being a problem - its going to kill everyone anyway so doesn't matter where you let it off!
Isn't the US famous for not even signing a lot of world treaties like climate accords and others?
This is an interesting subject and conversation, but it's moot having it in these culture-centric forums. I wonder if there are Russians discussing plausible scenarios in Vkontakte groups, or Chinese doing the same in whatever Alibaba group sites they use.
The problem is that we are all skewed by our media, our ideas and our culture. These type of discussions need the highest kind of political interactions.
It's fascinating, specially for someone who lives in a "third world" country, non-aligned to any of these 3 superpowers. Whatever transpires, we are at tge mercy of these (and no, US hasn't treated us "better").
My opinion is that there's no turning back on AGI development. I dont think current governments are capable of getting into an agreement of that size. Specially given the Isolationist stage in the cycle we live in. (In contrast with for example the CFC and Ozone layer issue we had in the 1990s, when the planet was in a globalist kind of stage)
The Russians signed it and then completely ignored the agreement.
That is an objective fact and has nothing to do with the rest of the cultural relativism you're engaging in.
Whataboutism about the US _not signing_ things is also irrelevant.
The whole AI treaty discussion is about a treaty countries will _actually sign_ and the relevance of Russia cheating on the BWC is that the signatures of some countries are absolutely worthless, which _is_ relevant.
While technology has empowered governments, it’s also empowered the individual, and more importantly shifted the material dynamics to better align the incentives of governments with the people. Democracy followed material change, it didn’t precede it. Democracy came about because it was optimal for a power seeking government, not out of the kindness of their heart.
A resource extraction based economy sees people as slaves. The true source of power is the resource, people are just a means to an end, so you mistreat the people as much as you can get away with in pursuit of the resource while avoiding revolt.
With stable infrastructure, the government makes far more from an educated, rich population that it can tax and use the innovation from. It’s against its own quest for power to interfere too much in the prosperity of its citizens. The incentives are aligned.
Solving the AI problem isn’t about stopping the tech or making a bunch of brittle laws. It’s always been about alignment: aligning the large AGI-like entities that are the modern state, the modern economy, representative democracy, or AGI itself, with human prosperity
This is a just so story. The main issue today is the lack of democracy in the country and the use of technology to surveil and govern a restive population as the government has less and less legitimacy. The narrative you are telling is the heroic tale of computing and the internet c. 1990-2010.
Yasha Levine wrote about how this narrative was preceded by a forgotten one where MIT students protested because the computers were going to be linked to government databases and share data on anti-Vietnam war activists. Despite protestations, activists were correct and this happened, and now it happens at huge scale.
Yes, and RMS was correct in his Right to Read and so many other things - we're seeing the slow death of the never-enshrined-in-law right to compute. Luckily open-source is big enough to slow this down; we should all be pretty amazed and appreciative that there even are open-weights models at all, out there, because it is a profoundly democratizing thing.
I don't think parent commenter means tech in the modern sense. Seems like they're describing a transition to democracy which started centuries ago, not decades.
> Democracy came about because it was optimal for a power seeking government, not out of the kindness of their heart
It's not clear in this context what you actually mean by "government." You are assigning agency to something in a way that seems like a reification. While a bureaucracy can seem to have a life of its own, isn't it generally people who seek power?
> Are there examples of where we have collective decided not to pursue knowledge? Successfully?
Intrinsically, the knowledge humans choose not to pursue will not be much publicized. There's limited value in calling attention to it and it doesn't make for good entertainment. Plenty of examples provided by other comments nonetheless.
> Perhaps I'm horribly naive, perhaps I just see the SciFi future I've spent my life reading and dreaming about on the horizon and I'm blinded by the reality, perhaps my ideals around "knowledge deserves to be free/accessible" are misguided. I don't know.
I don't personally think there's intrinsic benefit in disseminating arbitrary knowledge. There's quite some difference between the printing press and nukes.
If AGI means that AI+robotics can robustly substitute for human labor, and robots are cheaper and faster to build than humans, then (a) anyone ruthless enough can zerg rush and defeat any nations that don't discard humans, (b) no one without a massive robot army will be needed in any way by their rulers. If this isn't a recipe for a horrific outcome, what is?
AGI doesn't do away with nuclear MAD, it just messes with economics and makes many people temporarily jobless. Temporarily because in a literal sense RLVR needs verification to train off of, and a lot of jobs cant be easily checked if theyre done. this includes AI safety people, preschool teachers, psychologists, and probably a lot more, including most of their bosses
Nuclear weapons are mostly a threat to cities, and a zerg rush build would just not build any.
I think that if all we get is skill cloning for formally verifiable tasks, that is not "AGI" and does not doom us. It probably won't even doom software engineers, because their whole job is also not formally verifiable. I don't trust that the next ten trillion dollars of AI research won't give us data efficient continual learning and agency on par with humans, which is what I would call AGI.
It's not clear that "super intelligence" is a meaningful concept. This presumes that our concept of intelligence can continue to grow beyond human capacity as opposed to asymptotically approaching what humans recognize as "very intelligent". Perhaps, for instance, how we evaluate intelligence is bounded not by some quantitative capacity but rather our inability to agree on basic concepts/values. And do we even have tasks that a supposed superintelligence can tackle but humans cannot?
I predict that what we consider "super intelligence" is just sheer computational power, but any potential of a very capable agent is bounded by the needs/wants of the person wielding it. That is: even if we were to hand, say, Elon musk this "super intelligence", most humans would consider it relatively stupid because the person wielding it is still a person with stupid goals and values.
Or, to put it another way, I suspect we already do have a superintelligence and have longer than any of us have been alive, and it's just "the market", and it is still incapable of overcoming the limitations of a few morons wielding immense power.... power they will never yield to some intelligence with values and goals "more intelligent" than their own (if such a concept is even meaningful), and intelligence wasted on the values and goals they do have.
The Orthogonality Thesis (by Nick Bostrom) says that intelligence and ultimate goals are independent. Those non-instrumental goals can’t be stupid nor right or wrong. Increasing intelligence will not change the goals only the capability to reach them.
Do note that the orthogonality thesis is a hypothesis, not something we have demonstrated. Weak versions of it (e.g. it is possible to have an intelligent agent with arbitrary goals) are more likely to be true than stronger versions (e.g. intelligent agents we build will have goals uniformly selected from the space of all possible goals).
LLM + scale = more intelligent, this can be proven more than empirically, https://arxiv.org/abs/2409.15318 shows that neural nets can fit a number of independent AND-gate operations in their weights,
- if you have a system that is large enough to store, lets say 10^12 AND gates (all frontier llms can do this)
- and this system can produce outputs based on previous things it has outputed
its turning complete, and RLVR on it is optimization over the space of algorithms. If an algorithm exists to do a task, and the task can be verifiably done, this finds the algorithm to solve the task most often.
it is obvious that this scales, from much-worse-than-human to slightly-worse-than-human, therefore it 100% can exceed humans.
No one can run a nuclear reactor on their phone but can run an AI
We were discussing AI in the 90s and it's been discussed before that.
The answer was always the same; hardware can't hang.
Now it can and will get even better.
The SaaS era fueled by ZIRP and ignorant Congress was a fluke that from an engineering perspective didn't produce anything but hype and same old
The generation enriched and empowered by it is just as temporary as Boomers. Little point in enabling their appeals at the expense of scientific progress that helps all of humanity.
China won't. Russia won't.
It's ridiculous to me the level QQing coming from Americans exploiting child sweatshop labor so they are free to ignore their own biological needs and keep a "knowledge work" job (talk about first world privilege) handing them wealth to go tour the poor villages they exploit.
Those workers never had a choice between college or the mines. So sorry 300 million Americans in a world of 8 billion.
We don't even want these jobs given how much bitching I have listened to the last 10-15. IMO the job creators and Congress saw how Millennials liked to be on the computer and went way too far into enabling such banal output.
Make healthcare and housing the economic tentpole. Both still need jobs and technology. But at least the outcome isn't a generational Ponzi scheme engineered by Boomers to enrich them and then let it all collapse when the majority realize those stocks were never real.
Perhaps the burning of the library at Alexandria would qualify. How intentional that was is somewhat in question, but the world certainly turned its back on the only collection of written knowledge and let it turn to ash.
> Scholars have interpreted Cassius Dio's wording to indicate that the fire did not actually destroy the entire Library itself, but rather one or more Library warehouses near the docks.[87][81][8][89] Whatever damage Caesar's fire may have caused, evidently the Library was not completely destroyed.[87][81][8][89][3] The geographer Strabo (c. 63 BC – c. 24 AD) mentions visiting the Mouseion, the larger research institution to which the Library was attached, in around 20 BC, several decades after Caesar's fire, indicating that it either survived the fire or was rebuilt soon afterwards.[87][8] Nonetheless, Strabo's manner of talking about the Mouseion shows that it was nowhere near as prestigious as it had been a few centuries prior. It is unknown whether this was due to historical decline or catastrophic destruction.[8] Despite mentioning the Mouseion, Strabo does not mention the Library separately, perhaps indicating that it had been so drastically reduced in stature and significance that Strabo felt it did not warrant separate mention.[8] It is unclear what happened to the Mouseion after Strabo's mention of it.[60]
> Further evidence for the Library's survival after 48 BC comes from the fact that the most notable producer of composite commentaries during the late first century BC and early first century AD was a scholar who worked in Alexandria named Didymus Chalcenterus, whose epithet Χαλκέντερος (Chalkénteros) means "bronze guts".[90][87] Didymus is said to have produced somewhere between 3,500 and 4,000 books, making him the most prolific known writer in all of antiquity.[90][82] He was also given the nickname βιβλιολάθης (Biblioláthēs), meaning "book-forgetter" because it was said that even he could not remember all the books he had written.[90][91] Parts of some of Didymus' commentaries have been preserved in the forms of later extracts and these remains are modern scholars' most important sources of information about the critical works of the earlier scholars at the Library of Alexandria.[90] Lionel Casson states that Didymus' prodigious output "would have been impossible without at least a good part of the resources of the library at his disposal".[87]
The Library, or part of its collection, was accidentally burned by Julius Caesar during his civil war in 48 BC, but it is unclear how much was actually destroyed
Human embryo genetic modification has been effectively taboo if not banned until just recently since WWII and the aftermath of the Holocaust. I think some people in the US are proposing doing it now but I don't think anyone has tried it yet.
I like your optimism and I think you will be vindicated. AI is democratic and AI talent is globally distributed. It will just take a while to get online. AI labs do not have a monopoly on human talent, and open source AI only empowers independent science and meritocracy.
On a funny note, I think their prompt was:
"Hey Fable. Please attribute every piece of scientific and economic progress to AI until 2040. And predict every major geopolitical event. Make no mistakes."
There is no rein on progress. You can buy electric motorcycles without speed governors. And you can drive them on public roads (subject to traffic laws) as long as you're properly licensed, registered, and insured. The distinction between electric "bikes" and motorcycles is mostly artificial.
It’s telling terrorists how to make bombs better apparently. Continuing to lower the barrier for that kind of stuff is clearly a negative in the “knowledge deserves to be free” world.
Biological weapons? Yes, there is research on defense, but no big arsenals of weapons etc.
My impression from the origin of the bioweapons convention is that collectively people decided that these things are too dangerous in various ways for any advantage that might be derived from them.
I can think of many examples that I won't name but you can imagine in biology/medical fields where certain lines of investigation are not performed due to ethical and legal repercussions.
People overestimate progress in physical world.
2035:
robot population will soon be larger than the human one
I'd bet that in most places 9 years is about the time needed to build a residential building. I think a good way to think about this is to think of this as producing a serial car. From pitching and capital acquisition to building a prototype to software, regulatory and then the final product which needs multiple factories and supply chains. Yes, of course robots sound cooler and there are compounding effects yada yada, but on the other side there are as many obstacles as things that accelerate this product (like capital acquisition and fearmongering of gov to bend regulatory stuff faster).
Hey, author here! Scott did contribute, but less than before.
On his blog he says: "I did a lot of writing for AI 2027 and was listed as a co-author. Some of my writing made it into Plan A too, but it was a bit less. The difference is of degree rather than kind, but because of this - and to give me more latitude to discuss it the way I like with less PR blowback - we decided not to put me as a co-author this time. I continue to be proud of having a part in this, small as it may be. (related: everything in this post is my opinion only, and not officially endorsed by the AI Futures Project)"
LLM adoption is 30% in the charts I saw googling for "ai adoption". An example of capibility: I have had Claude one shot an RL agent that learns connect four in 30 minutes. That's PhD level stuff.
LLMS are 4 years old and the companies that sell them 10x every year. What evidence can you cite? Could you convince a disinterested 3rd party you have anything other than cope? What facts about the world make you think this is anything other than the new (and probably temporary) normal?
What predictions about the technology are the authors making that you do not believe?
There is plenty falsifiable in this in ai-2027.com, and they have not gotten everything right. But some things they have: for example, the Pentagon has already invoked export controls to restrict the deployment of a frontier model. This level of government oversight wasn't predicted until 2027 in the original scenario.
Self improvement of agent-1 is not achieved. Sure, people in AI labs write python code with AI, but I doubt it resulted in 50% algorithmic effiecency in training. Writing python code never was the bottleneck, if it was, AI labs could hire more people to do it. And this is core of the prediction.
Nothing follows from this empty platitude though, right? It can't inform you choices or decisions? It's just a disempowering thought?
They are buying up all the RAM today. Do you think "this is fine because in 5 years post-crash I can buy some cheap RAM"? If everyone with money is betting differently, do you have some information they don't, or is the whole economy just slipping away from you?
You experience luxuries today, that no king 1000 years ago could afford. Instant access to communication, food, medicine for the right price of course.
The consumer economy was great while it lasted but it's over now. We have machines that do useful mechanical work (engines) and useful intellectual work (llm-computers). Capital will move productive work from people to machines(if we let them), and the only jobs left will be delivery driver and warehouse, and then those will be gone too.
Human population was exponential and now its flat, but that's a function of what exacly? It could go back down to 1 billion or less. When jobs demanded a person supply was ready to match it. When jobs dont demand a person? Go to a degrowth rally take the temperature (and average age and child-per-person ratio) to get a taste of the future shape of supply and demand in a pessimistic world of sentences that don't have subjects just vague plattitudes. Are they net shutting down grade schools or building them in your neck of the woods?
Degrowth is a deeply unpopular policy around the world. Where would one even go to find a degrowth rally? I have to imagine everyone there would be there ironically.
> * they net shutting down grade schools or building them in your neck of the woods?*
My area (rural Iowa) has had several new schools built in the last 10 years. Net gain for sure.
AI is just a tool. It will be a tool as we constantly push the requirements of what qualifies as consciousness. Therefore I'm not worried about it. Almost all the predictions about future growth of technology were incorrect. We don't have flying cars, hoverboards, fusion, the matrix and so on. The goal of each of us is to live a better life. It always be and forcing people to give up luxuries for some external agenda based on slippery slopes is cruel and selfish.
We should have been under water, hunted by AI, overpopulated, killed by terrorist, smitten by god for our sins and so on. Luckily all it took was our privacy and a lot of tax money to survive.
Whenever I encounter these people I'm reminded of the meme about the baby who has doubled his weight in the three months since birth. At that growth rate, he'll weigh trillions of pounds by age 10.
Point taken. But it's interesting that the example you give is one of exponential decay rather than exponential growth, which is the context of this discussion. Where in nature has anything ever grown exponentially in perpetuity? Bacteria in a petri dish multiply exponentially. Until they don't.
It seems to me
we’re already at the top of the S curve, not at the toe of an exponential curve. At least with LLMs. Better training data will make small improvements, better architecture will make it less compute intensive, and all these “hyper-scale” data centers will make it cheap and ubiquitous. But none of that is it getting exponentially more intelligent.
I remember being blown away by o1-o3 family of models finally stringing together coherent agentic tool calls to write and execute scripts semi-reliably for workloads in the several minutes before they would start hallucinating/flailing. GPT 5 was a bit ahead of that, but barely
Now we take for granted that the latest models can juggle between multiple browser tabs, applications, databases, simulators, docker etc to write, execute, e2e test and deploy full-stack applications over hours managing up to dozens of subagents, relatively untouched, without taking down prod even 1% of the time
The models now reliably execute more than a human can fit into their own context. It's magic
Yes, and we haven't even really begun to nail down computer-use agents yet (can you believe they're still basically just OCR'ing screenshots?)
Once we have something that experiences a desktop interface more like a human does, an entire swathe of tooling that has heretofore been nigh-impossible to automate moves into the fold, and that'll be another explosion of folks finally getting to join the agentic workflow world on their industry specific apps...
It's not the same thing. For example, given a GUI with a titlebar, title, subtitle, text, and buttons, a human can instantly understand spatially the relationship between these items. But a naive OCR of such a GUI would be a flat stream of text that loses a ton of information.
Most companies have already given up on AI. It was a bigger disappointment than big data.
The only companies sticking with AI at this point are the major players who have chasing their masssive overinvestments to the bitter end, crappy coding shops, and for some reason Starbucks and they'll all suffering badly in a year their customers begin demanding massive amounts of human coding to repair all the issues with the AI code.
Can you share what's your setup for all that orchestration? I feel way behind just asking Claude Code for code edits. Is there any site where people share different AI setups, besides youtube?
Fwiw, don't buy into all the hype that you're falling behind. Yes, AI does cool things now, but I would say the impact is still unproven past indie hackers or early-stage startups. And a lot of the esoteric setups people have created with things like OpenClaw have become outdated as quickly as they were conceived.
The popular thing is now to setup loops (eg I setup hourly integrations for Claude/Codex to 1) scrape my Linear, claim achievable tasks, and push PRs or 2) do root cause analysis on customer issues that evaded automated filters, to name a few)
Though for me, my setup still feels mundane. I have AGENTS.md, CLAUDE.md etc and a few skill files. These are purposefully light - tons of examples online you can pull from online. Mine are fairly personal to my setup and products.
Importantly, I also allow Claude and Codex to bypass permissions. Yes, there is a risk they wipe my machine. The productivity upside has been worth it, for me (haven't been burned yet, ~9+ months into running models this way, I have backups, use cloud etc).
As far as maintaining quality, one of the most helpful guardrails over the past year, for me, has been requiring my agents to pipe their changes to local reviewers through OpenCode, Cursor, etc agents to have a council of models with different biases reviewing the changes, and autonomously working towards a completed objective. No matter how good Claude or Codex gets, for example, I will probably always want a different model checking its work. Like GLM, (now with 4.5) Grok, Composer.
Several OpenAI, Anthropic, xAI employees, and popular AI engineers post on X and share helpful tips & updates. Highly recommend for keeping a pulse on startups and AI. I haven't found something close, honestly, other than when I spend time in SF talking to people.
5.6 to 5.0 is a big enough of a jump to say yes. if it was 5.4 to 5.6 it would be a bit easier to say it only feels true because of that, but 5.6 is definitely better than 5.0. I don't have anything empirical to point at though, which is your point, but August 2025 for 5.0 vs July 2026 is almost a year later, and it's not just vibes that it's better, despite not having an objective metric to point at. It would be more scientifical to have numbers and shit to point at and there are some benchmarks out there, but you have to dig into them and really understand them in order to believe in exactly what they're testing, and I'm betting you haven't.
> AI labs may also be able to reap tremendous benefit from these inference-scaled models by using them as part of the training process. If so, the large scale-up of compute resources could go into post-training rather than deployment. This would have very different implications for AI governance.
> ...
> So iterated distillation and amplification provides a plausible pathway for scaling inference-during-training to rapidly create much more powerful AI systems. Arguably this would constitute a form of ‘recursive self-improvement’ where AI systems are applied to the task of improving their own capabilities, leading to a rapid escalation.
So "inference scaling is required to scale capabilities" doesn't mean that we're reaching the top of the S-curve in intelligence. If anything, it could mean a shorter timeline and more unpredictable landscape for governance (e.g. due to securing weights no longer as effectively preventing escalation, more in the article).
> So "inference scaling is required to scale capabilities" doesn't mean that we're reaching the top of the S-curve in intelligence.
On its own it wouldn't. But that article came before the later article https://www.tobyord.com/writing/hourly-costs-for-ai-agents which adds the claim that inference (along with everything else being employed at present) is scaling poorly with increasing task lengths. Now maybe the December 2025 claim is wrong, or maybe things will change soon, but the February 2025 article surely doesn't establish either of those.
I completely agree, but I also think that an industry disrupting architecture tweak akin to the “Attnetion is all you need” is VERY possible to emerge at any moment.
It feels like the cognitive gaps on current LLMs are indeed structural, but also that if we solve that structural issue with a new or extended transformer type of architecture, we’ll be looking at a whole new ballgame.
People have been saying this since GPT-1. This idea that we can only squeeze a little bit more of intelligence out of LLMs isn't a new one. And thus far, it has always been wrong.
I found the AI 2027 paper to be overly optimistic, but not wholly fantastical. This paper feels wildly speculative, and relies on premises I am not confident even pass surface reasoning. Even under optimistic conditions, we are not going to see robots "capable of 95% of all cognitive and physical tasks" by 2035. Nor do I think a 74% unemployment rate is even remotely possible. Economic collapse would implode AI development long before those figures were plausible.
The "and physical" is the part I'm particularly skeptical of. Sure, drones are scary, but nobody's really solved getting a robot to deliver a package to your front porch in a civilian setting, and it seems unlikely to be solved quickly.
The book Sentient is not about AI but abount the most amazing physical senses some other animals have.
The theme of the scientific findings is that while humans excel with none of our physical sensors, we do very well across the board in making use of them thanks to our relatively huge brains.
And fantastical amounts of compute power is exactly what are handing over to AI. The fact that their training data isn't perfect may matter less.
Nobody’s even solved a self-driving vehicles yet, not in in the sort of “they took over everything and put every uber and truck driver out of business” kind of way.
Maybe they will soon but it’s massively far behind the kind of timeframe AI 2027 would have suggested.
I disagree. It has not been figured out enough to take humans out of the equation from a functional standpoint, not just a regulatory one.
This past 4th of July weekend a Waymo ran over an actively burning firework in a low speed scenario, the kind of thing a 95 year old driver would have avoided.
It has. For example mechanization of argiculture in places where it didnt coincide with a manufacturing boom (latin america, india, africa) resulted in shantytowns and long term unemployment.
If I took you back to 2020 and said in a little over 5 years there will basically be no human coders writing code anymore you'd almost certainly not believe me.
And similar things can be said about many technologies in recent history – cars replacing the horse, first flight to man on the moon, even the creation of early internet to its mass adoption.
You're talking generally a decade or 2 for society to completely change from the rapid advancement of a new technology.
I'm not saying I agree with the 2035 prediction, but it doesn't seem impossible to me, if AI can help us improve the pace that we're already developing disruptive robotics.
In 2010 the idea of self-driving cars and autonomous delivery drones seemed very sci-fi and a long way out. But today, just 15 years on, these things are increasingly starting to be rolled out.
If they dropped that 95% number to 50-60%, I think I'd probably lean towards agreeing. Not because it makes sense in my gut, but because the logical part of my brain knows exponential trends (if one exists) do things that we wouldn't instinctively predict. But even if you assume exponentials 95% does seem very high.
For a specific technology to become viable, you often need progress in several individual technologies to occur. Some of these will be exponential and surprise us, but many will see slower progress.
Either way, I suspect progress in robotics will be slower than AI, but I also think we'll see a lot of investment in robotics over the next decade given that AI opens up a lot of new potential applications for robotics.
I suspect any task that doesn't require extreme battery life or extreme dexterity will be doable by robots much sooner than most people think today. I think we could have decent-ish humanoid robots within a decade – robots that can prepare food, clean hotel rooms, empty bins, etc. I think Elon's bet on Optimus is quite likely to be viewed as very ahead of the curve with time.
However, it will take much longer for robots to do everything a human body can, but then most jobs don't require we push the physical limits of human body.
That's why I think 60% is reasonable. And even then I'm not saying it will happen, I just don't think that's a bad bet if given even odds.
You say "just" 15 years, but Waymo is still only available in a few cities. That seems more like a slow, cautious rollout to me, not a fast takeoff. Society has had a lot of time to get used to (and tired of) the idea and come up with regulations.
My guess is that the deployment of other types of robots will be a similarly slow grind.
That's unlike the Internet, smart phones, and coding agents, which got user adoption at a much quicker pace.
This is dangerously naive and misguided. They claim to want to avoid centralization of control but propose a world police state of AI regulation. Governments exerting this much control will only end in war and tyranny.
If the production, and distribution of the tech is confined to one or two companies, then those two companies and the resulting power that comes to them, makes them the de facto world government. (Democratic choice may be a farce but it is a useful friction on net.)
If they own all the RAM, models, and the means to do any work, then you are at their mercy. They will buy all the RAM, leaving you none, then all the transport, then all the electricity. You will be as boxed out of the current economy as the Amish and it will get its plug pulled.
Gradual Disempowerment is the default plan right now. War and tyrrany are not remotely the worst case scenario. I'll take Butlerian Jihad over being turned into cattle any day.
Imagine solving for equilibrium with two classes of beings. One requires agricultural land and 20 years to become individually productive and barely maintains a healthy population in a entertainment saturated landscape. The second eats only electricity and is ready to work on day 1. Round 1 goes to the strongest gorilla for sure, round 100?
If LLMs had come to earth in spaceships would you have welcomed them into your work and your home?
Did anyone else catch the logical inconsistency between Plan C and A?
Plan C:
> "... fewer and fewer humans are needed to conduct AI R&D, meaning that covert projects are easier and easier to pull off without detection."
Plan A:
> "... training AIs requires large numbers of AI chips. Most AI chips are in giant datacenters.50 AI datacenters are typically big enough to be visible from space, and power-hungry enough to require conspicuous infrastructure. New AI chips can only be manufactured at a handful of fabrication plants (fabs), located mostly in Taiwan, South Korea, the US, and China. The US and China negotiate with the countries that have a major role in the chip supply chain, and they require each major datacenter owner (and their upstream suppliers, including chip fabs) to publicly declare their major purchases and sales."
Plan A requires properties of AI training that Plan C requires do not exist.
By legislative design? If a nuclear bomb could be made with hardware store finds the world would already be over. Big collaborative works raise the stakes and the observability for surveillance. Apply for a job at a defense supplier or even and energy company.
If AI production is limited to big labs and big data centers then it is de facto contained and monitored. If you know where all the ASML machines are then you know the reproduction rates of chips. If no one can buy or build the machines required to concentrate uranium or plutonium to critical levels then the threat is contained and monitored.
You can dig up all the Uraninite you want. It was never much of a secret that uranium had dark applications. The machines and processes where thankfully big and expensive enough that only the most focused bad actors could aquire them and then hold the world hostage to the degree they do. If al-qaeda or isis could have used $40 bombs from home depot instead of expensive planes they would have (and they do).
You have to legislate and control the big, expensive, and slow things. Dynamite and phentanyl are so dangerous because they move much more easily. Freedom does not have to be a suicide pact. If the inconvenience of requiring prescriptions or access to dynamite reduces harm then it is net positive?
In Plan C the government essentially misses the opportunity to implement the multinational deal while the threat of covert projects is still low (fewer latest-gen chips unaccounted by tracking measures, worse models/algorithms to use for RSI). That's why it says the probability of a deal is lower and lower each month rather than outright zero.
This is by far the most realistic optimistic AI takeoff scenario I've seen, and more specifically it's the first one I've read that deals with both the AI alignment and power concentration issues in a sufficient way, even in a world where hard alignment is assumed (in this scenario the AIs are assumed to be misaligned until ~2038-39).
Bravo, and I hope it has the impact on the AI safety field it deserves to have.
... and in the meantime people are looking at their AI bills and realizing tokens aren't worth what they cost. The frontier is getting the cost down, not getting intelligence up. In a cage match between this guy and "Ed", Ed wins.
Sounds like another Chinese Op to me; Ensuring Chinese compliance would be incredible hard to enforce or to check.
Look, I am scared of where we are heading, but I cannot see how we can change the dilemma towards mutual cooperation unless, as humans tend to do, only react massively after something really bad happens.
I recommend actually reading their recommendation, because they get into the weeds about precisely how the US and China could address this in a trustless/auditable way. The TL;DR is that basically all of the relevant compute can be tracked.
Pretty accurately what? So now OpenBrain has an Agent-1 that makes their algorithmic progress 50% faster than other companies? If it was 50% more CVEs, that would be something, but I doubt any meaningful self improvement is achieved which the competitors are slower due that, which is the core of the prediction for 2026.
But do they at any useful scale? China is probably the only other country deploying AI at any appreciable fraction of their economy, and it's certainly much less than the US.
NYT reported today that Russia and China are funding anti-datacenter and anti-ai hysteria on western social media.
Always easier to boost something already existing on social media than manufacture it themselves, then wildly blow it out of proportion to make it seem urgent and important.
Everyone lives in a world deeply affected by social media. Even if you've never looked at a screen your entire life, you have spent thousands of hours talking to and being informed by people who did.
World, yes, as people around you including politicians and businessmen have had their brains pickled.
But I doubt the influence. I’ve been free for years, and I can always spot who’s spends a lot of time in TikTok/Twitter/Instagram—it’s like talking to someone from another planet. It mostly sounds weird and sad, more apt to annoy and alienate their friends than inform or influence them.
Russia and China aren’t the ones constantly telling us AI will put us all out of a job, here’s why that’s a good thing, and why the government should dedicate billions of dollars to incumbent AI providers.
Surprisingly enough the us constitution specifies a right to free speech. Meanwhile who is funding the past two decades of pie in the sky bullshit from silicon valley in the media? Is promoting and unpopular opinion illegal? Since when do we judge the merits of an argument based on who articulates it?
Which then gives more ammunition to folks like Kevin O'Leary et al to pretend any objections to data centers or AI are Chinese plots, or "hysteria" as you put it.
While I can admit some of the anti-data center arguments are overblown, many are more than valid in my opinion. Data centers are fundamentally extractive technologies. They are enormous, windowless boxes that take resources from one location to make someone else in a far off location enormously rich and powerful, with extremely few benefits to the local community.
Plus, as another commenter mentioned, it's not exactly like the Chinese and Russians have been fanning the flames that AI is going to take all of our jobs - it's the leaders of the frontier AI companies in the US saying that. Remind me again why I think putting up a giant data center in my state, that was proposed to use more electricity than my state already currently uses, is a good thing for the average joe where I live??
Honestly, I feel like many commenters here are in their own bubble and don't understand how much AI and tech generally is widely viewed as a net negative for society by huge swaths of the the population, and I don't really think it's an unwarranted perception.
Most of the issues with datacenters could be solved by a) investing in energy b) ramping up RAM and chip production, and c) enforcing already long established rules around industrial water management.
These tech companies are already investing heavily into solar, natural gas, and nuclear https://techcrunch.com/2025/03/30/data-centers-love-solar-he... this would be normal stuff in China where they spend the last decade investing heavily in solar and are bringing something like 60 nuclear reactors online.
These datacenters aren't particularly consumptive of water compared to most other industries in that regard and we've already seen states enforce rules against Meta who immediately paused their datacenter when water issues were detected (following mandatory monitoring).
Chip production is lagging but most projections I've seen is it will normalize in about 5yrs. Not to mention there will be further demand for robotics and self driving cars, so ramping up chips should be a normal thing like ramping up green/nuclear energy. Delaying it won't solve any current issues.
410 comments
[ 3.3 ms ] story [ 78.2 ms ] threadhttps://www.astralcodexten.com/p/introducing-plan-a
They being the US and China and by agreement.
It would be ideal, but there’s far too much money on the table to overcome human nature.
So my hope is we hit some kind of limits naturally.. Wishful thinking?
If there is any movement to pause AI development, it will come from the general public's dislike of these companies. Not from the AI safety angle.
As an AI safetyist, one’s closest ally (in a distributed coordinated way) is the populist misinformer. Fascinating.
Are there examples of where we have collective decided not to pursue knowledge? Successfully?
I guess nuclear weapons might be the best example though research doesn't seem have to actually "stopped" as much as gone underground and we still have country trying to climb that ladder.
But I don't know how relevant that is to LLMs/AI. It almost feels like pandora's box is open and our only option is continue to improve them. There is clearly value in what they do and while I can absolutely see the dangers, for example: authoritative governments and surveillance, I'm not convinced to throw the baby out with the bathwater.
All of technology back to the printing press (and probably before that) could also be said to make it easier for governments to oppress their citizens. Making laws (and enforcing them!) to prevent governments from doing these things feels like that route forward, not trying to stick our heads in the sand.
Perhaps I'm horribly naive, perhaps I just see the SciFi future I've spent my life reading and dreaming about on the horizon and I'm blinded by the reality, perhaps my ideals around "knowledge deserves to be free/accessible" are misguided. I don't know.
I worry that any attempt to limit their use and development will be abused and misdirected. We are already seeing people like Anthropic doing this, they are trying to use anti-AI sentiment to engage in regulatory capture. Go watch Dario’s speeches about how open weight models are dangerous and how they are “not really open”. Everyone can see that much of this “safety” conversation is ultimately just a tactic to shut potential competitors out of the market and establish a monopoly/duopoly.
"Stopping" LLM research just means it will be in the hands of a few who can abuse it. I'd rather a state of M.A.D. but instead of a handful of countries/governments it's millions/billions of people with access to the models (open ideally). Again, perhaps horribly naive or misguided, I understand that bioterrorism could (is?) a real problem as well as more "mundane" things like building a bomb (nuclear or otherwise).
I just feel like limiting access to governments or "blessed" entities is even worse.
If we slow down on ASI voluntarily we’d be allowing a gap to open up that would make the difference between colonial europe and colonized Asia/Africa look trivial. It would be insane.
An easy choice to make if the alternative is everyone dying instead.
This is a settler-colonial mindset that reflects all the bad things we did onto everyone else. Notably, it's a current US ally that is most guilty of this.
without sharing tech to make the ASI, you'd hope humanity could work together to determine how to align an AI for our common benefit.
That's one outcome, certainly, but not the only one nor, I contend, the most likely one.
A most likely outcome of ASI is human extinction, because there's more paths to an ELE outcome for humans from ASI than there is for non-extinction level outcome.
Your outcome is only possible if:
1. ASI is never able to escape the confines it is placed in.
2. ASI is benevolent to humans.
3. ASI decides, in the spirit of its benevolence, that it should restrict its involvement in humans.
If all three of the above conditions are met, then sure, your outcome is possible. If not, humanity as we know it will end.
It is unlikely that those 3 conditions will all hold, though.
If ASI is trying to wipe out all humans, we probably deserved it. Unironically!
Consider this: All that hardware that's going into those datacentres right now? In 5 years or so it'll all be on the secondary market... an influx of cheaper compute like you've never seen.
Certain powerful wealthy people aren't omnipotent, them losing out isn't the only blocker to progress.
As for historic precedents: Human cloning, human genome editing, and mirror life seem like one precedent; nuclear weapons and nuclear energy another; come to think of it I think drone delivery was strangled by regulations too...? Plan A isn't a proposal to never build superintelligence, it's a proposal to build it more cautiously and transparently.
If we had a way to make genetic edits to make humans a lot smarter, a lot stronger and live a lot longer? Or a way to quick-grow human bodies to adulthood in a couple years? Capabilities that private actors or countries may want, ethics be damned? That would be closer to what we have with AI right now.
You should consider reading the wikipedia page about Parkinson’s disease.
Now, we're getting better at predicting polygenic traits, and we're getting beefy multi-edit pipelines that might provide a meaningful advantage over embryo selection working in multiple animals. But as of yet, the advantage of genetic editing in humans over just doing aggressive IVF and dredging the embryos for desirable traits is minor.
Bit of a chicken and egg problem there. Can't advance the tech fast without actively using it, can't actively use the tech until it's advanced enough for the benefits to override ethical concerns. So it's getting there, but at a glacial pace.
Following the invention of recombinant DNA technology, scientists convened the Asilomar Conference in 1975. They established a voluntary self-moratorium on certain types of genetic engineering until strict laboratory containment protocols were created.
In the 1980s, bioethicists, theologians, and researchers established a hard ethical line between somatic editing (treating an existing patient's non-reproductive cells) and germline editing (altering future generations).
No one has performed the latter form of genetic engineering except for Chinese scientist He Jiankui in 2018. In response, Beijing heavily tightened its laws, classifying heritable gene editing as a high-risk medical technology subject to the penal code. He Jiankui was sentenced to three years in prison.
https://en.wikipedia.org/wiki/Sundial_(weapon)
Edit: Mind you, I wonder if the design for Sundial is stored somewhere...
This is an interesting subject and conversation, but it's moot having it in these culture-centric forums. I wonder if there are Russians discussing plausible scenarios in Vkontakte groups, or Chinese doing the same in whatever Alibaba group sites they use.
The problem is that we are all skewed by our media, our ideas and our culture. These type of discussions need the highest kind of political interactions.
It's fascinating, specially for someone who lives in a "third world" country, non-aligned to any of these 3 superpowers. Whatever transpires, we are at tge mercy of these (and no, US hasn't treated us "better").
My opinion is that there's no turning back on AGI development. I dont think current governments are capable of getting into an agreement of that size. Specially given the Isolationist stage in the cycle we live in. (In contrast with for example the CFC and Ozone layer issue we had in the 1990s, when the planet was in a globalist kind of stage)
That is an objective fact and has nothing to do with the rest of the cultural relativism you're engaging in.
Whataboutism about the US _not signing_ things is also irrelevant.
The whole AI treaty discussion is about a treaty countries will _actually sign_ and the relevance of Russia cheating on the BWC is that the signatures of some countries are absolutely worthless, which _is_ relevant.
A resource extraction based economy sees people as slaves. The true source of power is the resource, people are just a means to an end, so you mistreat the people as much as you can get away with in pursuit of the resource while avoiding revolt.
With stable infrastructure, the government makes far more from an educated, rich population that it can tax and use the innovation from. It’s against its own quest for power to interfere too much in the prosperity of its citizens. The incentives are aligned.
Solving the AI problem isn’t about stopping the tech or making a bunch of brittle laws. It’s always been about alignment: aligning the large AGI-like entities that are the modern state, the modern economy, representative democracy, or AGI itself, with human prosperity
Yasha Levine wrote about how this narrative was preceded by a forgotten one where MIT students protested because the computers were going to be linked to government databases and share data on anti-Vietnam war activists. Despite protestations, activists were correct and this happened, and now it happens at huge scale.
http://yashalevine.com/surveillance-valley
thank, mr 习
It's not clear in this context what you actually mean by "government." You are assigning agency to something in a way that seems like a reification. While a bureaucracy can seem to have a life of its own, isn't it generally people who seek power?
... recently, as in the last 10 years?
Studying human bio-diversity since WW2 is the most obvious example, though it hasn't been entirely successful.
Genomics is what finally broke the barrier, especially in the last decade or so.
Intrinsically, the knowledge humans choose not to pursue will not be much publicized. There's limited value in calling attention to it and it doesn't make for good entertainment. Plenty of examples provided by other comments nonetheless.
> Perhaps I'm horribly naive, perhaps I just see the SciFi future I've spent my life reading and dreaming about on the horizon and I'm blinded by the reality, perhaps my ideals around "knowledge deserves to be free/accessible" are misguided. I don't know.
I don't personally think there's intrinsic benefit in disseminating arbitrary knowledge. There's quite some difference between the printing press and nukes.
> Are there examples of where we have collective decided not to pursue knowledge? Successfully?
human GMO, some bioweopns, I'm sure theres a long list of awful stuff no one wants to exist.
AGI doesn't do away with nuclear MAD, it just messes with economics and makes many people temporarily jobless. Temporarily because in a literal sense RLVR needs verification to train off of, and a lot of jobs cant be easily checked if theyre done. this includes AI safety people, preschool teachers, psychologists, and probably a lot more, including most of their bosses
I think that if all we get is skill cloning for formally verifiable tasks, that is not "AGI" and does not doom us. It probably won't even doom software engineers, because their whole job is also not formally verifiable. I don't trust that the next ten trillion dollars of AI research won't give us data efficient continual learning and agency on par with humans, which is what I would call AGI.
I predict that what we consider "super intelligence" is just sheer computational power, but any potential of a very capable agent is bounded by the needs/wants of the person wielding it. That is: even if we were to hand, say, Elon musk this "super intelligence", most humans would consider it relatively stupid because the person wielding it is still a person with stupid goals and values.
Or, to put it another way, I suspect we already do have a superintelligence and have longer than any of us have been alive, and it's just "the market", and it is still incapable of overcoming the limitations of a few morons wielding immense power.... power they will never yield to some intelligence with values and goals "more intelligent" than their own (if such a concept is even meaningful), and intelligence wasted on the values and goals they do have.
- if you have a system that is large enough to store, lets say 10^12 AND gates (all frontier llms can do this) - and this system can produce outputs based on previous things it has outputed
its turning complete, and RLVR on it is optimization over the space of algorithms. If an algorithm exists to do a task, and the task can be verifiably done, this finds the algorithm to solve the task most often.
it is obvious that this scales, from much-worse-than-human to slightly-worse-than-human, therefore it 100% can exceed humans.
We were discussing AI in the 90s and it's been discussed before that.
The answer was always the same; hardware can't hang.
Now it can and will get even better.
The SaaS era fueled by ZIRP and ignorant Congress was a fluke that from an engineering perspective didn't produce anything but hype and same old
The generation enriched and empowered by it is just as temporary as Boomers. Little point in enabling their appeals at the expense of scientific progress that helps all of humanity.
China won't. Russia won't.
It's ridiculous to me the level QQing coming from Americans exploiting child sweatshop labor so they are free to ignore their own biological needs and keep a "knowledge work" job (talk about first world privilege) handing them wealth to go tour the poor villages they exploit.
Those workers never had a choice between college or the mines. So sorry 300 million Americans in a world of 8 billion.
We don't even want these jobs given how much bitching I have listened to the last 10-15. IMO the job creators and Congress saw how Millennials liked to be on the computer and went way too far into enabling such banal output.
Make healthcare and housing the economic tentpole. Both still need jobs and technology. But at least the outcome isn't a generational Ponzi scheme engineered by Boomers to enrich them and then let it all collapse when the majority realize those stocks were never real.
> Scholars have interpreted Cassius Dio's wording to indicate that the fire did not actually destroy the entire Library itself, but rather one or more Library warehouses near the docks.[87][81][8][89] Whatever damage Caesar's fire may have caused, evidently the Library was not completely destroyed.[87][81][8][89][3] The geographer Strabo (c. 63 BC – c. 24 AD) mentions visiting the Mouseion, the larger research institution to which the Library was attached, in around 20 BC, several decades after Caesar's fire, indicating that it either survived the fire or was rebuilt soon afterwards.[87][8] Nonetheless, Strabo's manner of talking about the Mouseion shows that it was nowhere near as prestigious as it had been a few centuries prior. It is unknown whether this was due to historical decline or catastrophic destruction.[8] Despite mentioning the Mouseion, Strabo does not mention the Library separately, perhaps indicating that it had been so drastically reduced in stature and significance that Strabo felt it did not warrant separate mention.[8] It is unclear what happened to the Mouseion after Strabo's mention of it.[60]
> Further evidence for the Library's survival after 48 BC comes from the fact that the most notable producer of composite commentaries during the late first century BC and early first century AD was a scholar who worked in Alexandria named Didymus Chalcenterus, whose epithet Χαλκέντερος (Chalkénteros) means "bronze guts".[90][87] Didymus is said to have produced somewhere between 3,500 and 4,000 books, making him the most prolific known writer in all of antiquity.[90][82] He was also given the nickname βιβλιολάθης (Biblioláthēs), meaning "book-forgetter" because it was said that even he could not remember all the books he had written.[90][91] Parts of some of Didymus' commentaries have been preserved in the forms of later extracts and these remains are modern scholars' most important sources of information about the critical works of the earlier scholars at the Library of Alexandria.[90] Lionel Casson states that Didymus' prodigious output "would have been impossible without at least a good part of the resources of the library at his disposal".[87]
Part of it did!
> Was it not the most extensive library of human writing at that time?
No.
More importantly, the original assertion:
> the world certainly turned its back on the only collection of written knowledge and let it turn to ash
is definitely not true.
On a funny note, I think their prompt was:
"Hey Fable. Please attribute every piece of scientific and economic progress to AI until 2040. And predict every major geopolitical event. Make no mistakes."
Isn’t that like all of the Middle Ages where we replaced knowledge with an alternate religious reality.
For start, previous era was also deeply religious. So it switched religions, if anything new one was more friendly toward knowledge.
My impression from the origin of the bioweapons convention is that collectively people decided that these things are too dangerous in various ways for any advantage that might be derived from them.
I'd bet that in most places 9 years is about the time needed to build a residential building. I think a good way to think about this is to think of this as producing a serial car. From pitching and capital acquisition to building a prototype to software, regulatory and then the final product which needs multiple factories and supply chains. Yes, of course robots sound cooler and there are compounding effects yada yada, but on the other side there are as many obstacles as things that accelerate this product (like capital acquisition and fearmongering of gov to bend regulatory stuff faster).
On his blog he says: "I did a lot of writing for AI 2027 and was listed as a co-author. Some of my writing made it into Plan A too, but it was a bit less. The difference is of degree rather than kind, but because of this - and to give me more latitude to discuss it the way I like with less PR blowback - we decided not to put me as a co-author this time. I continue to be proud of having a part in this, small as it may be. (related: everything in this post is my opinion only, and not officially endorsed by the AI Futures Project)"
LLMS are 4 years old and the companies that sell them 10x every year. What evidence can you cite? Could you convince a disinterested 3rd party you have anything other than cope? What facts about the world make you think this is anything other than the new (and probably temporary) normal?
There is plenty falsifiable in this in ai-2027.com, and they have not gotten everything right. But some things they have: for example, the Pentagon has already invoked export controls to restrict the deployment of a frontier model. This level of government oversight wasn't predicted until 2027 in the original scenario.
They are buying up all the RAM today. Do you think "this is fine because in 5 years post-crash I can buy some cheap RAM"? If everyone with money is betting differently, do you have some information they don't, or is the whole economy just slipping away from you?
You experience luxuries today, that no king 1000 years ago could afford. Instant access to communication, food, medicine for the right price of course.
The consumer economy was great while it lasted but it's over now. We have machines that do useful mechanical work (engines) and useful intellectual work (llm-computers). Capital will move productive work from people to machines(if we let them), and the only jobs left will be delivery driver and warehouse, and then those will be gone too.
Human population was exponential and now its flat, but that's a function of what exacly? It could go back down to 1 billion or less. When jobs demanded a person supply was ready to match it. When jobs dont demand a person? Go to a degrowth rally take the temperature (and average age and child-per-person ratio) to get a taste of the future shape of supply and demand in a pessimistic world of sentences that don't have subjects just vague plattitudes. Are they net shutting down grade schools or building them in your neck of the woods?
> * they net shutting down grade schools or building them in your neck of the woods?*
My area (rural Iowa) has had several new schools built in the last 10 years. Net gain for sure.
We should have been under water, hunted by AI, overpopulated, killed by terrorist, smitten by god for our sins and so on. Luckily all it took was our privacy and a lot of tax money to survive.
strongly, no. its just hard to distinguish them. for example, radioactive decay. cmon
you could make the argument for human total GDP, which looks like https://ourworldindata.org/grapher/global-gdp-over-the-long-..., but then again, you could say we just haven't reached the sigmoid in it. I personally doubt we will
Even then, you can just compare the progress in open models. Leaps and bounds from where they were 6 months ago.
Now we take for granted that the latest models can juggle between multiple browser tabs, applications, databases, simulators, docker etc to write, execute, e2e test and deploy full-stack applications over hours managing up to dozens of subagents, relatively untouched, without taking down prod even 1% of the time
The models now reliably execute more than a human can fit into their own context. It's magic
Once we have something that experiences a desktop interface more like a human does, an entire swathe of tooling that has heretofore been nigh-impossible to automate moves into the fold, and that'll be another explosion of folks finally getting to join the agentic workflow world on their industry specific apps...
Literally every major company that has embraced AI coding has suffered devastating downtime this year as a direct result of AI induced failures.
They've also been slower at adopting just-released tools that startups are using.
The only companies sticking with AI at this point are the major players who have chasing their masssive overinvestments to the bitter end, crappy coding shops, and for some reason Starbucks and they'll all suffering badly in a year their customers begin demanding massive amounts of human coding to repair all the issues with the AI code.
The popular thing is now to setup loops (eg I setup hourly integrations for Claude/Codex to 1) scrape my Linear, claim achievable tasks, and push PRs or 2) do root cause analysis on customer issues that evaded automated filters, to name a few)
Though for me, my setup still feels mundane. I have AGENTS.md, CLAUDE.md etc and a few skill files. These are purposefully light - tons of examples online you can pull from online. Mine are fairly personal to my setup and products.
Importantly, I also allow Claude and Codex to bypass permissions. Yes, there is a risk they wipe my machine. The productivity upside has been worth it, for me (haven't been burned yet, ~9+ months into running models this way, I have backups, use cloud etc).
As far as maintaining quality, one of the most helpful guardrails over the past year, for me, has been requiring my agents to pipe their changes to local reviewers through OpenCode, Cursor, etc agents to have a council of models with different biases reviewing the changes, and autonomously working towards a completed objective. No matter how good Claude or Codex gets, for example, I will probably always want a different model checking its work. Like GLM, (now with 4.5) Grok, Composer.
Several OpenAI, Anthropic, xAI employees, and popular AI engineers post on X and share helpful tips & updates. Highly recommend for keeping a pulse on startups and AI. I haven't found something close, honestly, other than when I spend time in SF talking to people.
> AI labs may also be able to reap tremendous benefit from these inference-scaled models by using them as part of the training process. If so, the large scale-up of compute resources could go into post-training rather than deployment. This would have very different implications for AI governance.
> ...
> So iterated distillation and amplification provides a plausible pathway for scaling inference-during-training to rapidly create much more powerful AI systems. Arguably this would constitute a form of ‘recursive self-improvement’ where AI systems are applied to the task of improving their own capabilities, leading to a rapid escalation.
So "inference scaling is required to scale capabilities" doesn't mean that we're reaching the top of the S-curve in intelligence. If anything, it could mean a shorter timeline and more unpredictable landscape for governance (e.g. due to securing weights no longer as effectively preventing escalation, more in the article).
On its own it wouldn't. But that article came before the later article https://www.tobyord.com/writing/hourly-costs-for-ai-agents which adds the claim that inference (along with everything else being employed at present) is scaling poorly with increasing task lengths. Now maybe the December 2025 claim is wrong, or maybe things will change soon, but the February 2025 article surely doesn't establish either of those.
It feels like the cognitive gaps on current LLMs are indeed structural, but also that if we solve that structural issue with a new or extended transformer type of architecture, we’ll be looking at a whole new ballgame.
If you don't care about getting the drone back, it does simplify the problem somewhat.
The theme of the scientific findings is that while humans excel with none of our physical sensors, we do very well across the board in making use of them thanks to our relatively huge brains.
And fantastical amounts of compute power is exactly what are handing over to AI. The fact that their training data isn't perfect may matter less.
But during peacetime, you don't make money running a delivery service that way, so it's not going to replace those jobs.
Maybe they will soon but it’s massively far behind the kind of timeframe AI 2027 would have suggested.
This past 4th of July weekend a Waymo ran over an actively burning firework in a low speed scenario, the kind of thing a 95 year old driver would have avoided.
250 years of constant automation has never produced large scale unemployment, despite obsoleting everyone's jobs several times over.
“I’ve been pulling my sled across this lake for 50 winters even when the temperature went above freezing. Never fell through!”
In the industrialized world, old professions disappear all the time, and are replaced with new ones. I saw somewhere that it's about 2% per year.
And similar things can be said about many technologies in recent history – cars replacing the horse, first flight to man on the moon, even the creation of early internet to its mass adoption.
You're talking generally a decade or 2 for society to completely change from the rapid advancement of a new technology.
I'm not saying I agree with the 2035 prediction, but it doesn't seem impossible to me, if AI can help us improve the pace that we're already developing disruptive robotics.
In 2010 the idea of self-driving cars and autonomous delivery drones seemed very sci-fi and a long way out. But today, just 15 years on, these things are increasingly starting to be rolled out.
If they dropped that 95% number to 50-60%, I think I'd probably lean towards agreeing. Not because it makes sense in my gut, but because the logical part of my brain knows exponential trends (if one exists) do things that we wouldn't instinctively predict. But even if you assume exponentials 95% does seem very high.
For a specific technology to become viable, you often need progress in several individual technologies to occur. Some of these will be exponential and surprise us, but many will see slower progress.
Either way, I suspect progress in robotics will be slower than AI, but I also think we'll see a lot of investment in robotics over the next decade given that AI opens up a lot of new potential applications for robotics.
I suspect any task that doesn't require extreme battery life or extreme dexterity will be doable by robots much sooner than most people think today. I think we could have decent-ish humanoid robots within a decade – robots that can prepare food, clean hotel rooms, empty bins, etc. I think Elon's bet on Optimus is quite likely to be viewed as very ahead of the curve with time.
However, it will take much longer for robots to do everything a human body can, but then most jobs don't require we push the physical limits of human body.
That's why I think 60% is reasonable. And even then I'm not saying it will happen, I just don't think that's a bad bet if given even odds.
My guess is that the deployment of other types of robots will be a similarly slow grind.
That's unlike the Internet, smart phones, and coding agents, which got user adoption at a much quicker pace.
If they own all the RAM, models, and the means to do any work, then you are at their mercy. They will buy all the RAM, leaving you none, then all the transport, then all the electricity. You will be as boxed out of the current economy as the Amish and it will get its plug pulled.
Gradual Disempowerment is the default plan right now. War and tyrrany are not remotely the worst case scenario. I'll take Butlerian Jihad over being turned into cattle any day.
Imagine solving for equilibrium with two classes of beings. One requires agricultural land and 20 years to become individually productive and barely maintains a healthy population in a entertainment saturated landscape. The second eats only electricity and is ready to work on day 1. Round 1 goes to the strongest gorilla for sure, round 100?
If LLMs had come to earth in spaceships would you have welcomed them into your work and your home?
We already have at least 5 companies only in the US. Your whole premise is false.
elaborate. regulation -> war, how?
I am not sure where they believe that amount of capital could come from. It would require central bank level money printing never seen before.
https://ai-2040.com/supplements/compute-supplement
If we're producing 10x as much, why not print 10x as much money? The goods and services each dollar could buy would remain similar.
Plan C:
> "... fewer and fewer humans are needed to conduct AI R&D, meaning that covert projects are easier and easier to pull off without detection."
Plan A:
> "... training AIs requires large numbers of AI chips. Most AI chips are in giant datacenters.50 AI datacenters are typically big enough to be visible from space, and power-hungry enough to require conspicuous infrastructure. New AI chips can only be manufactured at a handful of fabrication plants (fabs), located mostly in Taiwan, South Korea, the US, and China. The US and China negotiate with the countries that have a major role in the chip supply chain, and they require each major datacenter owner (and their upstream suppliers, including chip fabs) to publicly declare their major purchases and sales."
Plan A requires properties of AI training that Plan C requires do not exist.
If AI production is limited to big labs and big data centers then it is de facto contained and monitored. If you know where all the ASML machines are then you know the reproduction rates of chips. If no one can buy or build the machines required to concentrate uranium or plutonium to critical levels then the threat is contained and monitored.
You can dig up all the Uraninite you want. It was never much of a secret that uranium had dark applications. The machines and processes where thankfully big and expensive enough that only the most focused bad actors could aquire them and then hold the world hostage to the degree they do. If al-qaeda or isis could have used $40 bombs from home depot instead of expensive planes they would have (and they do).
You have to legislate and control the big, expensive, and slow things. Dynamite and phentanyl are so dangerous because they move much more easily. Freedom does not have to be a suicide pact. If the inconvenience of requiring prescriptions or access to dynamite reduces harm then it is net positive?
My early analysis of the analysis:
https://lifearchitect.substack.com/p/the-memo-special-editio...
Bravo, and I hope it has the impact on the AI safety field it deserves to have.
Look, I am scared of where we are heading, but I cannot see how we can change the dilemma towards mutual cooperation unless, as humans tend to do, only react massively after something really bad happens.
consider SALT and other treaties from USSR vs USA cold war. they checked
Always easier to boost something already existing on social media than manufacture it themselves, then wildly blow it out of proportion to make it seem urgent and important.
But unlike some of the others, I’m hearing anti-AI sentiment from a wide range of people who don’t even use social media.
But I doubt the influence. I’ve been free for years, and I can always spot who’s spends a lot of time in TikTok/Twitter/Instagram—it’s like talking to someone from another planet. It mostly sounds weird and sad, more apt to annoy and alienate their friends than inform or influence them.
While I can admit some of the anti-data center arguments are overblown, many are more than valid in my opinion. Data centers are fundamentally extractive technologies. They are enormous, windowless boxes that take resources from one location to make someone else in a far off location enormously rich and powerful, with extremely few benefits to the local community.
Plus, as another commenter mentioned, it's not exactly like the Chinese and Russians have been fanning the flames that AI is going to take all of our jobs - it's the leaders of the frontier AI companies in the US saying that. Remind me again why I think putting up a giant data center in my state, that was proposed to use more electricity than my state already currently uses, is a good thing for the average joe where I live??
Honestly, I feel like many commenters here are in their own bubble and don't understand how much AI and tech generally is widely viewed as a net negative for society by huge swaths of the the population, and I don't really think it's an unwarranted perception.
These tech companies are already investing heavily into solar, natural gas, and nuclear https://techcrunch.com/2025/03/30/data-centers-love-solar-he... this would be normal stuff in China where they spend the last decade investing heavily in solar and are bringing something like 60 nuclear reactors online.
These datacenters aren't particularly consumptive of water compared to most other industries in that regard and we've already seen states enforce rules against Meta who immediately paused their datacenter when water issues were detected (following mandatory monitoring).
Chip production is lagging but most projections I've seen is it will normalize in about 5yrs. Not to mention there will be further demand for robotics and self driving cars, so ramping up chips should be a normal thing like ramping up green/nuclear energy. Delaying it won't solve any current issues.
Its boogeyman thinking.