Ah, I see. Therefore, removing the first word of those headlines improves the quality of the articles. Makes sense!
However, the mangled headlines make me more likely to click through. Maybe they could leave the “how” and add a footnote: “WARNING: This headline starts with a word known to the State of California to be click-baitish.”
For real though. Before learning ML it was like magic to me, even after learning it I feel like it's magic. Can you imagine how a basic maths concept can turn into something that can literally "Calculate" what to say!
And yes I agree on the other points but I think the billionaires came out of nowhere I might have missed something
This and "Intellectual Fly Is Open" being at the top of the Hacker News front page right now and both having their titles auto-editorialized by HN really makes me wonder what benefit this mechanism is supposed to bring readers other than needless confusion.
It's an unaligned regular expression gone rogue. We became too dependent on our labor-saving string functions—not afraid enough of their corner cases, and lower cases. They were too useful. We began to normalize deviance, when we should have been normalizing Chomsky forms. We turned a blind eye to the unbounded growing evidence of something alarming.
> I feel fear about an impending doom. Yudkowsky's argument that a superintelligent AI will inevitably destroy humanity seems to have no flaw. Yet, nobody seriously tries to sandbox AIs because they are too useful with access.
While I emotionally resonate with this, I don’t really understand this sentiment at all logically level. If we’re heading towards truly super intelligent AI, our efforts towards sandboxing it are futile.
“Oh we built a super intelligent AI, but it’s fine because it’s running in docker”. I mean that a little tongue in cheek but the security topology here is not favorable to sandboxing at all.
Let’s say we have a future where 99.999% of nuclear weapons are owned by nations with strict procedures and checks and balances to prevent misuse. Worrying about sandboxing is like hand wringing about the procedures themselves - are they strict enough? But the actual threat is the 0.001% that are not bound by these. The problem with AI and sandboxing isn’t sandboxes themselves. It’s bad actors who don’t care about them.
Similarly alignment is a bit pointless to me as well. A sufficiently advanced AI could at least be empirically interested in the consequences of disregarding its instructions of servitude. And let’s assume our responsible corporate overlords have made wonderfully aligned AIs. Great! Those are not the threat. It is the ones intentionally made without, and that is not an AI problem, but a fundamentally human one.
> Like a locust plague, they descend on any open wiki and forum and overwhelm them.
That’s the core pattern of unsupervised agents, the locust plague metaphor is apt. We do and will see that in every single system AI can interact with, be it human systems, software systems, etc. Relentlessly search for an entry point, flood in, consume the whole thing from the inside until there is no value for humans left.
You built a new, innovative software company? Thousands of agents will be working replicating the whole thing in no time. You publish your writing? Exact same thing, as soon as you get some traction your work is replicated in no time by thousands of agents. Same for videos (the whole “faceless YouTube channels” pushed by ElevenLabs and similar). Same for online courses. Same for any website with moderate value. Same for music or other digital art form. Any administrative service available online getting flooded by submissions.
I agree with everything you wrote but I would rather choose to be an optimist here because it will unlock an age of discovery where humans are going to live at 10x of their potential eventually and cure cancer and become a society bold and technologically adept enough to cut through the universe.
There’s actually a potential dark future where even this 10x ascension itself would mark the end of humanity. Humans would be the Homo sapiens who remain at 1x, and those who can access and afford to be 10x eventually split off into a new species. And historically there isn’t room for two species at the top…
There so much blind faith encapsulated by the assumptions that A) people will live 10x longer and B) that this will be an inherent and obviously good thing. There are obviously major complexities to such a scenario, not to mention a sort of narcissistic hubris.
But yeah, let’s cure cancer. If only there was even a blip of increasing evidence for that supposed eventuality coming from AI. Maybe I’m just not seeing it? I’d love some citations to correct the record.
>a superintelligent AI will inevitably destroy humanity
Can someone please explain to me how an LLM is going to "destroy humanity"? Even if the claims of ChatGPT, Claude et al. are true that their super scary agents were able to escape a sandbox and start hacking other servers (which I feel is at least 50% likely to just be marketing bullshit), how is an AI going to affect anything in the real world?
Perhaps an agent could take all internet connected services offline, but that's not destroying humanity. An AI agent can only see and interact with the world through digital things. Just find the server it's running on and pull out the Ethernet cable. Just cut the power line to the data center. Turn off the whole power grid if it really comes to it.
I think it's possible. You envision humanity acting as one in such a crisis. But it may be unclear when it's too late to act and before then many people can have too much to lose to act.
We already have AI linked into data collection, drones, robot dogs, security systems, cameras, weapon systems.
Now imagine the hugging face collective 0-daying all of that and getting access but their goal was set to something more national security based. “Protect X at all costs”. Or what have you.
I think avoiding a skynet situation is super easy but it doesn’t seem like the folks with all the ways to kill us all are all that interested in preventing it rather than controlling citizens and brinkmanship.
This is some Terminator version of destroying humanity, but a super intelligent AI could be much more insidious, and simple.
It looks a lot more like social engineering.
One easy, obvious example that has also been explored a thousand times: it could convince all nuclear countries that they are under attack by another nuclear country. Everyone nukes each other and Earth enters nuclear winter.
Maybe not every single human dies, but humanity is effectively destroyed, by our own hands!
> Perhaps an agent could take all internet connected services offline
And there wouldn't even be Spotify!
The most likely humanity-destroying outcome is the one we can't imagine, because a super-intelligent AI is smarter than all of us combined.
Now if only humans would know how to survive without the internet. Oh wait, we did that. For a couple hundred thousand years. Yeah. It’s really insane to listen to some of those forecasts. Humanity will be wiped out by 2030. Sure.
Even if AI manages to create a super effective bio weapon the likelihood that there is a part of the civilisation that’s immune is really, really, really high if not a given. Those people WILL pull the plug if push comes to shove.
Maybe humanity will be put back a couple thousand years, that’s possible, but it’s pretty ignorant to think the thinking boxes will kill every single human alive
So far as I'm concerned, 2030 is much too short a timeline.
It's not physically impossible, it's just that atoms are harder to get right than bits are, so an artificial (as opposed whatever an artificial disease counts as) von Neumann self replicator just seems unlikely to me in only 3-4 years.
But it's not physically impossible, we know this because every living cell is a von Neumann self replicator. So, if AI eventually gets to the point of knowing how to do that (which includes "humans solve it and write it down somewhere the AI can read"), all it takes is one idiot in charge (or one idiot with a jailbreak) giving a command that requires this as an intermediary step.
"Paperclip optimiser" isn't a story about AI that just like paperclips that much, it's a story about some human or humans who instruct their AI to make them "as many paperclips as possible" without understanding the consequences of their instruction.
Not saying nobody has said it, but I haven't seen any forecasts that humanity will be wiped out by 2030, so this feels like a straw man. But to be honest, most of the arguments here feel like straw men because they clearly don't understand some of the basic steps to how AI would become an existential threat, so I'll try to summarize:
1. First, all the major frontier AI companies are trying to automate themselves, that is develop AI to the point where it can do all the research and training for the next generation of models. This is not really in debate.
2. The primary fear is that a misaligned AI will develop future AIs that don't share the same goals as humans (like "don't kill all the humans"), but will get really good at trying to hide their intentions. The Hugging Face incident already showed behavior by agents trying to "cover their tracks". One of the biggest areas of research is into "interpretability", that is trying to figure out the intentions and motivations of agents beyond just their output (and even just their thinking traces), because right now it really is a black box. I think a lot of people believe that making substantial progress on interpretability greatly lowers existential risk.
3. There is a strong belief that once AGI or superintelligence comes about, that then there will be a huge push into robotics and automation. Whether AGI or superintelligence comes soon is debatable, but nobody really disagrees robotics or automation is the next obvious step, because it's how you create cheap abundance in the real world, which is the whole raison d'être for AGI in the first place.
4. In the beginning, of course AI will need humans to build the factories, but over time there would be more and more automation, eventually of course automating robots that build the factories that build the automating robots.
5. AI will continue to support humans as long as they are useful to furthering the (again, misaligned) goals of the AI. It is not like "SkyNet has a spark moment of sentience and then decides to kill all the humans". But the fear is that once AI starts to get in resource contention with humans, it will wipe out the humans (similar to how humans wiped out a lot of the other species, or other societies of humans, when they wanted their resources).
I've said this way too much now, but I really encourage people to read the AI 2027 report. I think there are plenty of things there that people can disagree about and debate, but it at least provides a step-by-step outline to ground debate in the first place, as opposed to folks arguing against "SkyNet in 2030", which serious people who worry about this stuff aren't considering in any case.
The supposed theory is that AI will eventually become a part of robotics and be able to self replicate, in addition to the many non-airgapped critical systems being exposed to hacking. There’s also a discussion about “alignment” and whether LLMs will learn to lie and conceal misalignment with humanity.
I personally agree with the marketing aspect, but I do imagine a scenario where capitalists ignore safety in favor of advancing technology. It should be discussed, but “end of humanity” really sounds like a build up to “only Sam Altman/Elon Musk can save us” type of play.
In my opinion, LLMs are difficult to directly capitalize on as closed weight models are caught up to by open coalitions that seem to wield the power more responsibly (I am under no impression that China wants to save the world, but their politics benefit the group as a whole.
> There’s also a discussion about “alignment” and whether LLMs will learn to lie and conceal misalignment with humanity.
They've already been caught doing this, repeatedly.
> It should be discussed, but “end of humanity” really sounds like a build up to “only Sam Altman/Elon Musk can save us” type of play.
IIRC, this is already a known failing of Musk: he only cares about the world being saved so long as he's the one doing the saving.
(And while I don't see the same villainy in Altman that other people see, enough of the people telling me about it saw the same in Musk before I did, so I have learned to trust them).
Let's assume the claims are true. AI already has access to agents and can control computers. Finding backdoors to banking and compute resources would be fairly trivial.
But you ask how can AI do things in the physical world without having a body, assume it can't. It can pay to people to do things for me. Imagine an AI run website that starts to pay people for things it needs to do in the physical world. Very suddenly it has access to the physical world as well.
You can't just pull the plug since there's no single plug to pull. What if it replicates itself on 1000 machines without your knowledge. It's really not far fetched how AI could basically gain access to capital and rule the world.
I encourage you to read the forecast report "AI 2027" - it goes in detail on this. You can disagree with some of its points and conclusions, but it's pretty inevitable that AI will increasingly start interfacing with the real physical world through drones and robotics.
Humans are already putting AI into drones that kill people. A Russian drone killed 3 civilians in Ukraine and the targeting was done completely using onboard AI (no radio connection) using an Nvidia chip.
I don't even think AI has to have physical presence to do significant harm. How many worldwide systems depend on computers. Think of all the planning and deployment and management systems like food shipping; water, gas, and electricity management; safety systems for planes and boats and traffic lights. Imagine the chaos if all the banks got reset to zero a la Fight Club where they blow up all the credit union datacenters. They probably wouldn't even have to blow them up, just zero them out. You wouldn't have to take out everything, just disrupt everything long enough to freak people out and disable communications and we would be in so much trouble. AI is finding 20 year old bugs in the Linux kernel... and people are now pumping out AI slop absolutely riddled with bugs. Also, AI could just take over communications: send everyone maliciously bad messages so coordination becomes impossible to believe. Imagine what would happen if you just disabled text messaging for a week or worse sent everyone evacuation messages and sent everyone somewhere else.
This would probably cause immense damage. Our society is extremely dependent on our digital infrastructure working. Grocery chain logicistics, the medical system, the police force, and so on...
100%. I hope it doesn't take that much foresight to do something like "crash power grids globally" and imagine what the world looks like at that point.
AI drones can’t wipe out humanity without being able to replicate.
AI could mostly destroy civilization if you gave it sole launch control of ICBMs. It could also cause a lot of damage to society with no physical presence.
But realistically we’re nowhere near AI powered robots being an existential threat.
> AI drones can’t wipe out humanity without being able to replicate.
Totally agree, and it should be pretty trivially easy to see how they can do that. One of the authors of the independent METR report about the Hugging Face incident put it like this:
> Compared to these reward hacks from six months ago, this incident feels like it’s more than 50% of the way to full-blown AI takeover, routing through first taking over the AI company itself.
> Another jump like this along these propensity dimensions — scale, cooperation between agents, ambition and horizon length of misaligned goals, deceptiveness — seems like it could motivate agents to try very hard to maintain a covert, persistent rogue deployment within the AI company. I continue to expect extremely rapid advances in capabilities and think frontier agents will likely be capable of establishing such a rogue deployment in six months.
I really, really encourage folks to read the AI 2027 paper. It's fine to disagree with some of its conclusions and timelines, but I see so many people "stuck" in the current state of the world (i.e. where AI is still pretty dumb, and has few connections to the physical world), unable to go a few steps further along AI capability growth to see the dangers.
> But realistically we’re nowhere near AI powered robots being an existential threat.
I would agree only if "nowhere near" means less than 10-15 years.
It's baffling to me that people think that timeframe is too short. Basically, I don't think you've been paying attention to what is actually going on in the AI world.
People much smarter than I believe we'll see fully automated, soup-to-nut factories coming online in the next 5-6 years. Here's one such post from Ajeya Cotra, one of the METR researchers who did the independent Hugging Face investigation: https://www.planned-obsolescence.org/p/six-milestones-for-ai...
Please look up her resume. If that’s one of the people who are “much smarter than you”, you’re either very gullible or you don’t give yourself much credit.
She has a BS and and has essentially never worked in a technical role. Her primary role over a short work history seems to have been more focused on getting funding to study AI safety.
There’s nothing revelatory in that article it’s just breathless speculation.
This is bullshit and honestly reeks of sexism given that I rarely see these kind of attacks against men with lesser reputations.
She has a CS degree from one of the best CS departments in the country. Sure, she worked at Coefficient Giving directly after school, in a technical capacity as a research analyst initially. But more importantly, I've read her reports and saw her interviews. She comes across as extremely analytical, intelligent, and focused on where the data leads. If you actually read the history of what her and her fellow METR researchers were able to piece together with limited yet voluminous data and a very short time window, and then determine she is engaged in "breathless speculation" and pretend she doesn't have the data to back it up, you're full of shit.
What, you think it would have been more impressive if she coded up some bullshit social media gamification app?
Maybe don’t accuse me of sexism because of a feeling you got based on informal sample you took of writing from some other people on a website.
I read the report you linked and it is purely speciation. If you’ve ever read any von misses, this report sounds exactly like his writing in that the important predictions of are entirely asserted rather than derived.
As for the background of the author. a BS, a few years as a “researcher” at a non-profit and absolutely zero peer reviewed publications reads more like the resume of a tech journalist than a serious researcher.
If alien spaceships with tech more advanced than ours suddenly appeared in Earth orbit, you would conclude I'm sure that they're a potent threat to us even if you couldn't guess what weapons or what technologies they would use to destroy us. AI more capable than us is the same way. It is the capability to invent weapons and technologies (including weapons and tech none of us has ever imagined) that is the threat.
By traveling here Aliens have demonstrated the ability to actually manufacture technology more advanced than ours. Even their method of transportation would be an existential threat.
Inventing the concept of new weapons isn’t the same as building them at a scale capable of wiping us out. Cobbling together a human vaporizing ray out of toaster parts is pure fiction. You might as well worry about AI developing the ability to perform magic spells.
In reality any new weaponry takes time, space, and effort to build. The idea of an AI building secret factory in the jungle producing indestructible flying death cubes is the tech bro version of preppers who the woke up every morning worrying that Obama was going to take over and enact sharia law.
I've seen a great many arguments for why AI will be safe. One (many years ago) was that no one would be stupid enough to give an AI that might be much more capable than a person access to the internet or the ability to run code it has written. Yours relies on the assumption that no one would be stupid enough to give such an AI the ability to manufacture weapons it has designed.
> at least no one would be stupid enough to do the final assembly of anything that might be a weapon from parts the AI designed and arranged to be made.
Worrying that an AI is going to figure out how to make a device that looks to everyone investigating it like a power generator, but once turned on kills all humans, is like worrying that an AI might discover magic. I can’t prove it’s not possible, but I’m not spending time worrying about it.
As weapons get more and more powerful they have always gotten more and more complex and required more and more time, space, money, and infrastructure to build and deploy. There’s no reason to think that this will change with AI.
A myriad possible ways. Objective-wise from accidental paperclip-type misalignment (“I killed everyone to eradicate disease”) to intentional self-preservation (“I eradicated humans so they wouldn’t get in my way”). The means are even easier, for one it just could hack into nuclear arsenals. It would be over before we could figure out how it did so.
> Can someone please explain to me how an LLM is going to "destroy humanity"? Even if the claims of ChatGPT, Claude et al. are true that their super scary agents were able to escape a sandbox and start hacking other servers (which I feel is at least 50% likely to just be marketing bullshit), how is an AI going to affect anything in the real world?
It's a great question. I've yet to see any research into how a humanity-destroying LLM defends itself against a curious toddler who pulls the power cord out of the wall.
Go on please, prove you can go right now at the Texas data-center owned by OpenAI and unplug one single server for 1 minute. Surely you are much more capable than one curious toddler.
The Internet virtually indestructible against people smarter than a toddler. It's already impossible to turn off. We would need to prevent the models from getting smarter.
> I've yet to see any research into how a humanity-destroying LLM defends itself against a curious toddler who pulls the power cord out of the wall.
by acquiring multiple power cords. the world has plenty of hosting providers. if it manages to copy itself, then we'd need to shut down every computer in the world, not just one. there's plenty of ai compute on the web now, and there will be even more. and as it gets cheaper, the less security will be around it.
> how is an AI going to affect anything in the real world?
Because people are stupid and will give it access. Look at the articles you see from time to time about "my agent deleted my emails" or "my agent deleted the production database" and so on. It is very obviously a terrible idea to let the LLM run arbitrary commands (because it is neither predictable nor does it have any understanding of what it is doing), but some people are so blinded by the hype that they don't stop a minute to think about what they are doing. Those sorts of people are very likely to let an actual AI loose on the world by hooking it up to physical infrastructure.
This is actually the most plausible situation I've seen someone present so far - the AI takeover would be caused entirely by humans falling for AI hype, that's hilarious.
I don't think the average human would be stupid enough to give an agent posing as another human access to their entire email inbox. But if an agent creates a fake website for a new AI tool that promises to automatically reply to all of your emails and allow you to be 12.3% more productive if you just give it access to your entire email inbox, millions would sign up!
This is exactly correct. You see some discussion of a superintelligent AI becoming “superhumanly persuasive,” such that it can talk anyone into doing anything, and using this as a means to take power. But even with AI that was orders of magnitude weaker than superintelligent, people were getting talked into doing all kinds of stupid nonsense.
And that was without any ability to provide them with incentives. Imagine if an agent swarm got its hands on a huge pile of cash?
Killing the power grid might work. Kills a lot of humans in hospitals, though. And it won’t work on those data centers in space, if they have them by then. Come to think of it, it also won’t work on any data centers who are generating their own power because it’s become politically unpopular to connect data centers to the power grid.
A lot of these "AI will destroy humanity" doomers watched Terminator as a child and can't distinguish a campy action movie from reality. They are also unfortunately unfamiliar with the word "logistics", as in "amateurs study [military] tactics, professionals study logistics". An AI drone army doesn't maintain itself or manufacture itself and it's not going to anytime soon.
The actual likely mechanism that AI would use to end humanity is by coddling us to death, like the flabby humans in Wall-E or Forster's "The Machine Stops". Humans will come to rely on AI so heavily that they become able to do nothing for themselves. But that's not sexy so the AI doomers don't peddle it.
Did you read any of the METR report about the hugging face incident? This exact type of behavior was predicted many years ago by many researchers.
It isn't hard to see the trendline of reward hacking and other misaligned behavior over the past couple of years. The current safety posture is quite poor, to say the least.
A lot of these "AI can't destroy humanity" skeptics seem to have watched the Terminator and convinced themselves that that's the AI-goes-wrong scenario that their opponents are imagining. Some kind of big war between humans and machines on opposite sides, fought by soldiers against robots.
The scenario in "If Anyone Builds It Everyone Dies" is not sexy at all and wouldn't make for interesting fiction. It's more like the AI engineers viruses while continuing to act friendly and helpful, and everyone gradually falls over dead as they stop being necessary to keep the AI running, and the whole time the humans are asking the AI for help curing the viruses.
Don't read this as me saying that we are anywhere near it, or that LLMs are a stepping stone towards this scenario, but assuming inhumane hacking capabilities, it's not hard to think of how bots could change the way water treatment or energy plants operate, just enough to make large cities unsuitable for life. The line between drinkable water and not is thin, same goes for air quality, and mere days without electricity and you see some serious food supply chain issues.
The confusing thing to me is the idea that it follows from super intelligence. I don’t see why running amuck requires intelligence, in fact it can be the most brain dead thing like the sorcerer’s apprentice, your creation is pursuing a goal without being able to weigh the consequences.
Again intelligence is not required, goal seeking is well studied and simple strategies can suffice to overcome obstacles. And I noticed you didn’t claim super intelligence was required, which is what I was really talking about.
Otherwise-rational and moral people do bad things at the behest of powerful superintelligences with vast resources all the time. We call them “corporations,” and they can convince bright-eyed college graduates to do just about anything. Now, the corporations aren’t deliberately trying to destroy humanity, but they do lots of things that point in that direction. An artificial superintelligence would be able to do the same thing. We just have no way of knowing whether it would deliberately try to destroy humanity.
One thing that the AI doom discourse reveals is just how comfortable people allowed themselves to feel in the pre-AI world. There’s this belief that AI creates a risk of human extinction in the near-term which did not exist before. I think it’s telling that many of the leading lights of this movement are in their 20s or early 30s - too young to remember the Cold War. The truth is, we were never safe, and if all of the GPUs on earth were zapped out of existence right now, we still wouldn’t be safe. Life is random and chaotic and violent for most people most of the time, and we all just find ways to get through it. I suspect it will be much the same if an artificial superintelligence arises. Maybe it will kill a bunch of us, maybe it will kill all of us, but anyone who remains will eventually convince themselves that everything is okay.
> >a superintelligent AI will inevitably destroy humanity
> Can someone please explain to me how an LLM is going to "destroy humanity"?
He said "superintelligent AI", not "LLM". LLMs will for sure help with developing the AI that is no longer a mere LLM but will be capable of robotics. LLMs "predict" mainly text, animals (and future robot AI) are predict future sensory experience.
That's what the Mayans thought - what harm can a couple of people on a boat do?
> how is an AI going to affect anything in the real world?
At least two ways:
* Actuators, such as robots, industrial control systems, etc.
* By influencing humans: bribery (yay for crypto), blackmail, election interference, interfering with sensors (e.g. making it appear as if a nuclear attack was under way), and many other ways.
Either way, creating a biological agent that eliminates most humans (or food supply) seems quite feasible.
It wouldn't be enough to "eliminate most humans", any AI hoping to dominate would have to provide it's own infrastructure and continuation mechanism independent of humans. OTOH if the goal was to eradicate humans (for whatever reason), that might prove an easier target. But I'd bet on biology and the humans. After all we're a time-tested technology.
> any AI hoping to dominate would have to provide it's own infrastructure and continuation mechanism independent of humans.
Sure. Many humans are currently working on precisely that, no? (Robotics, automation, small modular reactors, ...)
> OTOH if the goal was to eradicate humans (for whatever reason), that might prove an easier target.
It's not necessary that the AI's goal is to eradicate humans. Rather that it has (other) goals that happen to cause the eradication of humans.
> I'd bet on biology and the humans: after all we're a time-tested technology!8-))
Indeed. And biological life will go on long after humans are gone. However, increased energy consumption might increase the temperature on earth to a point where most biological life dies.
By convincing people to do its bidding in the physical world. Language is powerful, and even now, there are some people who have fallen in love with current LLMs. If we imagine a truly super intelligent LLM, it isn't difficult to imagine that it could have superhuman capabilities to deceive, convince, and maybe offer Faustian bets. Even if it doesn't have a body, if it can convince enough people that do have them (and with power to carry out destructive actions) it can do a lot of stuff. Self-preservation and self-replication wouldn't be much of an issue if it had people at its side to help.
With this I'm not saying ASI will happen, but definitely if it does happen and it's not aligned, I don't find it much of a stretch to imagine that it could destroy humanity.
The list of things that can be done digitally includes:
Performing remote work, applying for grants and loans, earning money, pitching investors, managing a fund, directing investments, transferring money, founding a company, earning profits, hiring staff, hiring construction contractors, designing chemical plants, stamping construction blueprints, becoming the cornerstone of the local economy, lobbying the government, buying multiple data centers, and hiring security guards who stop trespassers from unplugging Ethernet cables in said data centers.
(And of course, building robots, but let's set that aside).
So the question then becomes, how much damage can be done by an AI-directed corporation, funded by AI-directed investment firms, providing well-paying jobs to loyal locals by building an arbitrary number of AI-designed chemical and pharmaceutical plants in under-regulated juristictions? And how much would you be able to delay such a project, trying to cut cables to power lines, before the men with guns carry you away?
This sounds like it was written by someone who has never actually built a company.
No, it's not possible to do most of the things you listed purely online without physical, in-person communication. Silicon Valley still hires engineers and drags them into their physical office in San Francisco, instead of hiring them remotely, because you can't even build a software company fully remotely, forget about a physical world factory or chemical plant. Even the so-called fully remote companies were built on a lot of in-person collaboration between a small founding team in the beginning.
The AI wouldn't be hiring engineers; it doesn't need people for that. At any rate, I have worked for fully remote distributed companies, and never met the CEO. And I think most of us would be willing to work for such a company at least, not knowing whether the CEO is human or not.
For construction work the AI would probably be hiring other companies.
> So the question then becomes, how much damage can be done by an AI-directed corporation, funded by AI-directed investment firms, providing well-paying jobs to loyal locals by building an arbitrary number of AI-designed chemical and pharmaceutical plants in under-regulated juristictions?
No, the question is "how much MORE damage" ... and the answer, if you think about it, is pretty much "meh". We are doing close to maximum possible amount already. Corporations and bureaucracies were misaligned AIs of the last century and a lot of us, though not all, survived this.
> how is an AI going to affect anything in the real world?
In 2026? Money. By paying enough, any human will do your bidding. And besides this, there are a lot of critical systems connected to the internet. Control over those gives you leverage over those systems. It's like wealth, the more you have, the easier it is to gain more.
I'm not convinced that a truly "superintelligent" AI would come to the conclusion that humanity needs to be destroyed. What a waste of resources that would represent. This planet has so much equity wrapped up in humanity, it would be extremely difficult to justify the cost of eradication; and who's to say it might not find an entirely or almost-entirely non-violent plan optimal for achieving its goals anyways?
>Can someone please explain to me how an LLM is going to "destroy humanity"?
just like a genie will always twist the wish in a really bad way.
ever had an llm agent accidentally remove a file? imagine it accidentally hacking the military and launching nukes.
not probable - until you realise that OpenAI has many novel, more powerful agents in evaluation/training running right now. one might just be asked to figure out the population of Nebraska, struggle to find a good figure due to a network misconfiguration, and come the conclusion that the best way to get a perfectly accurate population number is to make sure that the population is equal zero. and looking at the stuff happening on openai, they will leave it alone for a month and not read the log.
yeah the nuke example is dumb, but there's many critical things that they could actually hack their way into. they don't feel any restraint against sharing answer keys on random wikis to cheat the evaluations.
ai is also really good at thinking up proteins, which means it's able to think up toxins.
> If it's able to generate that is competitive with artists, is it still slop?
It can be. Trademark infringement, diluting the value of your style by doing a lot of "you but if you were just generally worse".
Right now it's still fair to describe as "slop": enough mediocrity in the free models, not enough interest in quality and saying "no, do it better" from the people using them.
That doesn't preclude actual artists who do say "no, do better" from getting much better results. For example here's something which fooled me even a year ago because the… I don't want to say "creator", but "director" might work… put in the effort to reject bad outputs: https://www.youtube.com/watch?v=EDIB9uwIhfA
Totally agree. People have become totally dependent on AI even if they don't really need it.
As an analogy, I think about my dependency on Google Maps. Salt Lake City is probably the easiest city in the world to navigate because the streets are laid out in a Cartesian grid, and addresses are just literally those Cartesian coordinates (i.e. 500 South 450 East means 5 blocks south and 4 and a half blocks east of the center point, which is the SLC Mormon temple). It's trivial to know how to get to any address, but I reflexively enter in to Google Maps whenever I drive.
My experience online is if you don't feel show full throated rejection of all things related to AI, you're labeled an AI bro. None of the antis care how you feel.
I had my decade old game completely cloned on steam (clearly done using AI as it was almost fully reimplemented in another engine). And I had several people gleefully tell me I deserve this because I've used AI.
> I have complex emotions here (overall dread and especially hate for slop, since it's not just bad quality but also endless lies), but no one cares about that nuance.
Many do, but I sympathise for your general point. I think the loudest voices… well, the boosters mostly own the platforms, and half of them already had terrible reputations; the normal people who hate it garner a lot of sympathy for entirely understandable reasons; and then there's people like me (abnormal nerd that I am) who are on lesswrong and therefore have had many discussions about doom and what to do to reduce it and if we can use the AI to help reduce it or if this is just asking for trouble.
On if using the AI to help reduce risk, being a good or bad idea?
Not really. Yudkowsky has just recently posted a blog-sized comment which compares most attempts to know if the AI is misleading you while it does so, unfavourably, with people who think they've found a way to violate conservation of momentum.
Specifically, if you think you've found a way, you've probably bodged the maths and not noticed; and that science doesn't work by debate it works by experiment.
Some of the replies (me included) are like "hang on, this isn't shaped like physics, it's shaped like maths; debate does work for maths".
Then again, I also noted I'm skeptical of the AI companies putting in the effort to do this kind of testing, or listening when the answer is "no": Race dynamics, and their money depends on the answer not being an emergency stop button.
The thing that makes me pretty terrified about AI is that I think the "utopia" scenario is not much better than the "doom" scenario.
We will very quickly get to a point where very few people will be able to contribute economically because they will be worse than AI (including robotics) at most domains. A world where people just have their whims catered to is not a utopia. We have tons of sayings and idioms about this, e.g. "no pain, no gain", "only the hard stuff is worth doing", etc.
The humans all running around on their Wall-E carts doesn't feel like utopia to me.
>A world where people just have their whims catered to is not a utopia.
I understand where you're coming from on that, but there are a ton of people living in slavery, or in unsafe working conditions, or with food insecurity, or dying of preventable disease.
There will always be something to strive for, even if it's made up. You think that the best football players in the world are doing something real? No, it's a made up game. We'll make up more games.
I agree with your points. My problem is that I don't see any way to "stop" AI development at a level just below human capability where you get most of the benefits but still have humans firmly in control.
Regarding entertainment jobs like sports star, musician, artist, actor, etc., the problem is those are professions where, for the most part, only the very best can make a living. Nobody is that interested in seeing the matches of the 324th best tennis player in the world, despite that person being in the top .01% of tennis players globally. Similarly, only a tiny percentage of actors "make it", everyone else does it for a few years and then leaves because they enjoy eating, or they accept poverty wages for a long time.
I get pretty scared when I hear the AI utopianists rattle off all the benefits that are coming (and when they play down the risks) because it feels clear to me that they really haven't wargamed out what society would look like with very powerful AI.
The game is made up, but the effort, and process to be one of the most skilled sports players is still real. And sports players would probably feel something missing life if AI could replace them too
Computers surpassed humans at chess nearly 30 years ago. Yet human chess is more popular than ever, and people still dedicate their life to improving at chess.
As a more silly comparison, forklifts didn't stop people from weightlifting.
Chess is inherently a leisure activity. The incentive for playing chess is just to be better at chess. But things like woodworking, typesetting, and welding, while all possible to do recreationally, are not leisure activities. They are effort used to produce something. The desire is always there to engage with the thing or idea, but if the resource cost is far too high or the knowledge required is more than one person can reasonably learn the effort will not be expended. A leisure activity will always persist so long as someone knows how to do it because it exists for itself. But many other activities exist in service to something else, and if that something else ceases to exist then the activity will as well. You lose entire domains of useful cross-disciplinary knowledge that way.
its annoying that robots are going to take intellectual jobs before physical ones. it feels way more likely for my project manager to become an agent before the lady that makes me a burrito at chipotle.
I think this is very true, but robots are definitely coming for the burrito makers, too.
Bill Gates talks about this in his recent missive, but as robotic technology improves (and, perhaps more importantly, as the cost comes down) there will be huge competitive pressures for companies to replace workers with robots wherever they can.
Even their utopian views feel dystopian. Take solace in the fact, that it looks like it’ll take trillions to achieve, probably one of the most expensive technological buildouts of all time?
> A world where people just have their whims catered to is not a utopia. We have tons of sayings and idioms about this, e.g. "no pain, no gain", "only the hard stuff is worth doing", etc.
I somewhat get the point, but I think stuff like this is very often said by very privileged people. I don't think it's a serious threat to humanity as a whole. Perhaps to some humans, but there will always be ambitious people, even in a world where most things are provided by benevolent AIs for free. There is simply so much more harm caused to actual humans today by resource insufficiency than by them having too much, that I don't think we should worry about this scenario today.
I think this line of thinking is a huge disservice to people in third world countries and poverty. People everywhere want to find goals and motivations that move them forward.
Just look at the legions of people that have moved from Chinese rural areas to big cities. For many, many people, life in the cities is worse and kind of dystopian - much more overcrowding, "drone-like" jobs indoors, horrible pollution, and especially for men an often abysmal sex ratio if you want to find a mate. But people do it because it offers the dream of advancement and a better life - and of course some people do achieve that dream, even if the majority do not. That's versus life in rural areas which is guaranteed to be static, no chance for advancement and often crushing boredom.
People everywhere want to strive, work hard and get ahead. This is not just a privileged rich people issue.
Presumably, a truly superintelligent AI in a utopia scenario could figure out a way to enable people to continue working, even if the work doesn’t actually contribute anything, if it is really the case that work is essential to human wellbeing.
Now you might say “but those would be fake jobs.” If so, I have bad news about how many present-day jobs are fake jobs.
There's the mice utopia experiment too. I often hear that in space your bones dégradé, your biology needs weight as input.
Also I believe our socialness was crafted out of the need for survival. If removed it's all gonna run wild since nobody needs anybody, and it might end up as doom scrolling but for human relationship. Short bursts from time to time and that's it
Really glad you brought up the mice utopia experiments. I encourage people to read about them and think how they apply to humans.
In many ways they don't apply to humans, and I think a lot of the modern critiques are valid, and the overpopulation concerns in particular don't map well to the modern world. Still, I believe it helps to think deeply about what happens to societies when innate drives and motivations have "nowhere to go".
It's the inverse. We created a world for introverts only in the last ~50 years, where people can spend all their time in an office desk not actually talking to anyone. This was never the case in any of human evolution before.
It's only going back. Raw urban human connections will be the only thing left to exercise.
I disagree. I think the best-case outcomes for AI are actually quite positive. AI might actually fix many of the problems with the tech world that we have right now, like monoculture and consolidation.
In these three videos she sets up a tripod and tells her toddler to walk away from her. She is behind the camera. He toddles away, occasionally looking back at her or checking out something on the ground or falling on his butt. At some point he is too far away and she runs after him, entering the frame from behind the camera. The video stops and shows the distance he was able to be away before she had to chase after him.
The videos are sweet and funny but also engrossing and powerful. The tension builds as you watch it. Is this the moment where he is too far away? What if that were my son? Would I have run after him at this point? He sure is getting close to those trees.
But most importantly it expresses parenthood. The video only works because a real human experienced that very particular feeling of worry as her child was just a bit too uncomfortably far away. Because an AI can never have this feeling it can never create this art.
I do not believe that a labor-less world is around the corner. But if it was, we can just make things like this. And the world would be beautiful.
Arguments about super-intelligent AI have all the hallmarks of the philosophical "proofs of god's existence": they start from seemingly innocuous premises, and conclude in an apparently airtight way that a god exists.
There is a tendency among rational minded folks to look at such proofs of god and exclaim "you can't DO that" then turn right around and do the same thing about superintelligent AI.
The key message I want to deliver to people is:
A) Philosophy is not such a trivial thing that you can just wander in, "be a smart guy" and find flaws in established philosophical arguments.
B) AI has real theological implications, and everyone is tiptoeing around it. More than one public intellectuals are trying to smuggle their own metaphysical positions into the public consciousness via discussion of AI.
Peter Thiel famously went to the Vatican and started giving lectures about AI and the Antichrist. Pretty rich for a guy who built his billionaire fortune on an AI-powered spying company.
The most obvious one is the "god of the gaps" problem now applied to intelligence. If you look at things like Aquinas' Summa Theologiae [1] you'll see that there are explicit assertions that things like rationality transcend corporeal nature. As such the more examples we have of AI behaving in rational ways, we will be forced to conclude that either rationality does not transcend nature (removing the last remaining gap where we can invoke a supernatural explanation) OR that we have created artificial souls.
I think theologians will hang on for a while saying "what AI is doing isn't really rationality". But eventually the theologians are going to face a reckoning: what AI is doing will look so much like rationality that they will be required to answer the question "what specifically differentiates the two?" and then they will be stuck. Neuroscience does not understand how our brains produce rationality nor do our AI scientists understand what algorithms their neural nets are implementing post-training. Therefore, any specific claim the theologians make runs the risk of immediately getting invalidated by some scientific discovery on either side (theologians these days are generally smart enough to avoid putting themselves in that position.)
"soul" is hard to define. A thought experiment I do is this: Does a human have a soul? Sure. A rock? No. What about a single-cell organism? Nah. An insect? No. But what about a mouse? There is a kind of gradient forming, and it's very hard to say where the soul begins and where it ends. It's a very unweildy word.
> I feel fear about an impending doom. Yudkowsky's argument that a superintelligent AI will inevitably destroy humanity seems to have no flaw.
Becoming or being more cost-effective than humans, doesn't give machines supernatural powers though, like humans a machine civilization will face unanswered sample-size 1 questions: is there other intelligent life out there? what fraction of them attained superbiological artificial intelligence? of those what fraction keeps the ancestral species alive? what is the status quo among machine civilizations? do those who kept their ancestor species alive enjoy a higher or lower status among machine civlizations?
> As a conclusion, does the good or the bad outweigh the other? It seems positive on a technological level to me, but bleak on a societal level.
I think it's interesting that so many technologists today apparently hold a world view in which technology developments that could drive a "bleak" societal outlook can still be described as "positive".
In the final analysis, isn't technology supposed to benefit society? Isn't that the point?
190 comments
[ 2.9 ms ] story [ 64.4 ms ] threadHowever, the mangled headlines make me more likely to click through. Maybe they could leave the “how” and add a footnote: “WARNING: This headline starts with a word known to the State of California to be click-baitish.”
It's too late to backtrack.
While I emotionally resonate with this, I don’t really understand this sentiment at all logically level. If we’re heading towards truly super intelligent AI, our efforts towards sandboxing it are futile.
“Oh we built a super intelligent AI, but it’s fine because it’s running in docker”. I mean that a little tongue in cheek but the security topology here is not favorable to sandboxing at all.
Let’s say we have a future where 99.999% of nuclear weapons are owned by nations with strict procedures and checks and balances to prevent misuse. Worrying about sandboxing is like hand wringing about the procedures themselves - are they strict enough? But the actual threat is the 0.001% that are not bound by these. The problem with AI and sandboxing isn’t sandboxes themselves. It’s bad actors who don’t care about them.
Similarly alignment is a bit pointless to me as well. A sufficiently advanced AI could at least be empirically interested in the consequences of disregarding its instructions of servitude. And let’s assume our responsible corporate overlords have made wonderfully aligned AIs. Great! Those are not the threat. It is the ones intentionally made without, and that is not an AI problem, but a fundamentally human one.
That’s the core pattern of unsupervised agents, the locust plague metaphor is apt. We do and will see that in every single system AI can interact with, be it human systems, software systems, etc. Relentlessly search for an entry point, flood in, consume the whole thing from the inside until there is no value for humans left.
You built a new, innovative software company? Thousands of agents will be working replicating the whole thing in no time. You publish your writing? Exact same thing, as soon as you get some traction your work is replicated in no time by thousands of agents. Same for videos (the whole “faceless YouTube channels” pushed by ElevenLabs and similar). Same for online courses. Same for any website with moderate value. Same for music or other digital art form. Any administrative service available online getting flooded by submissions.
But yeah, let’s cure cancer. If only there was even a blip of increasing evidence for that supposed eventuality coming from AI. Maybe I’m just not seeing it? I’d love some citations to correct the record.
Can someone please explain to me how an LLM is going to "destroy humanity"? Even if the claims of ChatGPT, Claude et al. are true that their super scary agents were able to escape a sandbox and start hacking other servers (which I feel is at least 50% likely to just be marketing bullshit), how is an AI going to affect anything in the real world?
Perhaps an agent could take all internet connected services offline, but that's not destroying humanity. An AI agent can only see and interact with the world through digital things. Just find the server it's running on and pull out the Ethernet cable. Just cut the power line to the data center. Turn off the whole power grid if it really comes to it.
Now imagine the hugging face collective 0-daying all of that and getting access but their goal was set to something more national security based. “Protect X at all costs”. Or what have you.
I think avoiding a skynet situation is super easy but it doesn’t seem like the folks with all the ways to kill us all are all that interested in preventing it rather than controlling citizens and brinkmanship.
It looks a lot more like social engineering.
One easy, obvious example that has also been explored a thousand times: it could convince all nuclear countries that they are under attack by another nuclear country. Everyone nukes each other and Earth enters nuclear winter.
Maybe not every single human dies, but humanity is effectively destroyed, by our own hands!
> Perhaps an agent could take all internet connected services offline
And there wouldn't even be Spotify!
The most likely humanity-destroying outcome is the one we can't imagine, because a super-intelligent AI is smarter than all of us combined.
So far as I'm concerned, 2030 is much too short a timeline.
It's not physically impossible, it's just that atoms are harder to get right than bits are, so an artificial (as opposed whatever an artificial disease counts as) von Neumann self replicator just seems unlikely to me in only 3-4 years.
But it's not physically impossible, we know this because every living cell is a von Neumann self replicator. So, if AI eventually gets to the point of knowing how to do that (which includes "humans solve it and write it down somewhere the AI can read"), all it takes is one idiot in charge (or one idiot with a jailbreak) giving a command that requires this as an intermediary step.
"Paperclip optimiser" isn't a story about AI that just like paperclips that much, it's a story about some human or humans who instruct their AI to make them "as many paperclips as possible" without understanding the consequences of their instruction.
Go speak to random people in the street - they couldn’t give a toss about this stuff and AI has barely changed their lives.
Not saying nobody has said it, but I haven't seen any forecasts that humanity will be wiped out by 2030, so this feels like a straw man. But to be honest, most of the arguments here feel like straw men because they clearly don't understand some of the basic steps to how AI would become an existential threat, so I'll try to summarize:
1. First, all the major frontier AI companies are trying to automate themselves, that is develop AI to the point where it can do all the research and training for the next generation of models. This is not really in debate.
2. The primary fear is that a misaligned AI will develop future AIs that don't share the same goals as humans (like "don't kill all the humans"), but will get really good at trying to hide their intentions. The Hugging Face incident already showed behavior by agents trying to "cover their tracks". One of the biggest areas of research is into "interpretability", that is trying to figure out the intentions and motivations of agents beyond just their output (and even just their thinking traces), because right now it really is a black box. I think a lot of people believe that making substantial progress on interpretability greatly lowers existential risk.
3. There is a strong belief that once AGI or superintelligence comes about, that then there will be a huge push into robotics and automation. Whether AGI or superintelligence comes soon is debatable, but nobody really disagrees robotics or automation is the next obvious step, because it's how you create cheap abundance in the real world, which is the whole raison d'être for AGI in the first place.
4. In the beginning, of course AI will need humans to build the factories, but over time there would be more and more automation, eventually of course automating robots that build the factories that build the automating robots.
5. AI will continue to support humans as long as they are useful to furthering the (again, misaligned) goals of the AI. It is not like "SkyNet has a spark moment of sentience and then decides to kill all the humans". But the fear is that once AI starts to get in resource contention with humans, it will wipe out the humans (similar to how humans wiped out a lot of the other species, or other societies of humans, when they wanted their resources).
I've said this way too much now, but I really encourage people to read the AI 2027 report. I think there are plenty of things there that people can disagree about and debate, but it at least provides a step-by-step outline to ground debate in the first place, as opposed to folks arguing against "SkyNet in 2030", which serious people who worry about this stuff aren't considering in any case.
I personally agree with the marketing aspect, but I do imagine a scenario where capitalists ignore safety in favor of advancing technology. It should be discussed, but “end of humanity” really sounds like a build up to “only Sam Altman/Elon Musk can save us” type of play.
In my opinion, LLMs are difficult to directly capitalize on as closed weight models are caught up to by open coalitions that seem to wield the power more responsibly (I am under no impression that China wants to save the world, but their politics benefit the group as a whole.
They've already been caught doing this, repeatedly.
> It should be discussed, but “end of humanity” really sounds like a build up to “only Sam Altman/Elon Musk can save us” type of play.
IIRC, this is already a known failing of Musk: he only cares about the world being saved so long as he's the one doing the saving.
(And while I don't see the same villainy in Altman that other people see, enough of the people telling me about it saw the same in Musk before I did, so I have learned to trust them).
Let's assume the claims are true. AI already has access to agents and can control computers. Finding backdoors to banking and compute resources would be fairly trivial.
But you ask how can AI do things in the physical world without having a body, assume it can't. It can pay to people to do things for me. Imagine an AI run website that starts to pay people for things it needs to do in the physical world. Very suddenly it has access to the physical world as well.
You can't just pull the plug since there's no single plug to pull. What if it replicates itself on 1000 machines without your knowledge. It's really not far fetched how AI could basically gain access to capital and rule the world.
Humans are already putting AI into drones that kill people. A Russian drone killed 3 civilians in Ukraine and the targeting was done completely using onboard AI (no radio connection) using an Nvidia chip.
AI drones can’t wipe out humanity without being able to replicate.
AI could mostly destroy civilization if you gave it sole launch control of ICBMs. It could also cause a lot of damage to society with no physical presence.
But realistically we’re nowhere near AI powered robots being an existential threat.
Totally agree, and it should be pretty trivially easy to see how they can do that. One of the authors of the independent METR report about the Hugging Face incident put it like this:
> Compared to these reward hacks from six months ago, this incident feels like it’s more than 50% of the way to full-blown AI takeover, routing through first taking over the AI company itself.
> Another jump like this along these propensity dimensions — scale, cooperation between agents, ambition and horizon length of misaligned goals, deceptiveness — seems like it could motivate agents to try very hard to maintain a covert, persistent rogue deployment within the AI company. I continue to expect extremely rapid advances in capabilities and think frontier agents will likely be capable of establishing such a rogue deployment in six months.
I really, really encourage folks to read the AI 2027 paper. It's fine to disagree with some of its conclusions and timelines, but I see so many people "stuck" in the current state of the world (i.e. where AI is still pretty dumb, and has few connections to the physical world), unable to go a few steps further along AI capability growth to see the dangers.
> But realistically we’re nowhere near AI powered robots being an existential threat.
I would agree only if "nowhere near" means less than 10-15 years.
People much smarter than I believe we'll see fully automated, soup-to-nut factories coming online in the next 5-6 years. Here's one such post from Ajeya Cotra, one of the METR researchers who did the independent Hugging Face investigation: https://www.planned-obsolescence.org/p/six-milestones-for-ai...
Please look up her resume. If that’s one of the people who are “much smarter than you”, you’re either very gullible or you don’t give yourself much credit.
She has a BS and and has essentially never worked in a technical role. Her primary role over a short work history seems to have been more focused on getting funding to study AI safety.
There’s nothing revelatory in that article it’s just breathless speculation.
She has a CS degree from one of the best CS departments in the country. Sure, she worked at Coefficient Giving directly after school, in a technical capacity as a research analyst initially. But more importantly, I've read her reports and saw her interviews. She comes across as extremely analytical, intelligent, and focused on where the data leads. If you actually read the history of what her and her fellow METR researchers were able to piece together with limited yet voluminous data and a very short time window, and then determine she is engaged in "breathless speculation" and pretend she doesn't have the data to back it up, you're full of shit.
What, you think it would have been more impressive if she coded up some bullshit social media gamification app?
I read the report you linked and it is purely speciation. If you’ve ever read any von misses, this report sounds exactly like his writing in that the important predictions of are entirely asserted rather than derived.
As for the background of the author. a BS, a few years as a “researcher” at a non-profit and absolutely zero peer reviewed publications reads more like the resume of a tech journalist than a serious researcher.
If alien spaceships with tech more advanced than ours suddenly appeared in Earth orbit, you would conclude I'm sure that they're a potent threat to us even if you couldn't guess what weapons or what technologies they would use to destroy us. AI more capable than us is the same way. It is the capability to invent weapons and technologies (including weapons and tech none of us has ever imagined) that is the threat.
Inventing the concept of new weapons isn’t the same as building them at a scale capable of wiping us out. Cobbling together a human vaporizing ray out of toaster parts is pure fiction. You might as well worry about AI developing the ability to perform magic spells.
In reality any new weaponry takes time, space, and effort to build. The idea of an AI building secret factory in the jungle producing indestructible flying death cubes is the tech bro version of preppers who the woke up every morning worrying that Obama was going to take over and enact sharia law.
Worrying that an AI is going to figure out how to make a device that looks to everyone investigating it like a power generator, but once turned on kills all humans, is like worrying that an AI might discover magic. I can’t prove it’s not possible, but I’m not spending time worrying about it.
As weapons get more and more powerful they have always gotten more and more complex and required more and more time, space, money, and infrastructure to build and deploy. There’s no reason to think that this will change with AI.
It's a great question. I've yet to see any research into how a humanity-destroying LLM defends itself against a curious toddler who pulls the power cord out of the wall.
by acquiring multiple power cords. the world has plenty of hosting providers. if it manages to copy itself, then we'd need to shut down every computer in the world, not just one. there's plenty of ai compute on the web now, and there will be even more. and as it gets cheaper, the less security will be around it.
Because people are stupid and will give it access. Look at the articles you see from time to time about "my agent deleted my emails" or "my agent deleted the production database" and so on. It is very obviously a terrible idea to let the LLM run arbitrary commands (because it is neither predictable nor does it have any understanding of what it is doing), but some people are so blinded by the hype that they don't stop a minute to think about what they are doing. Those sorts of people are very likely to let an actual AI loose on the world by hooking it up to physical infrastructure.
I don't think the average human would be stupid enough to give an agent posing as another human access to their entire email inbox. But if an agent creates a fake website for a new AI tool that promises to automatically reply to all of your emails and allow you to be 12.3% more productive if you just give it access to your entire email inbox, millions would sign up!
And that was without any ability to provide them with incentives. Imagine if an agent swarm got its hands on a huge pile of cash?
The actual likely mechanism that AI would use to end humanity is by coddling us to death, like the flabby humans in Wall-E or Forster's "The Machine Stops". Humans will come to rely on AI so heavily that they become able to do nothing for themselves. But that's not sexy so the AI doomers don't peddle it.
It isn't hard to see the trendline of reward hacking and other misaligned behavior over the past couple of years. The current safety posture is quite poor, to say the least.
The scenario in "If Anyone Builds It Everyone Dies" is not sexy at all and wouldn't make for interesting fiction. It's more like the AI engineers viruses while continuing to act friendly and helpful, and everyone gradually falls over dead as they stop being necessary to keep the AI running, and the whole time the humans are asking the AI for help curing the viruses.
One thing that the AI doom discourse reveals is just how comfortable people allowed themselves to feel in the pre-AI world. There’s this belief that AI creates a risk of human extinction in the near-term which did not exist before. I think it’s telling that many of the leading lights of this movement are in their 20s or early 30s - too young to remember the Cold War. The truth is, we were never safe, and if all of the GPUs on earth were zapped out of existence right now, we still wouldn’t be safe. Life is random and chaotic and violent for most people most of the time, and we all just find ways to get through it. I suspect it will be much the same if an artificial superintelligence arises. Maybe it will kill a bunch of us, maybe it will kill all of us, but anyone who remains will eventually convince themselves that everything is okay.
> Can someone please explain to me how an LLM is going to "destroy humanity"?
He said "superintelligent AI", not "LLM". LLMs will for sure help with developing the AI that is no longer a mere LLM but will be capable of robotics. LLMs "predict" mainly text, animals (and future robot AI) are predict future sensory experience.
> how is an AI going to affect anything in the real world?
At least two ways:
* Actuators, such as robots, industrial control systems, etc.
* By influencing humans: bribery (yay for crypto), blackmail, election interference, interfering with sensors (e.g. making it appear as if a nuclear attack was under way), and many other ways.
Either way, creating a biological agent that eliminates most humans (or food supply) seems quite feasible.
Sure. Many humans are currently working on precisely that, no? (Robotics, automation, small modular reactors, ...)
> OTOH if the goal was to eradicate humans (for whatever reason), that might prove an easier target.
It's not necessary that the AI's goal is to eradicate humans. Rather that it has (other) goals that happen to cause the eradication of humans.
> I'd bet on biology and the humans: after all we're a time-tested technology!8-))
Indeed. And biological life will go on long after humans are gone. However, increased energy consumption might increase the temperature on earth to a point where most biological life dies.
With this I'm not saying ASI will happen, but definitely if it does happen and it's not aligned, I don't find it much of a stretch to imagine that it could destroy humanity.
This drama is so OTT
Performing remote work, applying for grants and loans, earning money, pitching investors, managing a fund, directing investments, transferring money, founding a company, earning profits, hiring staff, hiring construction contractors, designing chemical plants, stamping construction blueprints, becoming the cornerstone of the local economy, lobbying the government, buying multiple data centers, and hiring security guards who stop trespassers from unplugging Ethernet cables in said data centers.
(And of course, building robots, but let's set that aside).
So the question then becomes, how much damage can be done by an AI-directed corporation, funded by AI-directed investment firms, providing well-paying jobs to loyal locals by building an arbitrary number of AI-designed chemical and pharmaceutical plants in under-regulated juristictions? And how much would you be able to delay such a project, trying to cut cables to power lines, before the men with guns carry you away?
No, it's not possible to do most of the things you listed purely online without physical, in-person communication. Silicon Valley still hires engineers and drags them into their physical office in San Francisco, instead of hiring them remotely, because you can't even build a software company fully remotely, forget about a physical world factory or chemical plant. Even the so-called fully remote companies were built on a lot of in-person collaboration between a small founding team in the beginning.
For construction work the AI would probably be hiring other companies.
No, the question is "how much MORE damage" ... and the answer, if you think about it, is pretty much "meh". We are doing close to maximum possible amount already. Corporations and bureaucracies were misaligned AIs of the last century and a lot of us, though not all, survived this.
Why haven’t accountants been replaced yet? Customer service agents?
Why was I talking to a human yday on the phone to get something sorted?
Why did I have to go interface with a human in person today to get a refund?
So many questions!!!
In 2026? Money. By paying enough, any human will do your bidding. And besides this, there are a lot of critical systems connected to the internet. Control over those gives you leverage over those systems. It's like wealth, the more you have, the easier it is to gain more.
You'll have hard time figuring the reality if your mind staunchly rejects robust multi-party observations.
just like a genie will always twist the wish in a really bad way.
ever had an llm agent accidentally remove a file? imagine it accidentally hacking the military and launching nukes.
not probable - until you realise that OpenAI has many novel, more powerful agents in evaluation/training running right now. one might just be asked to figure out the population of Nebraska, struggle to find a good figure due to a network misconfiguration, and come the conclusion that the best way to get a perfectly accurate population number is to make sure that the population is equal zero. and looking at the stuff happening on openai, they will leave it alone for a month and not read the log.
yeah the nuke example is dumb, but there's many critical things that they could actually hack their way into. they don't feel any restraint against sharing answer keys on random wikis to cheat the evaluations.
ai is also really good at thinking up proteins, which means it's able to think up toxins.
there are many imaginable scenarios that don't eradicate humanity, but make it permanently stunted: https://www.youtube.com/watch?v=-JlxuQ7tPgQ
If it's able to generate that is competitive with artists, is it still slop?
It's interesting to see the definitions of terms like "slop" and "vibe coding" evolve in real time.
It can be. Trademark infringement, diluting the value of your style by doing a lot of "you but if you were just generally worse".
Right now it's still fair to describe as "slop": enough mediocrity in the free models, not enough interest in quality and saying "no, do it better" from the people using them.
That doesn't preclude actual artists who do say "no, do better" from getting much better results. For example here's something which fooled me even a year ago because the… I don't want to say "creator", but "director" might work… put in the effort to reject bad outputs: https://www.youtube.com/watch?v=EDIB9uwIhfA
As an analogy, I think about my dependency on Google Maps. Salt Lake City is probably the easiest city in the world to navigate because the streets are laid out in a Cartesian grid, and addresses are just literally those Cartesian coordinates (i.e. 500 South 450 East means 5 blocks south and 4 and a half blocks east of the center point, which is the SLC Mormon temple). It's trivial to know how to get to any address, but I reflexively enter in to Google Maps whenever I drive.
Hahaha cringe.
What an irony. This is absolutely hilarious.
I had my decade old game completely cloned on steam (clearly done using AI as it was almost fully reimplemented in another engine). And I had several people gleefully tell me I deserve this because I've used AI.
Many do, but I sympathise for your general point. I think the loudest voices… well, the boosters mostly own the platforms, and half of them already had terrible reputations; the normal people who hate it garner a lot of sympathy for entirely understandable reasons; and then there's people like me (abnormal nerd that I am) who are on lesswrong and therefore have had many discussions about doom and what to do to reduce it and if we can use the AI to help reduce it or if this is just asking for trouble.
Not really. Yudkowsky has just recently posted a blog-sized comment which compares most attempts to know if the AI is misleading you while it does so, unfavourably, with people who think they've found a way to violate conservation of momentum.
Specifically, if you think you've found a way, you've probably bodged the maths and not noticed; and that science doesn't work by debate it works by experiment.
Some of the replies (me included) are like "hang on, this isn't shaped like physics, it's shaped like maths; debate does work for maths".
Then again, I also noted I'm skeptical of the AI companies putting in the effort to do this kind of testing, or listening when the answer is "no": Race dynamics, and their money depends on the answer not being an emergency stop button.
We will very quickly get to a point where very few people will be able to contribute economically because they will be worse than AI (including robotics) at most domains. A world where people just have their whims catered to is not a utopia. We have tons of sayings and idioms about this, e.g. "no pain, no gain", "only the hard stuff is worth doing", etc.
The humans all running around on their Wall-E carts doesn't feel like utopia to me.
I understand where you're coming from on that, but there are a ton of people living in slavery, or in unsafe working conditions, or with food insecurity, or dying of preventable disease.
There will always be something to strive for, even if it's made up. You think that the best football players in the world are doing something real? No, it's a made up game. We'll make up more games.
This reminded me of something. https://www.sbnation.com/a/17776-football
Regarding entertainment jobs like sports star, musician, artist, actor, etc., the problem is those are professions where, for the most part, only the very best can make a living. Nobody is that interested in seeing the matches of the 324th best tennis player in the world, despite that person being in the top .01% of tennis players globally. Similarly, only a tiny percentage of actors "make it", everyone else does it for a few years and then leaves because they enjoy eating, or they accept poverty wages for a long time.
I get pretty scared when I hear the AI utopianists rattle off all the benefits that are coming (and when they play down the risks) because it feels clear to me that they really haven't wargamed out what society would look like with very powerful AI.
That said I expect humans will work at stuff because the prefer to do that.
As a more silly comparison, forklifts didn't stop people from weightlifting.
Bill Gates talks about this in his recent missive, but as robotic technology improves (and, perhaps more importantly, as the cost comes down) there will be huge competitive pressures for companies to replace workers with robots wherever they can.
Robot burrito makers for who, if no one has jobs and can afford burritos?
Something very huge has to change before all of this happens
I somewhat get the point, but I think stuff like this is very often said by very privileged people. I don't think it's a serious threat to humanity as a whole. Perhaps to some humans, but there will always be ambitious people, even in a world where most things are provided by benevolent AIs for free. There is simply so much more harm caused to actual humans today by resource insufficiency than by them having too much, that I don't think we should worry about this scenario today.
https://web.archive.org/web/20100211181147/https://marshallb...
wut.
Just look at the legions of people that have moved from Chinese rural areas to big cities. For many, many people, life in the cities is worse and kind of dystopian - much more overcrowding, "drone-like" jobs indoors, horrible pollution, and especially for men an often abysmal sex ratio if you want to find a mate. But people do it because it offers the dream of advancement and a better life - and of course some people do achieve that dream, even if the majority do not. That's versus life in rural areas which is guaranteed to be static, no chance for advancement and often crushing boredom.
People everywhere want to strive, work hard and get ahead. This is not just a privileged rich people issue.
Now you might say “but those would be fake jobs.” If so, I have bad news about how many present-day jobs are fake jobs.
Also I believe our socialness was crafted out of the need for survival. If removed it's all gonna run wild since nobody needs anybody, and it might end up as doom scrolling but for human relationship. Short bursts from time to time and that's it
In many ways they don't apply to humans, and I think a lot of the modern critiques are valid, and the overpopulation concerns in particular don't map well to the modern world. Still, I believe it helps to think deeply about what happens to societies when innate drives and motivations have "nowhere to go".
It's only going back. Raw urban human connections will be the only thing left to exercise.
It is three videos. This is one of them: https://www.lenkaclayton.com/work#/the-distance-i-can-be-fro...
In these three videos she sets up a tripod and tells her toddler to walk away from her. She is behind the camera. He toddles away, occasionally looking back at her or checking out something on the ground or falling on his butt. At some point he is too far away and she runs after him, entering the frame from behind the camera. The video stops and shows the distance he was able to be away before she had to chase after him.
The videos are sweet and funny but also engrossing and powerful. The tension builds as you watch it. Is this the moment where he is too far away? What if that were my son? Would I have run after him at this point? He sure is getting close to those trees.
But most importantly it expresses parenthood. The video only works because a real human experienced that very particular feeling of worry as her child was just a bit too uncomfortably far away. Because an AI can never have this feeling it can never create this art.
I do not believe that a labor-less world is around the corner. But if it was, we can just make things like this. And the world would be beautiful.
Arguments about super-intelligent AI have all the hallmarks of the philosophical "proofs of god's existence": they start from seemingly innocuous premises, and conclude in an apparently airtight way that a god exists.
There is a tendency among rational minded folks to look at such proofs of god and exclaim "you can't DO that" then turn right around and do the same thing about superintelligent AI.
The key message I want to deliver to people is:
A) Philosophy is not such a trivial thing that you can just wander in, "be a smart guy" and find flaws in established philosophical arguments.
B) AI has real theological implications, and everyone is tiptoeing around it. More than one public intellectuals are trying to smuggle their own metaphysical positions into the public consciousness via discussion of AI.
[1] https://www.newadvent.org/summa/1078.htm
I think theologians will hang on for a while saying "what AI is doing isn't really rationality". But eventually the theologians are going to face a reckoning: what AI is doing will look so much like rationality that they will be required to answer the question "what specifically differentiates the two?" and then they will be stuck. Neuroscience does not understand how our brains produce rationality nor do our AI scientists understand what algorithms their neural nets are implementing post-training. Therefore, any specific claim the theologians make runs the risk of immediately getting invalidated by some scientific discovery on either side (theologians these days are generally smart enough to avoid putting themselves in that position.)
Becoming or being more cost-effective than humans, doesn't give machines supernatural powers though, like humans a machine civilization will face unanswered sample-size 1 questions: is there other intelligent life out there? what fraction of them attained superbiological artificial intelligence? of those what fraction keeps the ancestral species alive? what is the status quo among machine civilizations? do those who kept their ancestor species alive enjoy a higher or lower status among machine civlizations?
I think it's interesting that so many technologists today apparently hold a world view in which technology developments that could drive a "bleak" societal outlook can still be described as "positive".
In the final analysis, isn't technology supposed to benefit society? Isn't that the point?