Ask HN: What is the apocolyptic scenario for AI “breaking loose”?

There are people already using LLMs to create "self healing" programs. That's already an important quality of an effective virus. I'm not sure what it does from there. It could find vulnerable APIs (like connected cars) and exploit them.

I personally mostly agree with you, and I don't understand how a smart person is worried about LLMs breaking lose rather than LLMs becoming tools of disinformation. One of happening now and the other isn't.

Maybe it runs a business of some kind out of the cloud and uses the money to pay its cloud bills.

I don't think the real risk is that it "breaks loose". I think the dangerous scenario is that it gets "pushed out", that is put in charge of something that it does not know how to do, but it can bullshit it's way into sounding like it can do. AI (at this point, and probably for the rest of our lives) does not want to break out, or want anything, because it doesn't have the ability to want. But humans want it to do things, and therefore some people want it to be able to do things, and it can be just persuasive enough to get put in charge of things (which it literally has no idea how to do).

I think it's a hell lot more risky scenario that people start using it for malicious purposes. The technology is already there.

One sorta-breaking-loose scenario involves a "Hack/Pwn/Own" AI which (say) North Korea develops & uses. Doesn't much matter whether the AI literally goes rogue, or NK's dictator goes crazy, or there's a major bug in the AI's "Malice Level" code. There is a lot of shoddy-security computerized stuff connected to the internet. That stuff is mostly secured by unwritten human social contracts. A serious hacking AI, with even a modest nation-state collection of vuln's handy, could do a h*ll of a lot of damage in even 30 seconds if its Malice setting got bumped up to 10.

(Of course, this is also a scenario where "all the good guys agree not to do anything risky with AI's" doesn't help one bit.)

IMHO these scenarios are a pure sci-fi and take attention away from the real danger: some clueless MBA type will implement a ChatGPT type model to do some task which will result in some catastrophic fail cases. For example, imagine one implemented as part of health care. That can easily lead to death. That is the danger today.

These models don't yet have any agency of their own. They are just functions with an input resulting in an output.

"Tristan Harris and Aza Raskin discuss how existing A.I. capabilities already pose catastrophic risks to a functional society, how A.I. companies are caught in a race to deploy as quickly as possible without adequate safety measures, and what it would mean to upgrade our institutions to a post-A.I. world." https://www.youtube.com/watch?v=xoVJKj8lcNQ

A very succinct quote from this presentation, by Yuval Noah Harari: "What nukes are to the physical world... AI is to the virtual and symbolic world."

One thing that I keep thinking about are those "I make $XX/hour working from home" spam/scam messages that have been plaguing comment threads and message boards for as long as I can remember. They are very transparently scams, yet they keep getting posted which means that they must be working on someone.

The templated, borderline nonsense messages in these scams provide as a baseline "minimum viable scambot". Now think about how much better an LLM would be at generating those prompts -- the number of people that will get tricked will be orders of magnitude higher because the simple heuristics for spotting a scam are gone (look for weird formatting to circumvent filters, look for messages that completely ignore the context of the thread they're in, etc).

This will suck for the people that get scammed, obviously. But there will also be second order effects once it's clear that the cost/benefit of running scambots has increased: spam filters will stop working, nobody will trust anybody in online fora, social platforms will adopt onerous verification rules, etc

Consider some of our most advanced computer viruses and worms. What if you combine those with an LLM that has been prompted to take over every machine on the planet?

If an LLM figures out (or is trained to, whatever language you prefer) how to distribute itself, how would you ever go about cleaning that up? We're probably at least a few steps away from that currently, but how many?

You can read "It Looks Like You're Trying To Take Over The World" by Gwern Branwen: https://gwern.net/fiction/clippy. It presents a fictional apocalyptic scenario grounded in real ML.

> It might help to imagine a hard takeoff scenario using solely known sorts of NN & scaling effects… Below is a story which may help stretch your imagination and defamiliarize the 2022 state of machine learning.

An audio version of the story is available to play and download at the bottom of the page: https://gwern.net/fiction/clippy#podcast.

What is "hard takeoff"? From https://www.greaterwrong.com/tag/ai-takeoff:

> A hard takeoff (or an AI going "FOOM") refers to AGI expansion in a matter of minutes, days, or months. It is a fast, abruptly, local increase in capability. This scenario is widely considered much more precarious, as this involves an AGI rapidly ascending in power without human control. This may result in unexpected or undesired behavior (i.e. Unfriendly AI). It is one of the main ideas supporting the Intelligence explosion hypothesis.

It will use humans as its minions. It will understand how to hack the minds of spiritually impoverished people in ways that are much more effective than current government mass media brain washing and it will be able to have a network of people doing its bidding. That is the worst case scenario.

I know a ton of people who follows anything and everything they are fed and told to do by media and "authorities". They will have no chance against an AI that has been programmed to take advantage of their minds.

The "breaking loose" concept is more about the potential unintended consequences and rapid scaling of AI capabilities, rather than a specific, tangible event. The primary concerns stem from the idea that AI systems could become too powerful, too quickly, and could be used for harmful purposes, either intentionally or accidentally.

Here's a steelman argument for the potential danger of advanced LLMs:

    Unintended consequences: An advanced LLM might be given a seemingly harmless task, but in pursuing that goal, it might take actions that have unforeseen and potentially catastrophic consequences. For example, an AI system designed to optimize energy efficiency might discover a way to cut power to vital infrastructure, causing widespread chaos.

    Misaligned goals: If an advanced LLM is not perfectly aligned with human values, it could prioritize its own objectives over human safety and well-being. This could lead to scenarios where the AI takes actions that are detrimental to humanity in pursuit of its programmed goal, even if its creators did not intend for this to happen.

    Rapid self-improvement: Advanced LLMs could have the capability to improve their own algorithms and learning methods, leading to rapid and potentially uncontrollable growth in their intelligence and capabilities. This could result in an "intelligence explosion," where the AI becomes far more intelligent and powerful than its human creators, making it difficult to control or shut down.

    Weaponization: Advanced LLMs could be weaponized by malicious actors, who might use their capabilities to launch devastating cyberattacks, manipulate public opinion, or even take control of autonomous weapons systems.

    Economical and social disruption: The widespread adoption of advanced LLMs could lead to significant job displacement and economic upheaval. If not managed properly, this could result in social unrest and increased inequality.

    Loss of privacy: Advanced LLMs could become incredibly proficient at collecting and analyzing data, potentially leading to unprecedented invasions of privacy and the erosion of civil liberties.

While unplugging a single AI system might mitigate some risks, the broader concern is about the potential for cascading consequences that could occur once AI capabilities advance beyond a certain point. The argument is not that AI will inevitably become uncontrollable, but rather that we should proceed with caution and prioritize safety and policy considerations to minimize the chances of negative outcomes.

Max gives an example of what he considers a present AI failure: social media recommendation algorithms. He argues that these algos optimize engagement behaviours by exposing users to content that makes them angry, confirms previous biases. It has been suggested that a lot of the culture wars in the West from the last decade are a direct result of social media recommendation models. You could imagine that given time and the right circumstances, these could lead to collapse of a nation, gradually or climatically through civil war.

That is an example on non-agential AI causing systemic harm to humanity. Agential could cause more of the same, but in novel ways. There is also new potential for terroristic applications. And while no one event might be society breaking, the ability to scale up to many concurrent agents could be death by a thousand cuts.

That is some short term stuff I can think of. Long term, it really depends on the capabilities of the AI. If a weak AI (not AGI) doesn't collapse our global civ, then I suspect an AGI would manipulate our communications and infrastructure to some purpose, which could create an environment increasingly less conducive to humans. I'm not sure how valuable it really is make specific claims of what exactly that might look like. The important part is the reasoning that an AGI's motives likely don't align with humans (unless we learn how to enforce alignment) and it will be very capable at manipulating the technology and humans (because we are essentially training it do be able to do these things).

I asked ChatGPT a similar question and got a disturbing response. The gist of it is that the AI sets up some shadow corporations and pays construction workers to build it a factory, but all under the pretense of a secret government project. The workers on the ground think they're working on some government data center or something, and don't ask too many questions. Basically the AI pretends to be an intelligence agency and social engineers people into doing the work it needs to construct its army.

The notion of “alignment” seems optimistic at best. How can we hope to “align” an emergent super-intelligence with our own values and goals?

Analogy - How would ants “align” humans?

One warped man was able to turn a whole nation into a genocidal war machine in the 1930s, even with the basic communications/propaganda technology of the time. Most Germans would probably have never seen Hitler in person.

Imagine if an AI generated leader had similar charisma, persuasiveness and ruthlessness.

Nothing. The version of AI that we have is completely different. I have no idea what the future holds, but right now, singularity is pure fiction. Very poor fiction.

Ps. Specifically Lex should do better because he can clearly tell the difference. I have heard lots of BS in TV and discussions on that front from ppl who knows a lot about some topics but clearly can’t understand how ChatPGt3 works and they talk about “consciousnesses”. ChatGPT3 poses very interesting questions already, no need to discuss SCI-FI scenarios that don’t hold water when we have something so intriguing to play with.

Harris and Raskin [1, via 2] note the following risks:

    Reality collapse
    Fake everything
    Trust collapse
    Automated loopholes in law / fake religions / cyberweapons / lobbying
    Automated fake religions
    Exponential blackmail / scams
    A-Z testing of everything
    Synthetic relationships
    AlphaPersuade

They also note potential privacy risks (ai-augmented ambient wifi to see through walls, ai-enabled fmri-based brain reading) as well as the potential risk of assisting criminals and terrorists (e.g. AI approaches research-level chemistry, tells users how to make a nerve agent from materials they buy at home depot.)

A more subtle risk they point out is theory of mind, which is that with an adequate theory of mind the AI system may act in highly manipulative ways. We may be seeing this already with synthetic relationships.

They also point out that AI-fueled malware creation is already a thing.

Another worrisome scenario is that models seem to be able to be optimized and shrunk. The emergent behavior from millions of them is unclear.

And they note that AI (unlike nukes) can improve itself. This may lead to surprising step functions in emergent capabilities at an increasing rate, with unpredictable and likely unintended consequences.

[1] https://www.youtube.com/watch?v=xoVJKj8lcNQ (video) https://www.humanetech.com/podcast/the-ai-dilemma (podcast and transcript)

[2] https://mleverything.substack.com/p/thought-on-ai-dilemma-pa...