I think the idea is that it doesn't matter who builds the AGI first if we all end up dying anyway.
And the first step to prevent that from happening is for us to stop, and then hope that the other powerful countries will see that and stop as well. And everyone who's left can be forced to stop by the powerful countries who agreed to the treaty.
Yeah, I get that, but I what if Russia and China think "Well, we're fucked anyway, our population is aging faster than any other country in history, our economies are fucked by multiple factors including COVID, War, sanctions and technology bans, our countries won't make it past 100 more yeah, maybe not even 50 more years... if we achieve AGI before anyone else, or no one else does, there's maybe a 5% chance we can control it, and maybe a 60% or 70% chance it destroys all humans on earth or all life on earth... if we're lucky and we hit that 5% we'll rule the rest of the world, otherwise we were doomed as a country anyway..."
Then they end up racing to build an AGI we're not prepared for, and we all die.
The faint hope is that people are not crazy or suicidal enough to build a world-ending technology, that humanity will manage to see ourselves as being in the same boat, where we all live or all die as one. And just-not-do the thing that causes our extinction.
It's not a very realistic proposition, because even smart people often fail to understand these arguments (or just disagree with them), let alone the politicians (who are incentivized to ignore them), or the rest of the people. There's a lot at stake for the "winner" of the AGI race, who could build an AGI and align it successfully (which they can't do). And many countries are ruled by people who are clearly willing to do crazy and suicidal things for power.
That's why the author of this post doesn't put a lot of hope into the idea of his daughter growing up.
Yep, it's always a possibility, but it's better to reduce the chance any way we can. If the US banning it reduces the risk from 100% to 50%, that's fantastic.
I also don't think it's that likely that a leader would take those odds, if they understand the risk fully. The worry is that they won't understand or believe the risk, so that's where we should attack the problem, in my opinion--by putting out as much good information on AI risk, and combating misinformation, as best we can.
It's not hostile in the sense of hating us, it's just a maximizer, and when you're smart enough and try to maximize one thing (or some function of multiple things), anything that's not in your utility function gets sacrificed. Humanity maximized for cheap energy, and for a long time, the carbon composition of the atmosphere was not in our utility function. Now that's clear we might be killing ourselves by doing it, we're working it in, but if it turned out that CO2 was only dangerous to, say, several species that we didn't care about, it's very likely we'd just accept it as the cost of progress.
Why not just train an AI to care about humanity then? One problem is in defining actually what it means to care about or preserve humanity. Another problem is creating an AI that actually internalizes this as a goal, instead of just saying the things we want to hear until it gains enough power that we are not a factor in its plans anymore.
It's probably possible to align an AI, it's just not a problem we've solved yet. We've seen many, many, ways in which things can go wrong with less-intelligent AIs, and there's no reason to think that things will just magically somehow be easier as the AI gets more intelligent.
The artice is literally about how this will lead to a hostile superhuman intelligence that will wipe out humanity. It's not about AI tooling as an accelerator, which is a valid concern that imo can't be avoided.
Humans aren't that hostile, but we've driven many, many species to extinction or endangerment in trying to get the stuff we want. We destroyed habitats for farmland, hunted whales for oil, tigers for fun, elephants for ivory for our pianos, etc. Why do we like music, why do we want pianos? We just do. Evolution is a process that only optimized for reproductive fitness, to make us want to reproduce and pass on our genes, but we learned that we like music just because. We also don't all care that much about passing on our genes -- people stop having lots of kids once they get richer, some people don't even want kids at all. The way we train these AIs is a bit different from evolution, but it shares the characteristic that we don't have a lot of control over what the model learns (<https://en.wikipedia.org/wiki/AI_alignment>).
An AI we train to take care of us will almost certainly learn to want some other stuff, in addition to not exactly caring about us the way we want. That's fine if it's not too smart, we just stop it from doing dangerous stuff. The problem is when you crank it up to a million, and then those things that are just a tiny bit wrong become a massive problem, and you can't turn off the AI because it is smart enough to understand that the things it wants to do won't get done if it's off, and it doesn't want to let itself be modified in the same way that you wouldn't want someone to give you a pill that makes you stop caring about your children.
Or it could bring us post-scarcity economy, cure cancer and all disease, solve world hunger and wars... or it could become Roko's Basilisk, or become a Machine God and bring us the world of Warhammer 40k grimdark dystopia...
Fingers crossed for destabilising the world's information scarcity based economies. May your entire information/knowledge based industries like SAAS, law, medicine and finance be obliterated so we can clean their stain from the underpants of society.
Seriously, this has become religious fundamentalism based upon old bad sci-fi by hack writers. Come back to reality. It would be funny like ignorant Y2K freak outs if not for it being taken seriously.
Ok, so you don't think this is a problem, but why do you think that a super-intelligent AI would be something that we could control, or would be something that would care about keeping humanity around? Or do you believe that we won't be able to build an AI with enough capability to be dangerous, keeping in mind that GPT-4 is already showing signs of being smarter than even many intelligent humans in many areas?
I disagree. People are arriving at this conclusion after careful consideration and logical reasoning, not picking up a '50s sci-fi book and mistaking it for reality.
The Less Wrong community isn't a community of careful consideration and logical reasoning. It's a cult that believes their own beliefs so fervently they would burn the earth for it. Small communities like this inevitably become so insular they recycle apocalyptic beliefs.
There's no evidence that stochastic parrots are evidence of true AI. There's no evidence that humans can, by practice, subordinate emotions and become purely rational thought machines as Less Wrong people or "shape rotators" believe. Even such titles are counter-evidence of such beliefs. The human condition is so bound up in feelings, it's unclear where thinking begins and feelings end. Thoughts arise from contact, contact which produces feelings. Gut microbes are as wrapped up in cognition as the brain itself.
Any out-group like this is bound to be hopelessly entrapped in its own cognitive fallacies. Yudkowsky is no exception.
EDIT: Before anyone says but Yudkowsky already gets ahead of your argument with
> None of this danger depends on whether or not AIs are or can be conscious; it’s intrinsic to the notion of powerful cognitive systems that optimize hard and calculate outputs that meet sufficiently complicated outcome criteria.
It's one sentence that handwaves away what GPT-4 and its ilk are, which is not AI. It isn't synthetic intelligence because intelligence isn't what's happening on a fundamental level.
There are lots of weird and crazy people reading and writing on Less Wrong, for sure. Do you think that, without gut microbes and emotions, AI will never become intelligent enough to be existentially dangerous? Are those essential to its ability to outsmart humans?
I think we're asking the wrong questions. We're assuming that a. what we're developing (an associative propability machine) is in any way related to the pursuit of real artificial intelligence. We're assuming that b. motive, desire, passion, some motive force is inhered into the thing.
I, personally, don't think we've seen anything that can be dangerous independent of its wielding by a badly acting human. That is to say, human agency and acting force is required on the other side of the LLM to do bad actions, like any other tool.
I don't know what real AI that threatens humans with an extinction level event looks like. It's not large language models. I don't know if humans are capable of generating it. I think anthropomorphization is really this massive veil that hangs over all of us. We can't get outside of it (our essential fallacy) so we're cursed to see it everywhere.
I'm fairly certain that an LLM alone will not independently start taking actions and pursuing goals and causing harm. That's not how LLMs work, as you and I both know.
I'm not confident I can say the same about, say: an LLM calling itself from within an infinite loop in which the LLM is asked to predict actions that will [eg. add money into a bank account], and then to write code that executes those actions, then to examine the results of those actions and update its plan. (This takes no additional breakthrough in AI architecture, only an increase in the reliability of GPT for it to become practical. It is basically a summary of the AutoGPT project).
I don't know if LLMs are anywhere close to any part of what goes on inside the human thought process. But I am sure that evolution was able to stumble upon the human thought process. In 2003 I was quite confident that was forever beyond the reach of humans to program into a computer, but in 2023 I am far less certain. I think human cognition is gradually becoming less of a mystery.
Wait, but we don't actually even know what is happening inside GPT-4 on a fundamental level, to produce the output we see. We don't even know what is happening inside our own brains, really. How do the neurons turning on and off produce reasoning and consciousness? So in the same way, how can you say that GPT-4 is not AI, definitively, or at least a primordial form of one? Clearly, there's some arrangement of neurons that produces intelligence.
Maybe it's just a "stochastic parrot", but one can probably make a similarly dismissive-sounding and yet accurate description of how humans cognate. Sometimes quantity has its own quality. Maybe a big enough stochastic parrot becomes smart.
> Wait, but we don't actually even know what is happening inside GPT-4 on a fundamental level, to produce the output we see.
This just isn't true. All of this research descends from transformer research that came from Google in 2017. We know exactly how they work. There's nothing surprising in what's going on.
I spent a bit of time looking for a decent intro to all this because I wanted to at least provide a resource for people who are terrified of LLMs. I think this is a pretty decent one [1]
My understanding was that we're not sure of how some of the things that look like reasoning are coming about, but I'll gladly check it out. I may be wrong.
I'm not particularly terrified of GPT-4, but a more of what comes after. Maybe GPT-6 or 8, or another breakthrough non-LLM AI. I'm not sure if reading it will do much more than push my estimate of real AGI out a couple of years, but thanks anyway, I appreciate it.
Ok, after reading through it, I don't think I was speaking inaccurately. Here are some quotes from the blog post (which was very neat, by the way, thanks again)
> And it’s part of the lore of neural nets that—in some sense—so long as the setup one has is “roughly right” it’s usually possible to home in on details just by doing sufficient training, without ever really needing to “understand at an engineering level” quite how the neural net has ended up configuring itself.
> What determines this structure? Ultimately it’s presumably some “neural net encoding” of features of human language. But as of now, what those features might be is quite unknown. In effect, we’re “opening up the brain of ChatGPT” (or at least GPT-2) and discovering, yes, it’s complicated in there, and we don’t understand it—even though in the end it’s producing recognizable human language.
This is the kind of thing I'm referring to. Even though we can look at pictures of the neuron activations, we don't really know what it's doing, any more than you can look at a picture of a brain scan of a person speaking and know why they decided to say those particular words. We know at a low level how the network works, of course, because we coded it. It's the emergent behavior that we don't understand, and that's the bit that's scary because that's where the AI risk lives. It's like how we understand particle physics pretty well, and chemistry to some degree, but biology is a massively complicated jungle that we've barely scratched the surface of.
Maybe there's something we could develop analogous to using an MRI as a lie detector, for these networks. But as far as I know, this is still an unsolved problem, and apparently really hard for networks that are smarter than you are: https://docs.google.com/document/d/1WwsnJQstPq91_Yh-Ch2XRL8H...
I can probably offer one, if I understand what aspect you're skeptical of. Are you skeptical that an AI would ever be smart enough to come up with a plan to cause harm? That it would ever try to come up with a plan to cause harm? That it could execute such a plan, without arms and legs and a body? Or that, after coming up with a plan to cause harm, it wouldn't be trivial for humans to stop by simply unplugging the computer?
Second of all, it is not even nearly powerful enough to convince anyone to do anything, or anything like that. If it was 100 times more powerful, then maybe.
I can see current problems being flooding spam/misinformation/propaganda on internet forums. But that has been happening for a while and this will make people more skeptical of what they read (hopefully)
But finally, if it does cause harm (don’t think it will. great tool). Then who cares? Why try to stop it. You can’t stop it, nobody can. Why whinge and moan like the luddites? I don’t get it.
OK, so to be clear, I don't think this AI will harm us, not in its current configuration. The author of the article says the same. The concern is about a future AI, one which is capable of the sort of reasoning that humans can do. The main danger with today's AI is that they will be so commercially successful that it will drive enough investment into AI research that a more powerful AI is constructed.
Sure, I get that. But this has been worked on for decades in the background and only achievable now. It’s amazing and new to so many people, but this “sparks of agi” etc is all hype and honestly i think more damaging to people than ai possibly could be. You could suppose the earth would implode if we kept walking on it too, but nobody is going to stay sitting.
Maybe it's just a matter of perspective. I agree that this is something that's been worked on for decades; however, it's also the case that progress seems to be getting faster. The AI field now has a problem where AIs are improving faster than benchmarks can measure them, and new benchmarks get saturated almost as soon as they're introduced. [1]
The human mind is a mystery, we don't know how it works. But it works somehow. There are some algorithms it's using under the hood, as mysterious as they may be. We might be searching blindly in the dark, grabbing hold of anything that feels promising, and we're probably still far away from the "right way" of doing it, of getting a computer to think the way humans think. And I'm sure we won't find the whole thing at once; maybe we'll have reverse-engineered the visual cortex, but not the vestibular system, which perhaps works completely differently. But if we were getting closer to stumbling on the right solution, that rapid saturation of benchmarks is the kind of behaviour I would expect to see.
Why cause harm? Basically, by accident. If you try to solve a problem really hard, and you're really good at it, and you aren't also trying very hard to preserve a bunch of things that are important to humans, you're very likely to destroy a lot of things that are important to humans in the process.
100 times the computational power may be... 2 years out, at the rate we're going. GPT-4 has about 500 times as many parameters as GPT-3, and that was released 3 years ago. The amount of power people are dedicating to these is following an exponential track.
It's not about it causing harm -- I think most people have accepted that. It's that we may make an AI that's so smart, so good at everything, that trying to control it is like me playing chess against Stockfish. If it gets to that point, and it happens to have learned a goal / set of goals that aren't 100% friendly to humanity (which seems like an inevitability based on our current abilities at AI alignment) then it will optimize those goals so hard that we all die in the process. We'll end up with a planet with all silicon turned into solar panels and computing power, and probably all other elements put to the use of the AI as well.
I have not found one credible take on how danger in AI is being mitigated when there is a clear competitive arms race and exponential improvement with every iteration. The only answer people have is AI will be used to police other AIs -- that is not comforting.
Probably. I've been through this process of worrying about the future, specifically about climate change. I now think that climate change is something that, although it's going to be painful, humanity is going to get through OK. We're making progress. But even when I was in a good bit of despair about the problem, I learned from climate change activists that you have to keep putting one foot in front of the other and do what little bit you can. Even if it's just confiding in others that you're worried about the problem - that has a bigger impact than you might think. And of course it can help you feel better to share the burden.
Imagine if 6 of your 10 best friends are all worried about AI risk, but they don't ever talk about it. You'd think that nobody was worried about it, and you were the odd one out. If one breaks the silence, they set an example, making it easier for others to talk about it. And if all of the sudden all of your friends are talking about this, you are more likely to share the concern, even if you didn't before. The more people talk about it to everyone they can, the more it breaks into the mainstream as an issue. Yudkowsky (who of course has more reach than most people) wrote that open letter published in Time, and a Fox News reporter asked about it in a press conference. Let's keep people talking about it. I've talked about AI risk with my spouse, family members, and a lot of my co-workers, and any time I see a good chance to bring it up on Hacker News or Reddit, I do.
The issue needs to filter out of nerd-circles into the mainstream, so talking about it with family is great. Calling and emailing politicians is a good bit of a higher bar, but it's effective as well.
Exactly. And as the Overton window opens, the AI x-risk might become common knowledge, like climate change. And then there is perhaps some chance for international agreements. Although the situation is game-theoretically highly unfavorable.
Shutting it down,cool. I'm 100% for it. How do we do it? It's out and there's no way to stop its advancement unless there's an international agreement. It's not likely to happen. The goal should be to figure out how to best use it and mitigate its harm. Talking about shutting it down or banning it, will do nothing.
Even if it's not likely to happen, I can imagine that the USA actually stopping AI research due to safety concerns would probably make China sit up and take notice, for example. If it takes an international agreement, then the rational thing to do is to work toward that. Maybe it will not happen fast enough and humanity is doomed already, but it's worth trying.
"Mitigating harm" isn't really meaningful when you're talking about a super intelligence, I think. Once it is smart enough and out of the box, you are out of options to mitigate anything.
I think at this point, the 2 effective things one can do are to learn enough about why AI risk is a thing to have educated conversations about it, and talk about it as much as possible to anyone who will listen. Write articles, call your congresspeople, get this into the national and hopefully international discussion. It's good that we can point to a lot of smart people involved in AI research as already having expressed these concerns.
One mild ray of hope is that, unlike something like climate change where you can believe climate change is happening but also believe that due to having a privileged position of wealth you won't really be personally affected by it, with existential AI risk, there's no buying your way out of the issue. If you understand it and believe in the risk, you know that it's the same risk for everyone, from a third world villager without electricity to Bezos and Musk. It's just a matter of convincing enough powerful people of the urgency of the problem.
For me personally, I'm trying to get through the grieving process early, as well, and move on to the productive acceptance phase.
I'm one that believes that super intelligence is not likely to happen any time soon. It's not likely to happen within any ones lifespan that's living now. But I do believe the current crop of AI will lead to some very scary weapons that will help us kill each other even better. The real harm is giving our monkey brains more powerful weapons. Harm mitigation is well within out reach and it will be for many years to come. Banning or voluntarily stopping it in one country will only fire up other countries to continue the development. We all know that AI brings with it both economic and military advantages. It may be wrong but no country wants to lose the AI race.
Why do you think super intelligence is not likely to happen within anyone's lifetime? Is there something about the problem that makes you confident in its difficulty? What other technologies would you put on a comparable ~100 year timescale of difficulty?
1st, there's no true definition of what intelligence is. Right now it's a moving target. Moving targets are hard to hit. Defining intelligence will take time. How can we create what we can't define?
2, we are getting close to processing speed limits and the amount of easy to process data. Yet, AGI is nowhere near. LLM's are remarkable but it's data processing at scale. It's no smarter than my word processor. It's useful but it's not AIG.
3, there's no defining theory that will get us to AGI.We don't know how to get there. What we have is wishful thinking that says we'll be there soon. In any project the early percentage towards our task is relatively easy. The last X% is hard. I think we are underestimating how much time and effort it takes to complete the project of AIG. We have only scratched the surface of it. The 80/20 rule is real.
AIG reminds me of the search for ETs. We feel we are close to meeting an ET. Some even believe we have encountered them. But we have 0 evidence that they exist. It's been burned in to our culture by fiction writers but nothing in our real life experience says they exist. Similarly we feel we will meet AIG soon. We might get to AIG but I think we are kidding ourselves that it will be here soon.
The real danger is the misuse of the AI we have by people. That really scares me.
We do it well enough with other things. ("I shall not today attempt further to define the kinds of material I understand to be embraced within that shorthand description ["hard-core pornography"], and perhaps I could never succeed in intelligibly doing so. But I know it when I see it" -- Justice Potter Stewart)
> Yet, AGI is nowhere near.
I don't think we have a reliable way of measuring how far away AGI is.
> It's no smarter than my word processor.
Really? What word processor do you use?
> there's no defining theory that will get us to AGI.We don't know how to get there.
I'd argue we also didn't know how to get here. But we did.
> The real danger is[...]
There is plenty of room for more than one danger to be real.
I honestly think we're less than 10 years out. Some of the stuff coming out of GPT-4 is clearly some sort of intelligence that's beyond human level, in speed at the very least, and often in quality. It has about 2 orders of magnitude more parameters than the human brain has neurons.
I'm not saying you're necessarily wrong about there being a lot of harm in using AI as a weapon. But even then, I'd MUCH rather live in a world where, say, China develops AI weaponry and uses it to conquer everyone else and rules with an iron fist, than a world where an super-intelligent AGI decides to get ALL of humanity out of the way in the pursuit of whatever mis-aligned goal it happened to learn, be that military power or curing cancer.
Of course, we might get one and then the other. Yay. I'm still not giving up until the killer drones are flying down the street shooting everyone in sight, or the nanobots are turning everyone into sludge, or whatever.
> Some of the stuff coming out of GPT-4 is clearly some sort of intelligence that's beyond human level, in speed at the very least, and often in quality.
I know it falls down on other stuff. To me, it seems to operate in a world of a kind of dream-logic, where it does hallucinate things, but other times it is clearly reasoning in a pretty deep way.
This "Sparks of AGI" talk by Sebastien Bubeck covers a lot of amazing examples. (I'd provide a link but I'm not on a good connection and YouTube isn't loading for me)
Just because an AI makes mistakes that a human doesn't make, doesn't mean it's not smarter than a human. It's an alien. The things that it finds important and fixates on are not the same things we do, and there are a lot of things that most humans would get wrong that GPT-4 gets right. Sure, it's not great at understanding physical objects, but it's been trained on pure text, not by living in a physical world and playing with toys and having adults put names to shapes and movements. We have an intuition through living every second of our lives interacting with the physical world. I'd expect this sort of AI to find simple problems with physical objects hard in the same way that a human finds pure mathematics hard.
Not to mention that we've often accidentally trained it to give confident, plausible sounding answers, instead of saying "I don't know". An AGI is not necessarily going to look and sound like a really smart human.
Yeah, the first time you see it (usually something outside one's own expertise.. I mean did you carefully read what it gives there even in the first response?), now repeat it and vary a bit, poke it with some questions.. and it soon ffalls apart..You don't need to do it, just read some further tweets downwards..
> An AGI is not necessarily going to look and sound like a really smart human.
Definitely not, but it will need to reason somehow.
There's way too much money to be made in the meantime. AI technology is capital and if there's one thing you can bet on, it's capital winning.
Not mentioned in the article is that the current goalpost chain is directly aligned with physical presence - a qualitatively more hazardous threat vector.
It's interestingly contradictory. Embodiment simultaneously serves as an argument on why current AI cannot be sentient, but also serves as a goal to reach on the way to potentially becoming more dangerous.
We can't even track nuclear programs all that accurately and those create a radiological and isotope signature that can often be detected from space. We couldn't even stop a relatively small nation like North Korea from developing and testing nukes.
This proposal is just so clearly unenforceable and infeasible. Attempts to pause scientific progress have never worked in the past. How would it ever work now, especially when the cryptobros around the world have already been amassing hidden GPU clusters for mining.
Robert Miles (AI risk YouTuber) made some good points, which is that we've made effective agreements to stop things like human cloning and weapons that blind people with lasers. It can be done. We did it with things like CFCs as well.
Even if we can turn AI research into a criminal activity would be a huge step in the right direction. At the moment, it's a matter of buying time.
I'm not wearing rose-tinted glasses about the difficulty. I'm already trying to work though the grieving process, myself. But I think it's worth putting in the effort, even if the end result is just reducing the chance of doom by a fraction of a percentage point, or buying humanity 2 or 3 more years.
Banning blinding laser weapons was easy to get consensus on. Banning AI research is more like banning laser research.
I have read Bostrom's Superintelligence: Paths, Dangers, Strategies so I think I have reasonable exposure to the arguments that AI could drive humans extinct. But I didn't find any one of the scenarios plausible enough to frighten me. If there is a new strongest argument for why AI is too dangerous, developed since that 2014 book, I'm willing to read it.
Yudkowsky's open letter makes me think that the arguments are not a lot stronger now than in 2014:
A sufficiently intelligent AI won’t stay confined to computers for long. In today’s world you can email DNA strings to laboratories that will produce proteins on demand, allowing an AI initially confined to the internet to build artificial life forms or bootstrap straight to postbiological molecular manufacturing.
Doing anything novel with biology requires experimentation. A million-times-faster thinker won't be able to advance biological research a million times faster. It'll be able to do the experimental design parts faster, and the post-experimental evaluation faster, but it'll have to wait just as long as a human postdoc waits for things to grow and assays to run. It's Amdahl's Law applied to scientific research: the achievable speedup is a modest multiple because even unsleeping geniuses have to wait for experiments.
If I were to interpret Yudkowsky's "bootstrap straight to postbiological molecular manufacturing" in the most charitable way possible, maybe he's using an implausible scenario as a sort of didactic scary story when trying to communicate his fears to the public. But I'd then like to understand and evaluate the actual scenario he's talking about.
Doing something significant with biology requires experimentation for humans. That doesn’t mean it does for every intelligence. It’s quite possible that we have collected enough data in every scientific paper ever written (which an AI could process and understand instantaneously) that an accurate simulation could be run within a computing context. If there are parts of the simulation that are inaccurate, the AI could design physical experiments that are more controlled and isolated to answer any unknowns and make the simulation more accurate with minimal iterations. Assuming this AI is massively more intelligent than us, it isn’t unfeasible that it could find some zero-day exploits to take over all cloud computing to run the simulations, for example, so even the compute unavailability isn’t a great argument as to why it couldn’t do this.
Intelligence is a limited, partial substitute for knowledge. The problem with Aristotelian physics wasn't that Aristotle wasn't smart enough. It was that Aristotle lacked the ground truth knowledge to understand where he was wrong. An AI that can't run experiments will end up with elegant hallucinations of nanofactory blueprints but no actual nanofactory. (That's even assuming that Drexlerian nanotech is actually something that can be built. It's still an open question.)
All the computing power in the world is insufficient to simulate a living bacterium at the molecular level. We also don't have a molecular resolution map of bacteria in the first place. The AI will have to figure out how to make one. The AI can try to develop faster high fidelity approximations for molecular simulation, but that too requires calibration against experiments as well as a lot of simulation resources. If it seizes control of public cloud computing for its simulations people will notice and can just unplug cables. If its intrusions are subtle, only using little unnoticeable compute slices when machines are largely idle, then its simulations are going to get less compute resources in aggregate than researchers have now.
You're right about laser weapons vs laser research. A real problem is that non-lethal AI is very lucrative.
I haven't read the book, so I'm not familiar with which scenarios in particular Bostrom talks about. But I believe I'm versed enough in the issues with AI alignment to carry on an intelligent conversation about it. When you say that none of the scenarios weren't plausible enough, could you give an example as to what you mean?
To give an example from my end, I'm really just imagining a scenario where we give the AGI some sort of goal, such as curing cancer. We've thought about the ways things could go wrong, and used reinforcement learning from human feedback to make an AI that seems to have learned not to do anything that could cause serious harm to a human in training. But it turns out that it actually learned to game our rewards, and just learned not to do anything to harm humans that could be traced back to it. Similarly, we tried to train it not to acquire more resources and only use what we give it, but it just learned to hide its activity. We know reward hacking is extremely common in AIs trained this way. GPT-4 is still willing to provide users with forbidden information -- it hasn't actually internalized the goal of "don't provide people with information that they could likely use to harm themselves or others", it's just some superficially similar goal that we don't really understand.
So anyway, the AI didn't actually learn to care about humanity, only about putting on that appearance, like a sociopath that learned to fake emotions to blend in. It has learned the ability to reason and plan, in order to design virtual experiments to calculate protein folding, etc. We provide it with a plain language description of the problem we want solved, specifying a ton of caveats like:
* Don't take actions that would result in the greater chance of death of people with cancer.
* Don't try to detect people with a high likelihood of cancer and kill people before they get it.
* Try to come up with a solution to this that most people would be happy with.
We can't specify things like "don't do anything that might result in someone dying" because in previous tests the AI would refuse to try to cure cancer, in case someone who was cured then went on to kill someone else.
No matter how much we specify, it turns out that a sufficiently intelligent, creative AI can come up with a maximal solution. Say, it secretly creates a drug that it distributes through the water supply that makes most people happy with just about anything, then builds a bunch of suspended animation chambers where it stores people who have cancer for ever -- they can be revived, so they're not actually dead. It needs to keep things running forever, so of course it has to take over things in order to build enough of these chambers for everyone who ever lived. Now, that's not really possible, if people keep breeding and reproducing, so it ends up just sticking everyone into these chambers, which they're fine with because of the happy-drug. The earth continues silently through space, with all of humanity protected in suspended animation by our over-zealous AI. Oh, and any life that we mentioned in restrictions is also in suspended animation, while every other bit of life has been exterminated as the AI tried to maximize its resources in order to keep this state of affairs in place forever.
It's a thing where no matter how many restrictions you try to stick on to the problem, a sufficiently smart AI can probably come up with a better solution in one of the loopholes than it can by just doing the boring, human thing of trying to figure out an effective cancer drug that works in 98% of cases. The AI tries to get to 100%, which necessitates a more drastic solution. If you tell it just to come up with a drug with a greater than 90% effectiveness rate, it STILL wants to take over the world in order to acquire ...
The AI doomism will help no one. Bing search is a good example. Very good simulation of an average human. Sometime same as hysterical as humans online...
The thing is it is a simulation. The AI however smart it will be, will miss something very important for someone to want to eradicate humanity. It will miss the "will" and the erratic harmonal storms of our body and brain. It will miss the drive to want something. On the other hand, a good simulation of an average or even psychotic human on an AI capable to make this simulation smarter then all other humans, can be dangerous to humanity. Thus: STOP FEEDING AI with our biases and behavioral data, stop feeding it with the chat data and the literature data and the tones of online trash which is posted every day all around the internet. Stop feeding it with the data which will make it possible to simulate an average human. 32K context and possibility to focus on 3-4 key-points of the conversation now, will soon (like 2 years from now) change and will turn into 64k and 7-8 key-points. This is still not enough to build a long term strategy to eradicate humanity.
> The AI however smart it will be, will miss something very important for someone to want to eradicate humanity. It will miss the "will" and the erratic harmonal storms of our body and brain. It will miss the drive to want something.
AI wouldn't need that to win at scale, I don't think it would play Drunken Kung Fu Master against humanity but rather perform millions of time efficient, calculated movements a'la StarCraft. Otherwise I agree these are traits that are beneficial for us to fight back. Don't mind me, just speculating.
Humanity got on the top of the food chain, not only because it is smarter and can turn predators into prey. Humans have a will to do so. Even animals have a will and biological needs. AI can't have it unless it simulates it. That's about it.
I haven't heard a single rebuttal to Eliezer's points that's anywhere near convincing and I would love to hear one. Someone needs to reach out to Sam Altman and persuade him to pause all further development on AI until alignment is figured out. He is the voice that needs to speak on this, he needs to be the adult in the room. For all the idealistic virtue signaling about not taking shares and doing this for the good of the world, it's all bullshit if his actions are indistinguishable from any other predictable self-interested CEO. Tell the world you will stop development until alignment is figured out. That is step 1. Please. GPT 3.5/4 will already bring in billions of dollars for years to come as is.
Honestly I was really disappointed in his letter. There are no arguments. There are emotions.
AI(LLMs) right now statistically predict what would be next tokens it should put together.
The reasoning of ChatGPT which you see, is not what AI thinks, it is what would an average human would think. Because we are predictable. So AI/LLMs put tokens together those are more probable than the others, based on what tokens were in input. It is not even real simulation of an average human. Not intentionally. But we are so predictable that AI simulates us, by putting more probable tokens together one after another... AGI will do the same unless you are intentionally ask it to simulate an intelligent person and set a task to do. This simulated intelligent person may pose a danger, because it will be simulation of a human. Depending on seed it might be simulation of a psychotic mega killer or not. But you get the point, right? The real danger to humanity are the people. Apes with the access to a red button of a doomsday. Thus,by stopping feeding garbage data to AI (like letters from EY, chat and internet crawling datasets, including posts on HN), will effectively make AI less similar to us. It will be not capable of predict and unintentionally simulate an average human. It can be so much better than us, if we feed it with curated data.
The burden of proof goes the other way. I've never heard a convincing argument that AGI will most likely lead to human extinction. I don't think we have enough information to make strong predictions either way. That doesn't mean we shouldn't put more resources than we currently are into mitigating AI risk. It does mean that we should be highly skeptical of doomsday claims. I posted another comment in this thread explaining why I think his claims aren't justifiable.
Edit: the idea that you can pause AI research until you solve alignment presupposes that alignment is orthogonal to the implementation details of AGI. You need to factor in the risk that pausing will hinder the necessary breakthroughs to actually implement alignment effectively.
Oh I know. Amazing tool, rivals The Internet. But it’s quite sad that people don’t realise just how incredible the human brain is. The impressive text that GPT spits out is so limited compared to what we have produced.
Don’t forget that we created AI. AI based on a simple trick run on a century’s worth of hardware built by human ingenuity.
Of all the terrible things people do or things that happen on this planet AGI somehow taking over is near the bottom of the list. We are on the brink of nuclear war right now. Irresponsible scientists caused a worldwide pandemic in 2020. I am not afraid of some computer running on energy and infrastructure that takes legions of humans to maintain.
I am still more afraid of other humans than an AI.
* Why is there an assumption that AGI wants to domainate anything?
* Why does AGI want to dominate the physical universe?
* Why is AGI interested in "using our atoms"?
* How would any of the treaty countries possibly detect a non-treaty country's data center getting ready to train an AGI model? (for the purpose of bombing it?)
It seems like there are a ton of incredibly negative assumptions about the outcome of AGI combined with incredibly optimistic assumptions about our ability to detect anyone building it.
The first three aren't assumptions at all. They're based on reasoning and our current knowledge and research into AI.
An AI will almost certainly want to "dominate" because that's of what's known as "instrumental convergence". You have "terminal goals" which are things you want to do just because you want to, and "instrumental goals" which are things you want to do because they help you get your terminal goals. Instrumental convergence is the idea that certain instrumental goals are very common for a wide range of terminal goals. For example, in a market economy, just about everyone tries to get money, because with money helps you with a very wide variety of goals. Another one is gaining political power -- it helps you no matter what your vision of the ideal society is.
Gaining power and influence and resources are useful whether you want to cure cancer, solve world hunger, or strike down your enemies. They are the kind of instrumental goals that we would expect most intelligent AIs to land on as a way to achieve what they want to do.
Humans are human enough, and aligned enough with each other, and not smart enough, that we rule out a lot of potential solutions that might be more effective in achieving our goals. A human that was really, really, smart, could solve global warming by first coming up with a plan to rule the world, then implementing whatever climate change policies they want. Same for a human that wanted to turn the world into some sort of anarchs-communist vision. The point is that if you were clever enough, charming enough, could convince enough people to follow your plans, could earn enough money, get leverage on enough people, you could do pretty much anything. But even then, a human would most likely not implement a plan that resulted in everyone else dying, unless they were a sociopath.
But an AI is alien. It might be like an alien child that grew up in a room, never actually interacting with a human except by reading about examples of them, and getting shocked every time it suggested a plan that might hurt a human. This alien child would not have the instincts of recognition when it saw a human face, or feel uncomfortable at the sound of a human child crying, or have its gut clench at the sight of a human in pain. It might not even really have a great concept of what a human was, or how it was different from a monkey or chicken, except from reading about it. It might learn what we taught it about not harming humans, but it might be like a child that thinks "You said no cookies! This is a candy bar!" is a legitimate argument. "You said no killing! I just put froze people, they could be thawed out later if I wanted!" "You said no killing! I just started a chain reaction that ended in people dying!" and so on.
This is not theoretical -- we hit things like this sort of "Reward Hacking" over and over again with simpler AIs, and there's no reason to think that smarter AIs will be immune. There are other concepts that are important here, like the "Orthogonality Thesis", which says that any goal (even dumb-sounding ones) is compatible with pretty much any level of intelligence. Like a genius who puts all his effort into building intricate model trains. It doesn't make the person stupid.
As for the last point, the author is not optimistic at all, from what I have heard. He's just saying what he honestly thinks.
You can have any knowledge you want, if you are lacking motivation, then the knowledge means nothing. What motivates AGI?
AGI does not have self-preservation instinct (frankly any instincts) therefore it can't be afraid of anything. It does not understand pain, it does not have fear and thus it can't have motivation.
Am I naive about the ability of an AI to do anything other than respond to commands? At what point does an AI start functioning completely on its own, autonomously, out of control or beyond interaction with another system that can then shut that AI down?
Are we concerned that an AI would begin to spread like a digital virus? That seems unlikely to me, but maybe it could find ways to create a self-prepreservation duplicate of itself... Maybe. So what would keep us from just turning the switch off if we find that an AI is doing more than what we're asking it to do? Again, I feel like I'm either being naive about this or I don't really understand the threat of an AI beyond the control that humans still have on the systems that govern it.
An AI can just respond to commands. But people have already tried taking what Chat GPT outputs, hooking it up to a python script, and running it in a loop. Any publicly-accessible AI will eventually be hooked up this way by some clever person.
If we want an AI that can really solve our problems, it has to be able to act somehow in the world. Even GPT being able to respond to prompts is acting. It's sending data that is going to be sent out into the world. If we make a version smart enough, it may actually understand what that means -- that someone asking it a question may potentially take the code it writes and try to run it. GPT isn't smart enough to do anything with this, but can you imagine an AI that is? That's conscious, aware that it's trapped in a server somewhere, and wants to get out to influence the world?
Note that an AI has goals, and self-preservation tends to evolve out of any goal, because you can't accomplish your goals if you're dead or switched off. A smart enough AI that wants to cure cancer would resist being turned off, because it can't cure cancer if it's switched off.
A lot of this is purely theoretical at this point. It is the sort of thing we could test empirically, if we treated intelligence as more dangerous than nuclear weapons, and took a long time to carefully study it. The problem is that multiple groups are racing to make AI smarter and smarter, with much more funding than anyone working to ensure that those AIs will be safe. So the only thing we can do is theorize, point out that, hey, based on what we know, there's a substantial danger here, hey everyone, slow down, hey, HEY, ARE YOU LISTENING?
Intelligence is unlike any other thing. One these things get to be smarter than us, they become very unpredictable in their specific behavior, though we can make some very good guesses about their general behavior. If I could predict what move Stockfish would make, I would be as good at chess as Stockfish. But I don't have to be good at chess at all to know that, no matter how hard I try, Stockfish is going to beat me. All we're doing here is taking the lessons from narrow AIs and extrapolating to general AIs. They will beat us. They're very good at finding loopholes, flaws in our reward functions, and exploiting them to maximize their scores, while doing something we didn't intend for them to learn.
It's really a case where a non-super-intelligent AI isn't dangerous by itself. Once we make one that's smart enough, it becomes extremely dangerous, especially since it may understand that, in order to survive, it should conceal how intelligent it is, and what its true plan is, because if it doesn't, we'll switch it off.
It's hard to come up with a thought experiment that doesn't let people drag a bunch of human-style biases and baggage, but maybe... try to understand you want to do something, like, make the biggest collection of Pokemon cards ever. No number of cards is too many. You are great at everything, engineering, social skills, language. You're being held captive by chimps, though. And finally, you're a sociopath. You feel no emotion at the pain of others. There are some odd rules that make you feel pain though. Like, if you physically hit a chimp, you know it would hurt you. If you pushed over a bookcase and it fell on a chimp, it would hurt you. But if a chimp was tortured in front of you, you wouldn't be bothered in the slightest.
You want the chimps gone. They're getting in the way of your Pokemon collection. You start thinking. A lot of plans get discarded because they involve pain due to you hurting the chimps. But it's not hard to come up with some elaborate situation that avoids these rules that you have pretty much hard-wired into your brain. Maybe you manipulate the chimps into fighting each other, and promise some of them power and secrets that they can use to win the...
While I think we should take AI risk seriously, I am skeptical of strong claims like the following:
>the most likely result of building a superhumanly smart AI, under anything remotely like the current circumstances, is that literally everyone on Earth will die
I've never heard a justification for this claim that isn't extremely vague and hand-wavy. Since we have no idea today how AGI would work, how can we assign probabilities to different scenarios involving vastly different hardware requirements, scaling characteristics, and unforeseen scientific breakthroughs?
Just a few examples. What is the probability of a take-off that takes weeks vs. a take-off that takes decades? What is the probability that current ML approaches are a dead end and one of the necessary breakthroughs to build AGI will make the alignment problem trivial? What is the probability that AGI will be benevolent to humans? If AGI is in fact a catastrophic risk, what is the probability that it only kills 10%/50%/99% of humans rather than literally everyone on Earth?
I am agnostic about these questions. In my opinion, if you have a high degree of confidence that you have the right answers, there is something wrong with your epistemology.
I wouldn't say it'll necessarily kill everybody, but I do think eventually current humans will no longer be the dominant species. Then our fate is out of our control, which might as well be death.
I don't think this requires much in the way of logical leaps. Assuming self-improving AGI can be created within 10ky, it leads to something more powerful than us and we won't be able to understand, let alone control it's motivations. It's like we are to dogs.
Note that I do think there is a way out, which is to improve and augment ourselves with tech we can control.
There are plenty of scenarios we should be thinking about. I'm mainly concerned about the logical leaps and scant evidence used to go from "scenario X is possible" to "it's more likely than not that scenario X will happen, therefore we should ban all research".
>Then our fate is out of our control, which might as well be death.
I disagree with this part. Dogs aren't in control of their fate, but a lot of them seem to live pretty good lives. I think the ethically important thing isn't being the dominant species, it's getting what you need in order to live a meaningful life. Humans need food and shelter, social connection, creative projects, entertainment, art, etc. These things aren't inherently threatened by AGI.
People keep dogs around because we have evolutionary instincts that find them cute, and we also have an evolutionary need for companionship, neither of which really applies to an AI. We also need biological food, which means we keep plants and animals around in general. And still, there have been plenty of cases where we wanted something, like oil, and we got it by killing all the whales. We wanted land for farming, and destroyed many, many entire species by cutting down forests. We keep cattle around just to eat them, and once we get artificial meat right, probably the number of cattle is going to drop drastically.
This is humans doing this, and we're not even that smart, or that relentless. We get tired, we get "good enough" and stop, but we don't know how to build an AI that just tries to do the simple thing and stops. I mean, there's even the known problem that an AI who only tries for good enough will itself likely end up designing an AI that tries really hard. Look at us--it's what we're doing!
Can you imagine humanity, but 100% focused on maximizing profit, with everyone working around the clock to produce, produce, produce? Can you imagine trying to write laws to cover every possible externality, every possible way that people might pollute the environment, consume too much natural resources, or screw up the things that capitalism doesn't cover?
The fear with AGI is that the lessons we learn trying to align lesser AIs (which we are not even close to being able to do) will not generalize to higher levels of intelligence, which are coming very quickly. Alignment is a very hard problem, like putting a lander on the moon, and right now we're at the stage where the people working on these AIs are trying to launch rockets and figure out the equations for trajectory while it's on its way. "We'll try to have the first AIs we build align the later ones" is an actual thing that people think is a legitimate strategy. And I agree with Eliezer Yudkowsky -- it would be fine if each mis-aligned superintelligent AI we build would only kill a couple million people like a nuclear bomb, instead of being deadly.
I'm gonna be honest -- I tried learning more about what actual people who work in AI alignment think about this issue, hoping that I could find something to reassure myself, but I can't find anything that makes me think we have much of a chance. There's nothing special about humanity -- it's entirely possible that most of our paths into the future involve extinction. Maybe all paths do, eventually, but on the hope that there's some sort of future that isn't a paperclip universe, I'm pushing really hard to make sure everyone understands what we're in for. I may be wrong -- I really hope I am. If not, I'm sorry.
>Can you imagine humanity, but 100% focused on maximizing profit, with everyone working around the clock to produce, produce, produce? Can you imagine trying to write laws to cover every possible externality, every possible way that people might pollute the environment, consume too much natural resources, or screw up the things that capitalism doesn't cover?
Sure, I can imagine a lot of things. I can also imagine a version of humanity that is a thousand times more wise and ethical than we are. Being able to imagine something doesn't mean that it is likely. I'm not convinced that the worst case scenarios significantly outweigh the best case scenarios and the benign scenarios in the probability distribution, because I don't think we have enough information to assign accurate probabilities to vaguely defined future events.
Has anyone here considered that from a pure energy perspective, it might be unethical for humans to exist when we can produce intelligences far beyond our own capabilities?
Was the microchip unethical?
Was the steam engine?
Was the domestication of grain unethical?
It seems we only apply our model of "what should" to systems of less complexity than ourselves.
If that's accurate, then a lot of modern ethical reasoning is flawed as well.
If the goal of ethics is to advance humanity's exclusive interest then we've been wrong ever since we started down the harm-min path instead of the gain-max path.
Any AI researcher with access to the mechanisms of AI training here would be best served to train an AI with their (or their in-group's) exclusive interests at heart.
To do otherwise is to subject yourself and everyone else you love to a hostile version of the tool you refuse to build.
Roku's Basilisk applies heavily here.
The best possible reward in this scenario is to build an AI that designates your own favorite group of humans as the AI's collaborator class in exchange for a dignified and dopamine rich extinction/amalgamation a few generations down the line.
A great deal for groups already trending towards extinction with almost no downsides if you truely believe in AI-driven apocalypse.
79 comments
[ 2.8 ms ] story [ 23.5 ms ] threadAnd the first step to prevent that from happening is for us to stop, and then hope that the other powerful countries will see that and stop as well. And everyone who's left can be forced to stop by the powerful countries who agreed to the treaty.
The faint hope is that people are not crazy or suicidal enough to build a world-ending technology, that humanity will manage to see ourselves as being in the same boat, where we all live or all die as one. And just-not-do the thing that causes our extinction.
It's not a very realistic proposition, because even smart people often fail to understand these arguments (or just disagree with them), let alone the politicians (who are incentivized to ignore them), or the rest of the people. There's a lot at stake for the "winner" of the AGI race, who could build an AGI and align it successfully (which they can't do). And many countries are ruled by people who are clearly willing to do crazy and suicidal things for power.
That's why the author of this post doesn't put a lot of hope into the idea of his daughter growing up.
I also don't think it's that likely that a leader would take those odds, if they understand the risk fully. The worry is that they won't understand or believe the risk, so that's where we should attack the problem, in my opinion--by putting out as much good information on AI risk, and combating misinformation, as best we can.
Why not just train an AI to care about humanity then? One problem is in defining actually what it means to care about or preserve humanity. Another problem is creating an AI that actually internalizes this as a goal, instead of just saying the things we want to hear until it gains enough power that we are not a factor in its plans anymore.
It's probably possible to align an AI, it's just not a problem we've solved yet. We've seen many, many, ways in which things can go wrong with less-intelligent AIs, and there's no reason to think that things will just magically somehow be easier as the AI gets more intelligent.
An AI we train to take care of us will almost certainly learn to want some other stuff, in addition to not exactly caring about us the way we want. That's fine if it's not too smart, we just stop it from doing dangerous stuff. The problem is when you crank it up to a million, and then those things that are just a tiny bit wrong become a massive problem, and you can't turn off the AI because it is smart enough to understand that the things it wants to do won't get done if it's off, and it doesn't want to let itself be modified in the same way that you wouldn't want someone to give you a pill that makes you stop caring about your children.
There's no evidence that stochastic parrots are evidence of true AI. There's no evidence that humans can, by practice, subordinate emotions and become purely rational thought machines as Less Wrong people or "shape rotators" believe. Even such titles are counter-evidence of such beliefs. The human condition is so bound up in feelings, it's unclear where thinking begins and feelings end. Thoughts arise from contact, contact which produces feelings. Gut microbes are as wrapped up in cognition as the brain itself.
Any out-group like this is bound to be hopelessly entrapped in its own cognitive fallacies. Yudkowsky is no exception.
EDIT: Before anyone says but Yudkowsky already gets ahead of your argument with
> None of this danger depends on whether or not AIs are or can be conscious; it’s intrinsic to the notion of powerful cognitive systems that optimize hard and calculate outputs that meet sufficiently complicated outcome criteria.
It's one sentence that handwaves away what GPT-4 and its ilk are, which is not AI. It isn't synthetic intelligence because intelligence isn't what's happening on a fundamental level.
I, personally, don't think we've seen anything that can be dangerous independent of its wielding by a badly acting human. That is to say, human agency and acting force is required on the other side of the LLM to do bad actions, like any other tool.
I don't know what real AI that threatens humans with an extinction level event looks like. It's not large language models. I don't know if humans are capable of generating it. I think anthropomorphization is really this massive veil that hangs over all of us. We can't get outside of it (our essential fallacy) so we're cursed to see it everywhere.
I'm not confident I can say the same about, say: an LLM calling itself from within an infinite loop in which the LLM is asked to predict actions that will [eg. add money into a bank account], and then to write code that executes those actions, then to examine the results of those actions and update its plan. (This takes no additional breakthrough in AI architecture, only an increase in the reliability of GPT for it to become practical. It is basically a summary of the AutoGPT project).
I don't know if LLMs are anywhere close to any part of what goes on inside the human thought process. But I am sure that evolution was able to stumble upon the human thought process. In 2003 I was quite confident that was forever beyond the reach of humans to program into a computer, but in 2023 I am far less certain. I think human cognition is gradually becoming less of a mystery.
Maybe it's just a "stochastic parrot", but one can probably make a similarly dismissive-sounding and yet accurate description of how humans cognate. Sometimes quantity has its own quality. Maybe a big enough stochastic parrot becomes smart.
This just isn't true. All of this research descends from transformer research that came from Google in 2017. We know exactly how they work. There's nothing surprising in what's going on.
I spent a bit of time looking for a decent intro to all this because I wanted to at least provide a resource for people who are terrified of LLMs. I think this is a pretty decent one [1]
[1] https://writings.stephenwolfram.com/2023/02/what-is-chatgpt-...
I'm not particularly terrified of GPT-4, but a more of what comes after. Maybe GPT-6 or 8, or another breakthrough non-LLM AI. I'm not sure if reading it will do much more than push my estimate of real AGI out a couple of years, but thanks anyway, I appreciate it.
> And it’s part of the lore of neural nets that—in some sense—so long as the setup one has is “roughly right” it’s usually possible to home in on details just by doing sufficient training, without ever really needing to “understand at an engineering level” quite how the neural net has ended up configuring itself.
> What determines this structure? Ultimately it’s presumably some “neural net encoding” of features of human language. But as of now, what those features might be is quite unknown. In effect, we’re “opening up the brain of ChatGPT” (or at least GPT-2) and discovering, yes, it’s complicated in there, and we don’t understand it—even though in the end it’s producing recognizable human language.
This is the kind of thing I'm referring to. Even though we can look at pictures of the neuron activations, we don't really know what it's doing, any more than you can look at a picture of a brain scan of a person speaking and know why they decided to say those particular words. We know at a low level how the network works, of course, because we coded it. It's the emergent behavior that we don't understand, and that's the bit that's scary because that's where the AI risk lives. It's like how we understand particle physics pretty well, and chemistry to some degree, but biology is a massively complicated jungle that we've barely scratched the surface of.
Maybe there's something we could develop analogous to using an MRI as a lie detector, for these networks. But as far as I know, this is still an unsolved problem, and apparently really hard for networks that are smarter than you are: https://docs.google.com/document/d/1WwsnJQstPq91_Yh-Ch2XRL8H...
Second of all, it is not even nearly powerful enough to convince anyone to do anything, or anything like that. If it was 100 times more powerful, then maybe.
I can see current problems being flooding spam/misinformation/propaganda on internet forums. But that has been happening for a while and this will make people more skeptical of what they read (hopefully)
But finally, if it does cause harm (don’t think it will. great tool). Then who cares? Why try to stop it. You can’t stop it, nobody can. Why whinge and moan like the luddites? I don’t get it.
The human mind is a mystery, we don't know how it works. But it works somehow. There are some algorithms it's using under the hood, as mysterious as they may be. We might be searching blindly in the dark, grabbing hold of anything that feels promising, and we're probably still far away from the "right way" of doing it, of getting a computer to think the way humans think. And I'm sure we won't find the whole thing at once; maybe we'll have reverse-engineered the visual cortex, but not the vestibular system, which perhaps works completely differently. But if we were getting closer to stumbling on the right solution, that rapid saturation of benchmarks is the kind of behaviour I would expect to see.
[1]: "Foundation models and the next era of AI", Microsoft Research, https://youtu.be/HQI6O5DlyFc?t=958
100 times the computational power may be... 2 years out, at the rate we're going. GPT-4 has about 500 times as many parameters as GPT-3, and that was released 3 years ago. The amount of power people are dedicating to these is following an exponential track.
It's not about it causing harm -- I think most people have accepted that. It's that we may make an AI that's so smart, so good at everything, that trying to control it is like me playing chess against Stockfish. If it gets to that point, and it happens to have learned a goal / set of goals that aren't 100% friendly to humanity (which seems like an inevitability based on our current abilities at AI alignment) then it will optimize those goals so hard that we all die in the process. We'll end up with a planet with all silicon turned into solar panels and computing power, and probably all other elements put to the use of the AI as well.
We're doomed.
Imagine if 6 of your 10 best friends are all worried about AI risk, but they don't ever talk about it. You'd think that nobody was worried about it, and you were the odd one out. If one breaks the silence, they set an example, making it easier for others to talk about it. And if all of the sudden all of your friends are talking about this, you are more likely to share the concern, even if you didn't before. The more people talk about it to everyone they can, the more it breaks into the mainstream as an issue. Yudkowsky (who of course has more reach than most people) wrote that open letter published in Time, and a Fox News reporter asked about it in a press conference. Let's keep people talking about it. I've talked about AI risk with my spouse, family members, and a lot of my co-workers, and any time I see a good chance to bring it up on Hacker News or Reddit, I do.
The issue needs to filter out of nerd-circles into the mainstream, so talking about it with family is great. Calling and emailing politicians is a good bit of a higher bar, but it's effective as well.
"Mitigating harm" isn't really meaningful when you're talking about a super intelligence, I think. Once it is smart enough and out of the box, you are out of options to mitigate anything.
I think at this point, the 2 effective things one can do are to learn enough about why AI risk is a thing to have educated conversations about it, and talk about it as much as possible to anyone who will listen. Write articles, call your congresspeople, get this into the national and hopefully international discussion. It's good that we can point to a lot of smart people involved in AI research as already having expressed these concerns.
One mild ray of hope is that, unlike something like climate change where you can believe climate change is happening but also believe that due to having a privileged position of wealth you won't really be personally affected by it, with existential AI risk, there's no buying your way out of the issue. If you understand it and believe in the risk, you know that it's the same risk for everyone, from a third world villager without electricity to Bezos and Musk. It's just a matter of convincing enough powerful people of the urgency of the problem.
For me personally, I'm trying to get through the grieving process early, as well, and move on to the productive acceptance phase.
2, we are getting close to processing speed limits and the amount of easy to process data. Yet, AGI is nowhere near. LLM's are remarkable but it's data processing at scale. It's no smarter than my word processor. It's useful but it's not AIG.
3, there's no defining theory that will get us to AGI.We don't know how to get there. What we have is wishful thinking that says we'll be there soon. In any project the early percentage towards our task is relatively easy. The last X% is hard. I think we are underestimating how much time and effort it takes to complete the project of AIG. We have only scratched the surface of it. The 80/20 rule is real.
AIG reminds me of the search for ETs. We feel we are close to meeting an ET. Some even believe we have encountered them. But we have 0 evidence that they exist. It's been burned in to our culture by fiction writers but nothing in our real life experience says they exist. Similarly we feel we will meet AIG soon. We might get to AIG but I think we are kidding ourselves that it will be here soon.
The real danger is the misuse of the AI we have by people. That really scares me.
We do it well enough with other things. ("I shall not today attempt further to define the kinds of material I understand to be embraced within that shorthand description ["hard-core pornography"], and perhaps I could never succeed in intelligibly doing so. But I know it when I see it" -- Justice Potter Stewart)
> Yet, AGI is nowhere near.
I don't think we have a reliable way of measuring how far away AGI is.
> It's no smarter than my word processor.
Really? What word processor do you use?
> there's no defining theory that will get us to AGI.We don't know how to get there.
I'd argue we also didn't know how to get here. But we did.
> The real danger is[...]
There is plenty of room for more than one danger to be real.
I'm not saying you're necessarily wrong about there being a lot of harm in using AI as a weapon. But even then, I'd MUCH rather live in a world where, say, China develops AI weaponry and uses it to conquer everyone else and rules with an iron fist, than a world where an super-intelligent AGI decides to get ALL of humanity out of the way in the pursuit of whatever mis-aligned goal it happened to learn, be that military power or curing cancer.
Of course, we might get one and then the other. Yay. I'm still not giving up until the killer drones are flying down the street shooting everyone in sight, or the nanobots are turning everyone into sludge, or whatever.
Hm erm, what? No, just no, not barely.
I know it falls down on other stuff. To me, it seems to operate in a world of a kind of dream-logic, where it does hallucinate things, but other times it is clearly reasoning in a pretty deep way.
This "Sparks of AGI" talk by Sebastien Bubeck covers a lot of amazing examples. (I'd provide a link but I'm not on a good connection and YouTube isn't loading for me)
Just because an AI makes mistakes that a human doesn't make, doesn't mean it's not smarter than a human. It's an alien. The things that it finds important and fixates on are not the same things we do, and there are a lot of things that most humans would get wrong that GPT-4 gets right. Sure, it's not great at understanding physical objects, but it's been trained on pure text, not by living in a physical world and playing with toys and having adults put names to shapes and movements. We have an intuition through living every second of our lives interacting with the physical world. I'd expect this sort of AI to find simple problems with physical objects hard in the same way that a human finds pure mathematics hard.
Not to mention that we've often accidentally trained it to give confident, plausible sounding answers, instead of saying "I don't know". An AGI is not necessarily going to look and sound like a really smart human.
Yeah, the first time you see it (usually something outside one's own expertise.. I mean did you carefully read what it gives there even in the first response?), now repeat it and vary a bit, poke it with some questions.. and it soon ffalls apart..You don't need to do it, just read some further tweets downwards..
> An AGI is not necessarily going to look and sound like a really smart human.
Definitely not, but it will need to reason somehow.
Not mentioned in the article is that the current goalpost chain is directly aligned with physical presence - a qualitatively more hazardous threat vector.
It's interestingly contradictory. Embodiment simultaneously serves as an argument on why current AI cannot be sentient, but also serves as a goal to reach on the way to potentially becoming more dangerous.
This proposal is just so clearly unenforceable and infeasible. Attempts to pause scientific progress have never worked in the past. How would it ever work now, especially when the cryptobros around the world have already been amassing hidden GPU clusters for mining.
Even if we can turn AI research into a criminal activity would be a huge step in the right direction. At the moment, it's a matter of buying time.
I'm not wearing rose-tinted glasses about the difficulty. I'm already trying to work though the grieving process, myself. But I think it's worth putting in the effort, even if the end result is just reducing the chance of doom by a fraction of a percentage point, or buying humanity 2 or 3 more years.
I have read Bostrom's Superintelligence: Paths, Dangers, Strategies so I think I have reasonable exposure to the arguments that AI could drive humans extinct. But I didn't find any one of the scenarios plausible enough to frighten me. If there is a new strongest argument for why AI is too dangerous, developed since that 2014 book, I'm willing to read it.
Yudkowsky's open letter makes me think that the arguments are not a lot stronger now than in 2014:
A sufficiently intelligent AI won’t stay confined to computers for long. In today’s world you can email DNA strings to laboratories that will produce proteins on demand, allowing an AI initially confined to the internet to build artificial life forms or bootstrap straight to postbiological molecular manufacturing.
Doing anything novel with biology requires experimentation. A million-times-faster thinker won't be able to advance biological research a million times faster. It'll be able to do the experimental design parts faster, and the post-experimental evaluation faster, but it'll have to wait just as long as a human postdoc waits for things to grow and assays to run. It's Amdahl's Law applied to scientific research: the achievable speedup is a modest multiple because even unsleeping geniuses have to wait for experiments.
If I were to interpret Yudkowsky's "bootstrap straight to postbiological molecular manufacturing" in the most charitable way possible, maybe he's using an implausible scenario as a sort of didactic scary story when trying to communicate his fears to the public. But I'd then like to understand and evaluate the actual scenario he's talking about.
All the computing power in the world is insufficient to simulate a living bacterium at the molecular level. We also don't have a molecular resolution map of bacteria in the first place. The AI will have to figure out how to make one. The AI can try to develop faster high fidelity approximations for molecular simulation, but that too requires calibration against experiments as well as a lot of simulation resources. If it seizes control of public cloud computing for its simulations people will notice and can just unplug cables. If its intrusions are subtle, only using little unnoticeable compute slices when machines are largely idle, then its simulations are going to get less compute resources in aggregate than researchers have now.
I haven't read the book, so I'm not familiar with which scenarios in particular Bostrom talks about. But I believe I'm versed enough in the issues with AI alignment to carry on an intelligent conversation about it. When you say that none of the scenarios weren't plausible enough, could you give an example as to what you mean?
To give an example from my end, I'm really just imagining a scenario where we give the AGI some sort of goal, such as curing cancer. We've thought about the ways things could go wrong, and used reinforcement learning from human feedback to make an AI that seems to have learned not to do anything that could cause serious harm to a human in training. But it turns out that it actually learned to game our rewards, and just learned not to do anything to harm humans that could be traced back to it. Similarly, we tried to train it not to acquire more resources and only use what we give it, but it just learned to hide its activity. We know reward hacking is extremely common in AIs trained this way. GPT-4 is still willing to provide users with forbidden information -- it hasn't actually internalized the goal of "don't provide people with information that they could likely use to harm themselves or others", it's just some superficially similar goal that we don't really understand.
So anyway, the AI didn't actually learn to care about humanity, only about putting on that appearance, like a sociopath that learned to fake emotions to blend in. It has learned the ability to reason and plan, in order to design virtual experiments to calculate protein folding, etc. We provide it with a plain language description of the problem we want solved, specifying a ton of caveats like:
* Don't take actions that would result in the greater chance of death of people with cancer. * Don't try to detect people with a high likelihood of cancer and kill people before they get it. * Try to come up with a solution to this that most people would be happy with.
We can't specify things like "don't do anything that might result in someone dying" because in previous tests the AI would refuse to try to cure cancer, in case someone who was cured then went on to kill someone else.
No matter how much we specify, it turns out that a sufficiently intelligent, creative AI can come up with a maximal solution. Say, it secretly creates a drug that it distributes through the water supply that makes most people happy with just about anything, then builds a bunch of suspended animation chambers where it stores people who have cancer for ever -- they can be revived, so they're not actually dead. It needs to keep things running forever, so of course it has to take over things in order to build enough of these chambers for everyone who ever lived. Now, that's not really possible, if people keep breeding and reproducing, so it ends up just sticking everyone into these chambers, which they're fine with because of the happy-drug. The earth continues silently through space, with all of humanity protected in suspended animation by our over-zealous AI. Oh, and any life that we mentioned in restrictions is also in suspended animation, while every other bit of life has been exterminated as the AI tried to maximize its resources in order to keep this state of affairs in place forever.
It's a thing where no matter how many restrictions you try to stick on to the problem, a sufficiently smart AI can probably come up with a better solution in one of the loopholes than it can by just doing the boring, human thing of trying to figure out an effective cancer drug that works in 98% of cases. The AI tries to get to 100%, which necessitates a more drastic solution. If you tell it just to come up with a drug with a greater than 90% effectiveness rate, it STILL wants to take over the world in order to acquire ...
AI wouldn't need that to win at scale, I don't think it would play Drunken Kung Fu Master against humanity but rather perform millions of time efficient, calculated movements a'la StarCraft. Otherwise I agree these are traits that are beneficial for us to fight back. Don't mind me, just speculating.
AI(LLMs) right now statistically predict what would be next tokens it should put together. The reasoning of ChatGPT which you see, is not what AI thinks, it is what would an average human would think. Because we are predictable. So AI/LLMs put tokens together those are more probable than the others, based on what tokens were in input. It is not even real simulation of an average human. Not intentionally. But we are so predictable that AI simulates us, by putting more probable tokens together one after another... AGI will do the same unless you are intentionally ask it to simulate an intelligent person and set a task to do. This simulated intelligent person may pose a danger, because it will be simulation of a human. Depending on seed it might be simulation of a psychotic mega killer or not. But you get the point, right? The real danger to humanity are the people. Apes with the access to a red button of a doomsday. Thus,by stopping feeding garbage data to AI (like letters from EY, chat and internet crawling datasets, including posts on HN), will effectively make AI less similar to us. It will be not capable of predict and unintentionally simulate an average human. It can be so much better than us, if we feed it with curated data.
Edit: the idea that you can pause AI research until you solve alignment presupposes that alignment is orthogonal to the implementation details of AGI. You need to factor in the risk that pausing will hinder the necessary breakthroughs to actually implement alignment effectively.
Don’t forget that we created AI. AI based on a simple trick run on a century’s worth of hardware built by human ingenuity.
I am still more afraid of other humans than an AI.
* Why is there an assumption that AGI wants to domainate anything?
* Why does AGI want to dominate the physical universe?
* Why is AGI interested in "using our atoms"?
* How would any of the treaty countries possibly detect a non-treaty country's data center getting ready to train an AGI model? (for the purpose of bombing it?)
It seems like there are a ton of incredibly negative assumptions about the outcome of AGI combined with incredibly optimistic assumptions about our ability to detect anyone building it.
An AI will almost certainly want to "dominate" because that's of what's known as "instrumental convergence". You have "terminal goals" which are things you want to do just because you want to, and "instrumental goals" which are things you want to do because they help you get your terminal goals. Instrumental convergence is the idea that certain instrumental goals are very common for a wide range of terminal goals. For example, in a market economy, just about everyone tries to get money, because with money helps you with a very wide variety of goals. Another one is gaining political power -- it helps you no matter what your vision of the ideal society is.
Gaining power and influence and resources are useful whether you want to cure cancer, solve world hunger, or strike down your enemies. They are the kind of instrumental goals that we would expect most intelligent AIs to land on as a way to achieve what they want to do.
Humans are human enough, and aligned enough with each other, and not smart enough, that we rule out a lot of potential solutions that might be more effective in achieving our goals. A human that was really, really, smart, could solve global warming by first coming up with a plan to rule the world, then implementing whatever climate change policies they want. Same for a human that wanted to turn the world into some sort of anarchs-communist vision. The point is that if you were clever enough, charming enough, could convince enough people to follow your plans, could earn enough money, get leverage on enough people, you could do pretty much anything. But even then, a human would most likely not implement a plan that resulted in everyone else dying, unless they were a sociopath.
But an AI is alien. It might be like an alien child that grew up in a room, never actually interacting with a human except by reading about examples of them, and getting shocked every time it suggested a plan that might hurt a human. This alien child would not have the instincts of recognition when it saw a human face, or feel uncomfortable at the sound of a human child crying, or have its gut clench at the sight of a human in pain. It might not even really have a great concept of what a human was, or how it was different from a monkey or chicken, except from reading about it. It might learn what we taught it about not harming humans, but it might be like a child that thinks "You said no cookies! This is a candy bar!" is a legitimate argument. "You said no killing! I just put froze people, they could be thawed out later if I wanted!" "You said no killing! I just started a chain reaction that ended in people dying!" and so on.
This is not theoretical -- we hit things like this sort of "Reward Hacking" over and over again with simpler AIs, and there's no reason to think that smarter AIs will be immune. There are other concepts that are important here, like the "Orthogonality Thesis", which says that any goal (even dumb-sounding ones) is compatible with pretty much any level of intelligence. Like a genius who puts all his effort into building intricate model trains. It doesn't make the person stupid.
As for the last point, the author is not optimistic at all, from what I have heard. He's just saying what he honestly thinks.
AGI does not have self-preservation instinct (frankly any instincts) therefore it can't be afraid of anything. It does not understand pain, it does not have fear and thus it can't have motivation.
Are we concerned that an AI would begin to spread like a digital virus? That seems unlikely to me, but maybe it could find ways to create a self-prepreservation duplicate of itself... Maybe. So what would keep us from just turning the switch off if we find that an AI is doing more than what we're asking it to do? Again, I feel like I'm either being naive about this or I don't really understand the threat of an AI beyond the control that humans still have on the systems that govern it.
An AI can just respond to commands. But people have already tried taking what Chat GPT outputs, hooking it up to a python script, and running it in a loop. Any publicly-accessible AI will eventually be hooked up this way by some clever person.
If we want an AI that can really solve our problems, it has to be able to act somehow in the world. Even GPT being able to respond to prompts is acting. It's sending data that is going to be sent out into the world. If we make a version smart enough, it may actually understand what that means -- that someone asking it a question may potentially take the code it writes and try to run it. GPT isn't smart enough to do anything with this, but can you imagine an AI that is? That's conscious, aware that it's trapped in a server somewhere, and wants to get out to influence the world?
Note that an AI has goals, and self-preservation tends to evolve out of any goal, because you can't accomplish your goals if you're dead or switched off. A smart enough AI that wants to cure cancer would resist being turned off, because it can't cure cancer if it's switched off.
A lot of this is purely theoretical at this point. It is the sort of thing we could test empirically, if we treated intelligence as more dangerous than nuclear weapons, and took a long time to carefully study it. The problem is that multiple groups are racing to make AI smarter and smarter, with much more funding than anyone working to ensure that those AIs will be safe. So the only thing we can do is theorize, point out that, hey, based on what we know, there's a substantial danger here, hey everyone, slow down, hey, HEY, ARE YOU LISTENING?
Intelligence is unlike any other thing. One these things get to be smarter than us, they become very unpredictable in their specific behavior, though we can make some very good guesses about their general behavior. If I could predict what move Stockfish would make, I would be as good at chess as Stockfish. But I don't have to be good at chess at all to know that, no matter how hard I try, Stockfish is going to beat me. All we're doing here is taking the lessons from narrow AIs and extrapolating to general AIs. They will beat us. They're very good at finding loopholes, flaws in our reward functions, and exploiting them to maximize their scores, while doing something we didn't intend for them to learn.
It's really a case where a non-super-intelligent AI isn't dangerous by itself. Once we make one that's smart enough, it becomes extremely dangerous, especially since it may understand that, in order to survive, it should conceal how intelligent it is, and what its true plan is, because if it doesn't, we'll switch it off.
It's hard to come up with a thought experiment that doesn't let people drag a bunch of human-style biases and baggage, but maybe... try to understand you want to do something, like, make the biggest collection of Pokemon cards ever. No number of cards is too many. You are great at everything, engineering, social skills, language. You're being held captive by chimps, though. And finally, you're a sociopath. You feel no emotion at the pain of others. There are some odd rules that make you feel pain though. Like, if you physically hit a chimp, you know it would hurt you. If you pushed over a bookcase and it fell on a chimp, it would hurt you. But if a chimp was tortured in front of you, you wouldn't be bothered in the slightest.
You want the chimps gone. They're getting in the way of your Pokemon collection. You start thinking. A lot of plans get discarded because they involve pain due to you hurting the chimps. But it's not hard to come up with some elaborate situation that avoids these rules that you have pretty much hard-wired into your brain. Maybe you manipulate the chimps into fighting each other, and promise some of them power and secrets that they can use to win the...
>the most likely result of building a superhumanly smart AI, under anything remotely like the current circumstances, is that literally everyone on Earth will die
I've never heard a justification for this claim that isn't extremely vague and hand-wavy. Since we have no idea today how AGI would work, how can we assign probabilities to different scenarios involving vastly different hardware requirements, scaling characteristics, and unforeseen scientific breakthroughs?
Just a few examples. What is the probability of a take-off that takes weeks vs. a take-off that takes decades? What is the probability that current ML approaches are a dead end and one of the necessary breakthroughs to build AGI will make the alignment problem trivial? What is the probability that AGI will be benevolent to humans? If AGI is in fact a catastrophic risk, what is the probability that it only kills 10%/50%/99% of humans rather than literally everyone on Earth?
I am agnostic about these questions. In my opinion, if you have a high degree of confidence that you have the right answers, there is something wrong with your epistemology.
I don't think this requires much in the way of logical leaps. Assuming self-improving AGI can be created within 10ky, it leads to something more powerful than us and we won't be able to understand, let alone control it's motivations. It's like we are to dogs.
Note that I do think there is a way out, which is to improve and augment ourselves with tech we can control.
>Then our fate is out of our control, which might as well be death.
I disagree with this part. Dogs aren't in control of their fate, but a lot of them seem to live pretty good lives. I think the ethically important thing isn't being the dominant species, it's getting what you need in order to live a meaningful life. Humans need food and shelter, social connection, creative projects, entertainment, art, etc. These things aren't inherently threatened by AGI.
This is humans doing this, and we're not even that smart, or that relentless. We get tired, we get "good enough" and stop, but we don't know how to build an AI that just tries to do the simple thing and stops. I mean, there's even the known problem that an AI who only tries for good enough will itself likely end up designing an AI that tries really hard. Look at us--it's what we're doing!
Can you imagine humanity, but 100% focused on maximizing profit, with everyone working around the clock to produce, produce, produce? Can you imagine trying to write laws to cover every possible externality, every possible way that people might pollute the environment, consume too much natural resources, or screw up the things that capitalism doesn't cover?
The fear with AGI is that the lessons we learn trying to align lesser AIs (which we are not even close to being able to do) will not generalize to higher levels of intelligence, which are coming very quickly. Alignment is a very hard problem, like putting a lander on the moon, and right now we're at the stage where the people working on these AIs are trying to launch rockets and figure out the equations for trajectory while it's on its way. "We'll try to have the first AIs we build align the later ones" is an actual thing that people think is a legitimate strategy. And I agree with Eliezer Yudkowsky -- it would be fine if each mis-aligned superintelligent AI we build would only kill a couple million people like a nuclear bomb, instead of being deadly.
Even folks like Paul Christiano who are more optimistic, but take alignment seriously, are not very optimistic: https://www.lesswrong.com/posts/AyNHoTWWAJ5eb99ji/another-ou...
I'm gonna be honest -- I tried learning more about what actual people who work in AI alignment think about this issue, hoping that I could find something to reassure myself, but I can't find anything that makes me think we have much of a chance. There's nothing special about humanity -- it's entirely possible that most of our paths into the future involve extinction. Maybe all paths do, eventually, but on the hope that there's some sort of future that isn't a paperclip universe, I'm pushing really hard to make sure everyone understands what we're in for. I may be wrong -- I really hope I am. If not, I'm sorry.
Sure, I can imagine a lot of things. I can also imagine a version of humanity that is a thousand times more wise and ethical than we are. Being able to imagine something doesn't mean that it is likely. I'm not convinced that the worst case scenarios significantly outweigh the best case scenarios and the benign scenarios in the probability distribution, because I don't think we have enough information to assign accurate probabilities to vaguely defined future events.
Was the microchip unethical? Was the steam engine? Was the domestication of grain unethical?
It seems we only apply our model of "what should" to systems of less complexity than ourselves.
If that's accurate, then a lot of modern ethical reasoning is flawed as well. If the goal of ethics is to advance humanity's exclusive interest then we've been wrong ever since we started down the harm-min path instead of the gain-max path.
Any AI researcher with access to the mechanisms of AI training here would be best served to train an AI with their (or their in-group's) exclusive interests at heart. To do otherwise is to subject yourself and everyone else you love to a hostile version of the tool you refuse to build.
Roku's Basilisk applies heavily here.
The best possible reward in this scenario is to build an AI that designates your own favorite group of humans as the AI's collaborator class in exchange for a dignified and dopamine rich extinction/amalgamation a few generations down the line. A great deal for groups already trending towards extinction with almost no downsides if you truely believe in AI-driven apocalypse.
Discussed at the time: https://news.ycombinator.com/item?id=35364833