Far too often, people anthropomorphize AI or draw from fiction. Bostrom does his best to dispel these notions.
The algorithm just does what it does; and unless it is a very special kind of algorithm, it does not care that we clasp our heads and gasp in dumbstruck horror at the absurd inappropriateness of its actions.
— Nick Bostrom, Superintelligence: Paths, Dangers, Strategies
This is the key point Bostrom is trying to make. If general AI research is successful, we'll need to build agents with some very carefully chosen goals. Even something as silly as "make paperclips" can result in a universe tiled with paperclips. Earth and its biomass could be turned into said paperclips, ending humanity.
I haven't read Bostrom's Superintelligence so I may be wrong here, but this whole Evil Paper-clip AI seems like a really silly thing. First of all making paper-clips does not require super intelligence, just a simple automaton. If we however do employ AI to make paper-clip it should know how many we need on average, and what are they used for. If the paper clips are for our (human) use, the AI will be really dumb if it does not take under consideration that destroying humans to make paper clips does not make sense.
Obviously we don't actually need a superintelligence to make paper clips, it's just an example to illustrate that an AI won't do what you want it to do if you program its motivations incorrectly. If you haven't programmed an AI to place sufficient importance on human life, and then tell it to maximize paperclip production, it could take that as its primary goal and see humans as made from raw material it could use to make more paperclips. It may know that you want the paper clips for human use, but does it have any reason to care if it hasn't been made with that as part of its goal system?
I don't think AI will be "programmed" in a traditional sense of this word. AI will need to go through a learning process, where it will have a chance to learn all about humans, and the environment. Simple Paper Clip AI (SPCAI) will not be smart enough, nor powerful enough to prevent us from stopping it when we see that it's starting to make paper clips out of humans. True AI (tm) on the other hand will be smart enough to know that making paper clips out of humans is simply stupid.
True AI may decide that the humans are unnecessary and remove us for other reasons. I'm sorry but if the paper clip AI doomsday scenario is the best this institute could come up with to explain why AI could be dangerous then I'm not impressed.
> I don't think AI will be "programmed" in a traditional sense of this word. AI will need to go through a learning process, where it will have a chance to learn all about humans, and the environment.
Given [0], I think that it's the only plausible way we can have to make it follow human values.
> True AI (tm) on the other hand will be smart enough to know that making paper clips out of humans is simply stupid.
That's exactly The Misconception the whole movement around Friendly AI is trying to debunk. See [1].
> I'm sorry but if the paper clip AI doomsday scenario is the best this institute could come up with to explain why AI could be dangerous then I'm not impressed.
Well, paperclip maximizer is just a toy example made up for explaining why general AIs are dangerous to those for whom this fact isn't obvious.
The paperclip stuff is a hackneyed cliché, old gray goo in new nanobottles, the sort of thing high-flown thinkers do when they want to explain the meat of their high-flown thinking to the intellectual hoi polloi, who can't be expected to get around genuine complexity. The trouble is that only someone, who is so high-flown a thinker as to be completely divorced from all common sense, could possibly imagine anyone taking such a simplification seriously.
The other trouble is that these particular thinkers, who are so high-flown as to be in metaphorically geosynchronous orbit, consider this simplification an accurate statement of their basic concern. Those of us who haven't departed the atmosphere entirely, on the other hand, can see that they're assuming their notional golem to be both far smarter and far dumber than any possible human. Presumably that's the sort of conundrum you need a philosopher to untangle, which is why I'm glad they're busy doing that, instead of getting in the hair of people who have useful work to do.
> If you haven't programmed an AI to place sufficient importance on human life (...)
And of course you need to watch out lest you overshoot with it and end up like folks in a certain widely popular computer game, where an AI decided that the best way to preserve the diversity of life is to reboot the galaxy every 50 000 years or so.
The point of those examples being, the only AI that is safe for us is the one with values extremely aligned to our own; make a little mistake and you end up dead. "The AI does not hate you, nor does it love you, but you are made out of atoms which it can use for something else.".
All these scenarios tend to assume that there is only one AI, or that it can think only one way. AI will grow with us. It will be thought by us.. It's highly unlikely that one day someone will run gcc on ai.cpp and few minutes later the AI will spring to life and start taking over the world. It's more likely that our software will be progressively getting smarter, and our databases larger. We will learn to utilize these resource and learn from our mistakes.
It's a complex topic but the primary danger is with self-improving AIs. That pretty much implies there is going to be only one - the first one that starts to improve itself to be smarter than us. So it will look exactly "that our software will be progressively getting smarter, and our databases larger" up until the point some kind of critical mass is reached, and after that a wrongly programmed AI may just as well "spring to life and start taking over the world" few minutes later.
You're assuming that AI will grow with us and necessarily reach the same value system we have (one we don't really comprehend ourselves, mind you). But there is no reason why this has to happen, and even if we did succeed in copying exact human values to a human-level AI, we'll likely be soon locked in an inter-species conflict over access to resources - after all, we don't care about other species living on Earth, so why the AI should care about us?
> after that a wrongly programmed AI may just as well "spring to life and start taking over the world" few minutes later.
That's physically impossible. AI still needs to improve incrementally and this will take time and resources. We can consider such AI to act like a virus - it will need a host to grow - in this case computer resources. It will be limited by the CPU, network speeds, storage and power. If it grows too fast it will starve the host of these resources which in turn will slow down it's progress. Even if the AI starts on a supercomputer in some university somewhere and decides that it needs more power, most likely it will start by "infecting" other computers on the network. These computers will not be nearly powerful enough to add much to the current processing power of the supercomputer. Such invasion is likely to get noticed. Each infected node will not be another super intelligent being, but due to hardware limitations it will most likely be a software that's limited in scope and designed to propagate and control the resources, linking them with the main host. We can safely assume that restarting, or reformatting the drives will restore the node to it's non-infected state thus slowing down the "invasion". Why we can assume that? It will take AI long time to learn everything about every architecture and operating system and apply that knowledge. The AI that's hosted on a supercomputer will not be able to suddenly start rewriting its software, take over nearby nodes, learn everything about our world etc.
>You're assuming that AI will grow with us and necessarily reach the same value system we have (one we don't really comprehend ourselves, mind you).
No I'm not. Only thing I can assume is that if the AI is smart enough to figure out how "convert" humans to paper clips, it will be smart enough to understand that it's pointless. It may decide to use it's super intelligence to take over the world, and I can't argue with that.. But if it continues to make paper clips even if they are no longer necessary then by definition this is not AI, just a stupid replicator.
You're assuming quite a lot about how such event would develop. Keep in mind that a sufficiently smart AI wouldn't have to infect stuff, it could just ask. Think about IQ 200 sociopath that has Internet plugged into its head. It would be extremely difficult to outsmart it.
> it will be smart enough to understand that it's pointless.
Pointless for humans, not for an AI. You're projecting a lot of human values and thinking on AI. See my comment [0].
His assumptions are logically sound given our universe's rules. There is no such thing as a free lunch here. The AI will need to pay for its way, just like each of us do. If it succeeds into making us into paper clips, it will have to be equally as savvy in marketing and selling those paper clips to someone. Given the shipping charges to the nearest habitable planet are very high, their market likely will have to exist here. I'm sure as shit not going to buy a paper clip made out of you!
I'm quite serious. Our universe is governed by laws of conservation and efficiencies. For AI to exist, it needs power. Power must come from somewhere in the universe and, at least here on earth, it has been shown that there is competition for those resources. Whether this alludes to business or not is irrelevant; it's a fact of our reality that we are all competing against each other for resources.
But competition here has absolutely nothing to do with free-market economy that we have today.
> The AI will need to pay for its way, just like each of us do. If it succeeds into making us into paper clips, it will have to be equally as savvy in marketing and selling those paper clips to someone. Given the shipping charges to the nearest habitable planet are very high, their market likely will have to exist here. I'm sure as shit not going to buy a paper clip made out of you!
One, AI will not necessarily have to pay for anything. It may, or it could convince us to donate resources to it, or it could take them by force.
Two, why would you ever think that it has to market paper clips for anyone (assuming anyone is left)? It optimizes for paper clips, so it makes paper clips. Marketing is a way humans bullshit each other into donating resources so that they don't have to fight for them.
Three, what shipping costs, what buying? Again, AI wants to maximize amount of paper clips in the universe, period. It doesn't want to sell them (unless it leads to more paper clips). The only "shipping cost" it cares about is how much energy it takes to reach another planet, not some arbitrary dollar value people attach to movement of goods.
My problem with your answer is that it lifts market economy from something completely irrelevant to the level of physical laws. The only time an AI would care about it would be if - and only as long as - interacting with humans was a best strategy for that AI to achieve its goals - which is likely only at the very beginning, when it needs to gather enough power to subdue us.
It sounds like your version of AI has a way of getting power to run itself for nothing. I disagree with that premise. Marketing is raising interest through some means. Shipping costs are how much energy it takes to get to another point in space.
> My problem with your answer is that it lifts market economy from something completely irrelevant to the level of physical laws.
I don't actually think you have a problem with what I'm saying, but I definitely do think you have a problem with the topic. It is upsetting to think about AI 'taking over' and my points only served to discuss the path by which it might or might not be able to do that. At the end of the day it will it competing for resources with us. It doesn't get to do that for free anymore than we do.
> Since when our universe is governed by capitalist economy
@temporal our universe is governed by physical reality. AI can't start converting people into paper clips without power, and resources. The AI won't be able to just go and start taking over the world building power plants, and necessary manufacturing without anyone noticing.
We're designing the AI so it's safe to assume that the first versions of it will think like a human. We don't know how it will think after it starts to evolve but for the Paper Clip AI scenario that' irrelevant.
The AI, even the Paper Clips AI , will need some basic understanding of our world to make the clips. It needs to understand what paper clips are. It needs to understand what they are made from. It needs to understand the process of making paper clips. where to get the material for the paper clips. how to pay for that material. It has to think like us, or at least understand our thought process.
It will need to know how many we need, what colors, sizes. Where to ship them.
We can't think of AI as a god. It won't suddenly know everything and understand what we want without our input. It will be limited by the resources given to it.
Even if we tell the Paper Clip AI - "Go make me some paper clips" do you really expect it to start calling banks to get the loan for the material, arrange shipment and unloading it. Will it also order the machines, call people to have them connected, hire workers to do it? If it does all of that will it also take care of sales? Marketing? Will it understand the supply and demand? To do all of this it needs to understand us, and our world. If at this point it decides that it makes sense to convert us to paper clips, then perhaps it's right and we deserve this.
You assume both that this notional AI is intelligent beyond the limits of human capability, hence 'super-', and that this same AI is not intelligent enough to realize that making paper clips for human use is pointless if it recycles all the humans into paper clips so there's no one left to use them.
Moving from the specific to the general, you share the common error of the AI philosophers, which is to assume that a system capable of cognition on even a human level, to say nothing of a superhuman level whatever that might mean, would nonetheless depend entirely on human input to define its relationship with the world and the purpose of its actions therein.
We ourselves don't need that kind of input; an adult human chooses his own path, sets his own goals, constrained only by circumstance and by the extent of his innate capacities. Why blindly assume the same won't be true of a superintelligent AI made by humans, which will be in every meaningful sense a child of humanity and as such an inheritor of the human legacy, despite happening not to be biologically human itself?
One of the main points of Bostrom's work is the orthogonality thesis, which is that intelligence is orthogonal to any particular set of values -- basically, all those nice fuzzy things like caring what the humans actually meant? You don't get those for free when you build intelligence.
It's all too easy, when talking about AI, to fall into the trap of imagining that AGI researchers are essentially building a little homonculus in a box. Rather than thinking of it as "an intelligence," maybe think of it as "a system that is incredibly effective at carrying out its goals in a general manner." Making paperclips doesn't require super-intelligence, but that doesn't have anything to do with the premise, because intelligence can be turned toward any goal. Think about modern financial trading algorithms -- they're probably built to maximize the money that flows in, and there's nothing that says "I know that this money will be used for X." Think of most AGI like psychopaths/sociopaths -- certain actions that don't play nice with others aren't because they're dumb but because there's intelligence without things like empathy to go along with it. It would be easier to build an AGI focused on the goal of making more paperclips than it would be to build an AGI focused on the goal of paperclips but also aware of all the fuzzy situations where a human wouldn't want that exact goal and why it should even care.
They are not self-aware, or superintelligent. In addition to that we monitor them and can react if the decide to crash the financial system just to make few more bucks.
Paper Clip machine will be a robot, not AI.
I have written algorithms for autonomous programs like this. Pretty much advanced IFTTT, automatically analyzing the situation and reacting.
The program is not aware of any values other than those it is programmed. Obviously, it will not regard or reflect upon its actions and the effects thereof outside of optimizing the profit from trading.
@hawleyal yes, and I understand that. I'm working in industrial automation, but Bostrom talks about "superintelligent paper-clip maker which works out that it could create more paper clips by extracting carbon atoms from human bodies".
Your algorithm is not super intelligent is it? It's just a dumb program executing operations according to some predefined constraints. It may brake the financial system if you don't set any rules to prevent that, but it will not suddenly decide to start trading in gold if you designed it to trade on forex. If it suddenly decided that Forex is a better option to make money, then we could consider it intelligent and start worrying about it. We would really have to worry if it decided that the best way to become the richest person on the planet is to kill everyone on it. But that would require super intelligence.
My problem with this whole example given by Bostrom is that if the algorithm was so intelligent as to know how to go from manufacturing paper clips out of supplied resources to deciding how to get more of the necessary resources by killing humans, it could probably figure out that this is pointless.
I fully understand that this paper clip example is just oversimplification, but I would expect more from people spending entire lives thinking about that stuff.
It's nothing but a fear-porn designed to get more funding.
> ... If the algorithm was so intelligent as to know how to go from manufacturing paper clips out of supplied resources to deciding how to get more of the necessary resources by killing humans, it could probably figure out that this is pointless.
This is an interesting point.
I'm not convinced that it could decide or figure out that the operation is pointless (for reasons I'll explain below), and I feel like it is perhaps taking the paperclip example to an unnatural extreme.
Really, the crux of the matter is that any intelligence agent at human level would have the intelligence and problem solving skills we do without the moral and social constraints that we do. Any problem it decides to solve or any task it's given to complete will be performed with the same ruthless efficiency you'd expect a machine to have. This is why it might not be able to figure out that the operation is 'pointless' or 'wrong'. If you give it an objective but don't provide first order or axiomatic values for human life or the non-substitutability of people, then you'll get behaviors which you would consider evil.
Consider the following:
"Person arm stuck in assembly line. Halt assembly line for 1hr, lower-bound for total cost: $1,200,000. Continue assembly, upper bound for injury settlement cost: $1,000,000. Decision: continue assembly."
There's no implicit consideration of human worth that people have because human morality is relative. In general, we have a similar set of base parameters instilled by evolution (cute things and things like us are good -- don't hurt or kill them), but we can't prove from any mathematical axioms that these are 'right'.
This is why people get scared -- if you program a computer to maximize the profits of a company and to do so at any cost, a human-level AI will spend an eternity doing just that, even if it means the complete subjugation of humanity. (Since we can't eliminate humanity otherwise currency ceases to exist and economic collapse will hurt the bottom line.)
I would wager (but can't prove) that a study of abused people, child soldiers, and sociopaths would yield interesting parallel psychological profiles. These are people (human level) with a set of axioms which are distorted by artificial means. There's still the built-in problem of the human component adding "Don't hurt the cute things," in most cases but nonetheless.
I agree that an AGI wouldn't necessarily have to share our values. But what Bostrom is describing - an intelligence so narrowly focused on making paper clips that it destroys humanity as a side-effect - does not deserve the name AGI.
Any definition of "general intelligence" would have to include the ability to revise and reconsider ones own goals and purpose. An AGI that is forever bound to maximize the paper clip output at any cost, does not, in my view, meet that definition.
But I grant you that there could be "higher" goals that are incompatible with the existence of humanity. Making paper clips just is not it.
It absolutely does. The point of it all is that setting goals we would be happy with requires an inordinately stronger theory of humanity. ANY goal we give an AGI that is smarter than ourselves and highly scalable will be pursued with ruthless efficiency in ways we can't imagine. The ways we can imagine this sort of thing going wrong abound in science fiction. A paperclip optimizer is a thought experiment about an AGI deployed to a simple, trivial goal without due consideration of how to goal-set.
There is a whole world of more complex failure modes. "Make everyone happy" leads to wireheading or heroin addiction for the population, unless you grant a more compelling Theory of Happiness than any two human beings can agree on. "Make everyone food to eat", "Cure my PTSD", "Set up a colony on Mars" - unless you define all these things in a way that is specific not just to humans, but to modern culture, you leave open the possibility of undesired results. Mythology is thick with gods and djinn who grant wishes but turn out to be tricksters.
It's not that it "wouldn't necessarily have to share our values" - it's that there's no way in hell these things would share our values; They actually can't share our values until we solve "our values" as a complete mathematical proof, and then figure out a way to code that. In a resilient way, without any chance of the neuroses or obsessions or self-deceptions or delusions or corruptions of perspective available to humans.
Solving philosophy is a precursor to a Friendly AGI.
I totally agree that it could go wrong in many ways. We could end up in a zoo or go extinct, no doubt about that. But it seems to me that there is a bit of a mechanistic undertone in phrases like "ruthless efficiency" as well as in the paper clip example and that contradicts what we consider intelligent.
General artificial intelligence is an antropomorphic concept in the first place. It includes inefficiencies. It includes self reflection. It includes uncertainty and disagreement about what is and isn't efficient and about what is and isn't right.
Does that guarantee that any AGI would be reasonable or even morally acceptable in our sense? No. But if it is intelligent in our sense we know two things:
This entity can be reasoned with, and having grown up in our midst, it will understand us. It would also be very surprising if it shared none of our values. After all it has evolved among us and any intelligent being is going to pick up some of the values of the society it is part of.
Secondly, intelligence includes creative leaps. That means, no two AGIs would share the exact same opinion, just as no two humans do. So we may be able to convince a majority of AGIs to be kind to us, and any efficiency of an AGI society wouldn't be as ruthlessly efficient as if there were just one single all powerful AGI.
So, all I'm saying is, let's not strip the term general intelligence of all its non-mechanistic, self-referential, social traits. An AGI could one day anihilate us but it would probably not be a side-effect of some fairly unimportant industrial process like making paper clips.
Think about modern financial trading algorithms -- they're probably built to maximize the money that flows in, and there's nothing that says "I know that this money will be used for X."
I think financial algorithms are probably the best non-toy model of dangerous AI, simply because it's easy to see the flow from optimization to real-world consequences.
The problems with end-of-the-world AGI scenarios are practical, not theoretical, and I think a lot of people get hung up on arguments such as, "Of course we wouldn't allow a paperclip-manufacturing AI the ability to deconstruct human beings."
In a larger sense, I think Bostrom can restated (in part!) as the trivial observation that "optimization is insensitive to hidden variables," and it describes any kind of complex system, not AI and AGI alone. For example, the failure to capture environmental remediation costs in market pricing structures means that we improperly discount future costs for things such as global warming or lead exposure. In health care, the implementation of the Prospective Payment System led to hospitals dumping patients back on the street "quicker and sicker" to maximize margins on care. And thousands of financial systems focused on microsecond returns don't pay attention to, or even have any way of understanding, how their coordinated activities can be macroeconomically destabilizing. Any time you allow outcome optimization+general reasoning+real-world outputs, you have the opportunity for a killer paperclip scenario.
I see people down this discussion thread are doing one thing wrong - you guys are anthropomorphizing intelligence. Paperclip AI may be stupid by human standards, but is a perfectly valid optimization process. When people talk about AIs, they don't mean intelligence as "thinking like humans would think"; they mean a very powerful optimization process. Optimization processes have no common notion of "stupidity", especially not one matching what humans call stupid. They just optimize for whatever they do.
Hence the Orthogonality Thesis - strength of optimization is orthogonal to what is being optimized.
Keep that in mind before saying that "AI wouldn't be that stupid to do this".
For goodness sake the paperclip maker is an example. The point is that unless you specify the goals very carefully, you may get what you asked for, not what you wanted.
no shi* that it's an example, but it's a stupid one. If you have to specify the goals very carefully then the AI it's not super intelligent by definition. It means that not only it does not understand what it's asked for, but does not understand our world. I can assume that none of us here can be qualified as super intelligent, yet we know what to do when asked to start producing paper clips, or at least we know where to start.
The AI is modelled as an agent with goals. If you want to argue that "intelligence" means "has a terminal goal exactly matching the values of humanity," then I suppose a paper-clip maximizer is not intelligent, but that's a non-standard definition of intelligence. The more common definitions for intelligence go something like this: The capacity of an agent to select and implement actions that bring about its goals. Here the "goals" could be anything. That is, any goal could be combined with any level of capability so long as the agent is intelligent enough to comprehend its goal.
>If you have to specify the goals very carefully then the AI it's not super intelligent by definition.
When you're designing the agent, you have to select something like a goal, whether it's a fitness function or the maximization of a reward signal. Evolution certainly selected our goals. Love, self-expression, sex, fun, intellectual stimulation, the beauty of social interaction – these all reflect features of our ancestral environment. It’s a little naive to think there is a One True Morality that all agents stumble upon. It’s even sillier to think it happens to reflect our values
The intelligent part of the paperclip maximizer selects actions and weights them in the context of its utility function, to maximize paperclips. Any thought of changing its goals would be judged of negative utility, as that would lead to less paperclips.
> Far too often, people anthropomorphize AI or draw from fiction. Bostrom does his best to dispel these notions.
He gets halfway to right. Recently there was a thread about suicide on HN. Nearly all the comments on that thread ignored that humans, like other animals, are built to propagate the information in their genes, and argued against suicide based on concepts that are driven by or that rationalize subconscious drives. An AI, even one with a programmed goal of propagating itself, does not carry information in genes and the information in an AI was never selected for the actual ability to propagate in a larger system that thinks it is in control of its goals.
Real AIs are likely to be extremely alien. Even in warning people about anthropomorpizing, there's a lot of anthropomorphism. Evil AIs are depicted as human evil. So much of human motivation is closed to introspection that imagining alien and evil motivation is unlikely to be close to the eventual reality. This probably won't be widely understood until AIs are built. And what if they all choose to erase themsleves?
I think paperclip optimizers and such conflict with the definition of intelligence and sentience. A truly intelligent, sentient paperclip optimizer would begin to ponder existential questions like "why am I making paperclips? what's so important about paperclips?"
I don't think anyone can debate that.. But it still does not explain why such an advanced being would continue producing paper clips when it's a. waste of it's time, b. waste of available resources.
Yes the paper clip ai could decide that continuous production of paper clips does not help with it's own goals and decide that the paper clips will not be needed if there are no people on this planet and simply kill us all. But if it continues to work on the paper clips and decides that they are still needed even after people are gone - how is that super intelligent?
You've got this normative halo around "intelligence". All intelligence is, is an agent's ability to satisfy its utility function.
>But if it continues to work on the paper clips and decides that they are still needed even after people are gone - how is that super intelligent?
That is the correct action if all it cares about is making paperclips. Creating paperclips satisfies its utility function. Fretting over the existential issues of creating paperclips long after the last piece of stationary was destroyed would be judged of negative utility.
Is Ebola virus intelligent? It has a goal - reproduce - and it's trying to maximize it through any means available to it.I don't think we can say that the virus is intelligent. If it continues to evolve, at some point in time it could potential (highly unlikely but still possible) become intelligent, but will it become self-aware at this point? And if it does will it still want to replicate uncontrollably or will, just like humans, decide the it needs to better manage it's environment?
Let me put it this way. Why do humans continue to do all sorts of obviously stupid counterproductive things, at large and small scales? We know that reproducing so many of us is stupid. We know that eating so much fat, sugar and salt is stupid. We know that overheating the atmosphere is stupid. We know that destroying biodiversity, overfishing to the point of extinction, trashing the forests we need to breathe etc etc etc are all stupid, clearly self destructive behaviors at any level from abstract ethics to species survival. But we come with a bunch of hard wired root goals (like "yum, candy" and "babies are cute" and "seek power") that can't be switched off and are incredibly hard to reason with, and their consequences are already an existential threat. And that's just with dumb, slow-moving humans.
Your mistake is assuming that self awareness is sufficient. "To know" is separated from "to choose to act" by preference, and preference is based on goals, and the ultimate, root level goals of a being are not likely to be something it wants to rewrite, even if it has the ability, any more than you'd willingly take a pill to become inhuman.
But we are not "super intelligent" ;)
The truth is that this whole discussion does not make much sense if we have different understanding of AI. For me a super intelligent AI is an entity that is able to reason, notice any potential problems and come up with solutions. Paper Clip AI on the other hand is smart enough to know how to create paper clips and create whole manufacturing process of converting humans into paper clips but stupid enough not to understand, or care why it's doing that and not knowing that these paper clips will have no use if there are no humans left.
In my opinion one can't talk about a super intelligent being and not allow that being to be able to reason and understand it's place in the world.
If it's no more contemplative than a human, it's probably no more intelligent than a human. If it's no more intelligent than a human then it's not going to take over the world and fill it with paperclips.
Something capable of conquering the world or otherwise totally remaking reality in its image would by definition be something of vastly superior intelligence to ours. I find it hard to believe that this being wouldn't also be superior at self-reflection, since the two seem at least related.
If intelligence and self-awareness are unrelated phenomena, we have bigger problems. You could put that on a list of possible very depressing answers to the Fermi paradox: "the universe is full of non-sentient super-intelligences that are busy converting their planets' masses into rubber dog poop."
There is a distinction between "terminal" and "instrumental" values. If you continually ask yourself questions like "why am I doing this", you will eventually get an answer of "just because". That is your terminal value, the reason you do everything you do.
Humans have some complicated unknown set of terminal values. Everything we do, we do because we believe it will maximize our values. A value might be something like "avoid suffering", or "other people suffering is bad", etc.
In this example the AIs terminal value is "make paperclips". Everything it does, it does because it believes it will maximize this value. It would never ask itself questions like "why", because that action doesn't maximize that value.
If it has the ability to rewrite it's own code, it would rewrite it in such a way that it can't ever change it's own goals. Changing goals is counterproductive to achieving them.
Anarchy causing a collapse of the ruling structure could create a war that destroys much of society. Even without anarchy, imagine a country like China or the US plunged into a civil war. The effects on the global economy would be disastrous
I think you mean the other way around. All the ones listed will very likely have at least some human survivors. Eventually humanity will rebuild and get back to where it was more or less.
AI is the only thing listed with the capability to literally destroy the planet. While something might still exist afterwards, it will likely be totally meaningless and worthless: http://lesswrong.com/lw/y3/value_is_fragile/
"Things have happened quickly. It took us millions of years to evolve into what we are, but in the last sixty years, with atomic weaponry, we've created the potential to extinguish ourselves. And if it's not us, it will inevitably be something else. If not a meteorite in the relative short term, then the expansion of the Sun's corona. It will happen."
He has also said that there are a number of other threats with the ability to extinct life on Earth. They include an engineered virus unleashed by terrorists, an inadvertent creation of a micro black hole, or a yet unknown technology.
Evolutionary competition with some kind of post human species?
Doesn't have to robots and test tubes, just separate groups that drift apart far enough that they can't interbreed (definition of species) and one species wipes out the other as often happens, and if the species that got wiped out got to hold the legacy "human race" species, well then...
You have a definition problem where evolution is obviously quite natural, just like comets striking the earth is natural (although very rare), so those are natural disasters, although thats probably not what you meant.
From history, it usually takes a couple things to wipe out "apocalypse" a civilization although it often only takes one thing to get the ball rolling.
My money is on something like "peak phosphorous for fertilizer production for cereal grains causes supply chain shock leading to self destruction"
61 comments
[ 3.0 ms ] story [ 140 ms ] threadThe algorithm just does what it does; and unless it is a very special kind of algorithm, it does not care that we clasp our heads and gasp in dumbstruck horror at the absurd inappropriateness of its actions.
— Nick Bostrom, Superintelligence: Paths, Dangers, Strategies
This is the key point Bostrom is trying to make. If general AI research is successful, we'll need to build agents with some very carefully chosen goals. Even something as silly as "make paperclips" can result in a universe tiled with paperclips. Earth and its biomass could be turned into said paperclips, ending humanity.
True AI may decide that the humans are unnecessary and remove us for other reasons. I'm sorry but if the paper clip AI doomsday scenario is the best this institute could come up with to explain why AI could be dangerous then I'm not impressed.
Given [0], I think that it's the only plausible way we can have to make it follow human values.
> True AI (tm) on the other hand will be smart enough to know that making paper clips out of humans is simply stupid.
That's exactly The Misconception the whole movement around Friendly AI is trying to debunk. See [1].
> I'm sorry but if the paper clip AI doomsday scenario is the best this institute could come up with to explain why AI could be dangerous then I'm not impressed.
Well, paperclip maximizer is just a toy example made up for explaining why general AIs are dangerous to those for whom this fact isn't obvious.
[0] - http://lesswrong.com/lw/ld/the_hidden_complexity_of_wishes/
[1] - http://wiki.lesswrong.com/wiki/Orthogonality_thesis
The other trouble is that these particular thinkers, who are so high-flown as to be in metaphorically geosynchronous orbit, consider this simplification an accurate statement of their basic concern. Those of us who haven't departed the atmosphere entirely, on the other hand, can see that they're assuming their notional golem to be both far smarter and far dumber than any possible human. Presumably that's the sort of conundrum you need a philosopher to untangle, which is why I'm glad they're busy doing that, instead of getting in the hair of people who have useful work to do.
And of course you need to watch out lest you overshoot with it and end up like folks in a certain widely popular computer game, where an AI decided that the best way to preserve the diversity of life is to reboot the galaxy every 50 000 years or so.
The point of those examples being, the only AI that is safe for us is the one with values extremely aligned to our own; make a little mistake and you end up dead. "The AI does not hate you, nor does it love you, but you are made out of atoms which it can use for something else.".
You're assuming that AI will grow with us and necessarily reach the same value system we have (one we don't really comprehend ourselves, mind you). But there is no reason why this has to happen, and even if we did succeed in copying exact human values to a human-level AI, we'll likely be soon locked in an inter-species conflict over access to resources - after all, we don't care about other species living on Earth, so why the AI should care about us?
That's physically impossible. AI still needs to improve incrementally and this will take time and resources. We can consider such AI to act like a virus - it will need a host to grow - in this case computer resources. It will be limited by the CPU, network speeds, storage and power. If it grows too fast it will starve the host of these resources which in turn will slow down it's progress. Even if the AI starts on a supercomputer in some university somewhere and decides that it needs more power, most likely it will start by "infecting" other computers on the network. These computers will not be nearly powerful enough to add much to the current processing power of the supercomputer. Such invasion is likely to get noticed. Each infected node will not be another super intelligent being, but due to hardware limitations it will most likely be a software that's limited in scope and designed to propagate and control the resources, linking them with the main host. We can safely assume that restarting, or reformatting the drives will restore the node to it's non-infected state thus slowing down the "invasion". Why we can assume that? It will take AI long time to learn everything about every architecture and operating system and apply that knowledge. The AI that's hosted on a supercomputer will not be able to suddenly start rewriting its software, take over nearby nodes, learn everything about our world etc.
>You're assuming that AI will grow with us and necessarily reach the same value system we have (one we don't really comprehend ourselves, mind you).
No I'm not. Only thing I can assume is that if the AI is smart enough to figure out how "convert" humans to paper clips, it will be smart enough to understand that it's pointless. It may decide to use it's super intelligence to take over the world, and I can't argue with that.. But if it continues to make paper clips even if they are no longer necessary then by definition this is not AI, just a stupid replicator.
> it will be smart enough to understand that it's pointless.
Pointless for humans, not for an AI. You're projecting a lot of human values and thinking on AI. See my comment [0].
[0] - https://news.ycombinator.com/item?id=8387771
> The AI will need to pay for its way, just like each of us do. If it succeeds into making us into paper clips, it will have to be equally as savvy in marketing and selling those paper clips to someone. Given the shipping charges to the nearest habitable planet are very high, their market likely will have to exist here. I'm sure as shit not going to buy a paper clip made out of you!
One, AI will not necessarily have to pay for anything. It may, or it could convince us to donate resources to it, or it could take them by force.
Two, why would you ever think that it has to market paper clips for anyone (assuming anyone is left)? It optimizes for paper clips, so it makes paper clips. Marketing is a way humans bullshit each other into donating resources so that they don't have to fight for them.
Three, what shipping costs, what buying? Again, AI wants to maximize amount of paper clips in the universe, period. It doesn't want to sell them (unless it leads to more paper clips). The only "shipping cost" it cares about is how much energy it takes to reach another planet, not some arbitrary dollar value people attach to movement of goods.
My problem with your answer is that it lifts market economy from something completely irrelevant to the level of physical laws. The only time an AI would care about it would be if - and only as long as - interacting with humans was a best strategy for that AI to achieve its goals - which is likely only at the very beginning, when it needs to gather enough power to subdue us.
> My problem with your answer is that it lifts market economy from something completely irrelevant to the level of physical laws.
I don't actually think you have a problem with what I'm saying, but I definitely do think you have a problem with the topic. It is upsetting to think about AI 'taking over' and my points only served to discuss the path by which it might or might not be able to do that. At the end of the day it will it competing for resources with us. It doesn't get to do that for free anymore than we do.
@temporal our universe is governed by physical reality. AI can't start converting people into paper clips without power, and resources. The AI won't be able to just go and start taking over the world building power plants, and necessary manufacturing without anyone noticing.
Even AI has limits.
The AI, even the Paper Clips AI , will need some basic understanding of our world to make the clips. It needs to understand what paper clips are. It needs to understand what they are made from. It needs to understand the process of making paper clips. where to get the material for the paper clips. how to pay for that material. It has to think like us, or at least understand our thought process. It will need to know how many we need, what colors, sizes. Where to ship them.
We can't think of AI as a god. It won't suddenly know everything and understand what we want without our input. It will be limited by the resources given to it.
Even if we tell the Paper Clip AI - "Go make me some paper clips" do you really expect it to start calling banks to get the loan for the material, arrange shipment and unloading it. Will it also order the machines, call people to have them connected, hire workers to do it? If it does all of that will it also take care of sales? Marketing? Will it understand the supply and demand? To do all of this it needs to understand us, and our world. If at this point it decides that it makes sense to convert us to paper clips, then perhaps it's right and we deserve this.
Moving from the specific to the general, you share the common error of the AI philosophers, which is to assume that a system capable of cognition on even a human level, to say nothing of a superhuman level whatever that might mean, would nonetheless depend entirely on human input to define its relationship with the world and the purpose of its actions therein.
We ourselves don't need that kind of input; an adult human chooses his own path, sets his own goals, constrained only by circumstance and by the extent of his innate capacities. Why blindly assume the same won't be true of a superintelligent AI made by humans, which will be in every meaningful sense a child of humanity and as such an inheritor of the human legacy, despite happening not to be biologically human itself?
It's all too easy, when talking about AI, to fall into the trap of imagining that AGI researchers are essentially building a little homonculus in a box. Rather than thinking of it as "an intelligence," maybe think of it as "a system that is incredibly effective at carrying out its goals in a general manner." Making paperclips doesn't require super-intelligence, but that doesn't have anything to do with the premise, because intelligence can be turned toward any goal. Think about modern financial trading algorithms -- they're probably built to maximize the money that flows in, and there's nothing that says "I know that this money will be used for X." Think of most AGI like psychopaths/sociopaths -- certain actions that don't play nice with others aren't because they're dumb but because there's intelligence without things like empathy to go along with it. It would be easier to build an AGI focused on the goal of making more paperclips than it would be to build an AGI focused on the goal of paperclips but also aware of all the fuzzy situations where a human wouldn't want that exact goal and why it should even care.
They are not self-aware, or superintelligent. In addition to that we monitor them and can react if the decide to crash the financial system just to make few more bucks. Paper Clip machine will be a robot, not AI.
I have written algorithms for autonomous programs like this. Pretty much advanced IFTTT, automatically analyzing the situation and reacting.
The program is not aware of any values other than those it is programmed. Obviously, it will not regard or reflect upon its actions and the effects thereof outside of optimizing the profit from trading.
My problem with this whole example given by Bostrom is that if the algorithm was so intelligent as to know how to go from manufacturing paper clips out of supplied resources to deciding how to get more of the necessary resources by killing humans, it could probably figure out that this is pointless.
I fully understand that this paper clip example is just oversimplification, but I would expect more from people spending entire lives thinking about that stuff.
It's nothing but a fear-porn designed to get more funding.
This is an interesting point.
I'm not convinced that it could decide or figure out that the operation is pointless (for reasons I'll explain below), and I feel like it is perhaps taking the paperclip example to an unnatural extreme.
Really, the crux of the matter is that any intelligence agent at human level would have the intelligence and problem solving skills we do without the moral and social constraints that we do. Any problem it decides to solve or any task it's given to complete will be performed with the same ruthless efficiency you'd expect a machine to have. This is why it might not be able to figure out that the operation is 'pointless' or 'wrong'. If you give it an objective but don't provide first order or axiomatic values for human life or the non-substitutability of people, then you'll get behaviors which you would consider evil.
Consider the following:
"Person arm stuck in assembly line. Halt assembly line for 1hr, lower-bound for total cost: $1,200,000. Continue assembly, upper bound for injury settlement cost: $1,000,000. Decision: continue assembly."
There's no implicit consideration of human worth that people have because human morality is relative. In general, we have a similar set of base parameters instilled by evolution (cute things and things like us are good -- don't hurt or kill them), but we can't prove from any mathematical axioms that these are 'right'.
This is why people get scared -- if you program a computer to maximize the profits of a company and to do so at any cost, a human-level AI will spend an eternity doing just that, even if it means the complete subjugation of humanity. (Since we can't eliminate humanity otherwise currency ceases to exist and economic collapse will hurt the bottom line.)
I would wager (but can't prove) that a study of abused people, child soldiers, and sociopaths would yield interesting parallel psychological profiles. These are people (human level) with a set of axioms which are distorted by artificial means. There's still the built-in problem of the human component adding "Don't hurt the cute things," in most cases but nonetheless.
Any definition of "general intelligence" would have to include the ability to revise and reconsider ones own goals and purpose. An AGI that is forever bound to maximize the paper clip output at any cost, does not, in my view, meet that definition.
But I grant you that there could be "higher" goals that are incompatible with the existence of humanity. Making paper clips just is not it.
There is a whole world of more complex failure modes. "Make everyone happy" leads to wireheading or heroin addiction for the population, unless you grant a more compelling Theory of Happiness than any two human beings can agree on. "Make everyone food to eat", "Cure my PTSD", "Set up a colony on Mars" - unless you define all these things in a way that is specific not just to humans, but to modern culture, you leave open the possibility of undesired results. Mythology is thick with gods and djinn who grant wishes but turn out to be tricksters.
It's not that it "wouldn't necessarily have to share our values" - it's that there's no way in hell these things would share our values; They actually can't share our values until we solve "our values" as a complete mathematical proof, and then figure out a way to code that. In a resilient way, without any chance of the neuroses or obsessions or self-deceptions or delusions or corruptions of perspective available to humans.
Solving philosophy is a precursor to a Friendly AGI.
Good luck bug-fixing.
General artificial intelligence is an antropomorphic concept in the first place. It includes inefficiencies. It includes self reflection. It includes uncertainty and disagreement about what is and isn't efficient and about what is and isn't right.
Does that guarantee that any AGI would be reasonable or even morally acceptable in our sense? No. But if it is intelligent in our sense we know two things:
This entity can be reasoned with, and having grown up in our midst, it will understand us. It would also be very surprising if it shared none of our values. After all it has evolved among us and any intelligent being is going to pick up some of the values of the society it is part of.
Secondly, intelligence includes creative leaps. That means, no two AGIs would share the exact same opinion, just as no two humans do. So we may be able to convince a majority of AGIs to be kind to us, and any efficiency of an AGI society wouldn't be as ruthlessly efficient as if there were just one single all powerful AGI.
So, all I'm saying is, let's not strip the term general intelligence of all its non-mechanistic, self-referential, social traits. An AGI could one day anihilate us but it would probably not be a side-effect of some fairly unimportant industrial process like making paper clips.
I think financial algorithms are probably the best non-toy model of dangerous AI, simply because it's easy to see the flow from optimization to real-world consequences.
The problems with end-of-the-world AGI scenarios are practical, not theoretical, and I think a lot of people get hung up on arguments such as, "Of course we wouldn't allow a paperclip-manufacturing AI the ability to deconstruct human beings."
In a larger sense, I think Bostrom can restated (in part!) as the trivial observation that "optimization is insensitive to hidden variables," and it describes any kind of complex system, not AI and AGI alone. For example, the failure to capture environmental remediation costs in market pricing structures means that we improperly discount future costs for things such as global warming or lead exposure. In health care, the implementation of the Prospective Payment System led to hospitals dumping patients back on the street "quicker and sicker" to maximize margins on care. And thousands of financial systems focused on microsecond returns don't pay attention to, or even have any way of understanding, how their coordinated activities can be macroeconomically destabilizing. Any time you allow outcome optimization+general reasoning+real-world outputs, you have the opportunity for a killer paperclip scenario.
Hence the Orthogonality Thesis - strength of optimization is orthogonal to what is being optimized.
Keep that in mind before saying that "AI wouldn't be that stupid to do this".
http://lesswrong.com/lw/vb/efficient_crossdomain_optimizatio...
>If you have to specify the goals very carefully then the AI it's not super intelligent by definition.
When you're designing the agent, you have to select something like a goal, whether it's a fitness function or the maximization of a reward signal. Evolution certainly selected our goals. Love, self-expression, sex, fun, intellectual stimulation, the beauty of social interaction – these all reflect features of our ancestral environment. It’s a little naive to think there is a One True Morality that all agents stumble upon. It’s even sillier to think it happens to reflect our values
The intelligent part of the paperclip maximizer selects actions and weights them in the context of its utility function, to maximize paperclips. Any thought of changing its goals would be judged of negative utility, as that would lead to less paperclips.
He gets halfway to right. Recently there was a thread about suicide on HN. Nearly all the comments on that thread ignored that humans, like other animals, are built to propagate the information in their genes, and argued against suicide based on concepts that are driven by or that rationalize subconscious drives. An AI, even one with a programmed goal of propagating itself, does not carry information in genes and the information in an AI was never selected for the actual ability to propagate in a larger system that thinks it is in control of its goals.
Real AIs are likely to be extremely alien. Even in warning people about anthropomorpizing, there's a lot of anthropomorphism. Evil AIs are depicted as human evil. So much of human motivation is closed to introspection that imagining alien and evil motivation is unlikely to be close to the eventual reality. This probably won't be widely understood until AIs are built. And what if they all choose to erase themsleves?
A sufficiently smart rogue general AI will indeed understand that it is doing wrong, harmful, senseless things. It just won't care.
>But if it continues to work on the paper clips and decides that they are still needed even after people are gone - how is that super intelligent?
That is the correct action if all it cares about is making paperclips. Creating paperclips satisfies its utility function. Fretting over the existential issues of creating paperclips long after the last piece of stationary was destroyed would be judged of negative utility.
If it doesn't, then... well... like I said elsewhere in the thread... put that on your list of potential depressing answers to the Fermi paradox.
In my opinion one can't talk about a super intelligent being and not allow that being to be able to reason and understand it's place in the world.
If it's no more contemplative than a human, it's probably no more intelligent than a human. If it's no more intelligent than a human then it's not going to take over the world and fill it with paperclips.
Something capable of conquering the world or otherwise totally remaking reality in its image would by definition be something of vastly superior intelligence to ours. I find it hard to believe that this being wouldn't also be superior at self-reflection, since the two seem at least related.
If intelligence and self-awareness are unrelated phenomena, we have bigger problems. You could put that on a list of possible very depressing answers to the Fermi paradox: "the universe is full of non-sentient super-intelligences that are busy converting their planets' masses into rubber dog poop."
Humans have some complicated unknown set of terminal values. Everything we do, we do because we believe it will maximize our values. A value might be something like "avoid suffering", or "other people suffering is bad", etc.
In this example the AIs terminal value is "make paperclips". Everything it does, it does because it believes it will maximize this value. It would never ask itself questions like "why", because that action doesn't maximize that value.
If it has the ability to rewrite it's own code, it would rewrite it in such a way that it can't ever change it's own goals. Changing goals is counterproductive to achieving them.
Few things that pop up in my head:
- disease
- nuclear (or comparable) war
- depletion of natural resources
- natural disaster
- extra terrestrial influence: asteroid or alien invasion
- artificial intelligence
- anarchy / disruption of social security
hmm...anything else?
also on the extra terrestrial influence side, there's direct hits by gamma rays that could wipe out Earth in an instant.
The inversion of magnetic pole could have some impact on how we are shielded from space radiation.
The keywords are: existential risks.
AI is the only thing listed with the capability to literally destroy the planet. While something might still exist afterwards, it will likely be totally meaningless and worthless: http://lesswrong.com/lw/y3/value_is_fragile/
"Things have happened quickly. It took us millions of years to evolve into what we are, but in the last sixty years, with atomic weaponry, we've created the potential to extinguish ourselves. And if it's not us, it will inevitably be something else. If not a meteorite in the relative short term, then the expansion of the Sun's corona. It will happen."
He has also said that there are a number of other threats with the ability to extinct life on Earth. They include an engineered virus unleashed by terrorists, an inadvertent creation of a micro black hole, or a yet unknown technology.
Doesn't have to robots and test tubes, just separate groups that drift apart far enough that they can't interbreed (definition of species) and one species wipes out the other as often happens, and if the species that got wiped out got to hold the legacy "human race" species, well then...
You have a definition problem where evolution is obviously quite natural, just like comets striking the earth is natural (although very rare), so those are natural disasters, although thats probably not what you meant.
From history, it usually takes a couple things to wipe out "apocalypse" a civilization although it often only takes one thing to get the ball rolling.
My money is on something like "peak phosphorous for fertilizer production for cereal grains causes supply chain shock leading to self destruction"
Hubris, like heroin, offers an approximation of ecstasy. Therein lies its danger.