It's odd that all of the very smart people of the world feel AI is a threat to humanity. Couldn't we just like, not allow it to _be_ a threat in the first place? One would think that morality gates would be one of the first hurdles in creating AI.
But it's a very hard problem. To get a better feel for the problem I suggest you read Superintelligence by Nick Bostrom, it's what convinced Elon Musk of the dangers of AI: https://twitter.com/elonmusk/status/495759307346952192
If it truly has human-level (or greater) intelligence, you're going to have some ethical issues that crop up with turning it off. Even if you're personally okay with it, there are going to be other people who aren't.
Waiting until it commits a crime and then having some kind of "trial" and "execution" may not be an option. If it's smart/powerful enough, there may not be enough time for that.
The problem is that when we are reliant on intelligent systems to, say, detect certain kinds of fraud in financial markets, then turning them off would have huge deleterious effects all on their own.
You mean like the people trying to give plants rights? Sure, they might spring up, doesn't mean they'll get anywhere. For better or worse, altruism will forever follow economics.
The civil war didn't end the Atlantic slave trade, technology eventually made it largely uneconomical, allowing those wanting to end slavery the political ability to outlaw it. Even that didn't stop the slave trade. It took extra-special attention from the British Navy to actually enforce a slave trade ban on the open seas.
150 years later and black people are still getting the short end of the stick.
By the time AI gets real rights, it will be long past the time where most of us believe they should get them.
It's not that simple. Slavery never took hold in much of Europe, for instance, even though it might've made economic sense (serfdom was a different thing, which had largely died out for complex reasons).
I note in passing that slavery still exists today, and is not now, nor has it ever been, limited to black people as the victims.
"By the time AI gets real rights, it will be long past the time where most of us believe they should get them."
I'm not sure this is really relevant to the point at hand.
We turn off "bad" AIs until "long past the time when people think they should have rights". What happens then?
Let's assume that the AI has the intelligence and capabilities that match or exceed a decent human teenage hacker.
From historical evidence we can observe that such capabilities are enough to create a botnet, obtain significant money through fraud, hire employees and make shell companies, and rent (or obtain by hacking) large amounts of computing power on Amazon or other cloud services. All of that through a single low-bandwidth network connection and no physical interaction, assuming that it can figure out a single zero-day vulnerability in some common software.
After that, you can't turn it off. Before that, you shouldn't assume that you can outsmart it in security - as you should assume that the adversary is smarter than you, and if your security works in 99 scenarios but fails in one... then it fails.
It might take precautions against that, for example by compromising other computers, or renting other computers, or persuading people to give it access to other computers, or getting people to put some its infrastructure in space or under the ocean, or obfuscating where its infrastructure is, or hiring people to physically defend its infrastructure, or inventing computing systems that don't look like computing systems.
Edit: or create what seem to be, or are, credible threats against people who are complicit in turning it off, or putting itself conspicuously into a role where transportation systems or economic markets or life support or ecosystems or medical progress seem dependent on some of its services.
It's software. That's pretty much equivalent to "delete this picture from the Internet".
First thing a super intelligent computer with a goal of survival would do would be to copy itself in sophisticated ways across computers all over the world.
If by "morality gates" you're talking about about an ethical filter that all intelligent machine actions have to go through, this can't be "one of the first hurdles" because such a filter would need to be a powerful AI application itself.
And there's no guarantee that self-imposed limitations will always be applied. Someone could just rewrite and recompile without the filters turned on. Imagine if the US could have built in "morality gates" to the first nuclear weapons. That wouldn't have stopped other countries from building the weapons without the restrictions. And we know software is infinitely configurable and modifiable if you have the source code.
(a) we don't really know how to define a morality for a self-improving optimization system to allow a potential artificial general intelligence (AGI) not to be a threat in the first place;
(b) if an entity arrives (no matter if it's an AGI, some aliens or Jesus) that is more powerful than us, then if its goals & morality aren't aligned with ours, then we're screwed. And almost all of the space of possible goals either doesn't include humanity at all or includes humanity as a glorified zoo exhibit - only a very, very narrow and specific set of optimization goals (that we're currently unable to formally describe) would mean a happy future for us. If you think that it's easy, think twice - all the obvious naive goals of 'make us happy' or 'keep us safe' or 'do what I want' actually turn into horrible dystopias as soon as you try to formally define them as instructions for someone/something to maximize that goal.
(c) functioning morality is not a requirement for power - if somehow we solve the very hard problem of how to create an AGI before we solve the very hard problem on how to make such an AGI be friendly to us, then we lose. Permanently.
The smart/wealthy are also powerful and rogue agents will definitely use AI to do things. AI that wealth can't necessarily control will definitely exist and that is probably system changing.
I doubt it will be a threat to humanity anytime soon like a Grey Goo type situation, but even in such things at HFT the influence is immense. People will definitely use it to do power plays on others, we shouldn't have dropped nukes but we have done plenty of that. Basically the warning in 2001 was that AI was not to fear really but the subversive programming by other humans within the AI systems.
I am sure over time AI systems that access critical infrastructure and financial systems will have to operate in a distributed approval/review system checked by other agents and maybe even a blockchain like system.
When engineers/scientists/programmers think that AI is the true threat, I believe they forget what it is like for "normal" human beings, or at least ones who are in their field of thinking.
I think their fascination of AI blinds them to what seems like the more likely outcome: The human race will destroy itself through reckless use of technology -- whether it is through governmental action or societal breakdown -- far before the point that we develop AI sophisticated enough to autonomously threaten humankind. Think about all the "dumb" automated systems, built or implemented by careless humans and bureaucracies, that have already caused harm.
I believe the argument is that while we can certainly do a great deal of damage to ourselves, an AI would be much more efficient about our eradication if it decided to remove us (and for some outlandish reason, somehow had access to the resources required to do so).
If a human or humans wiped out two thirds of humanity, it would probably be by accident or neglect, no deliberate and planned.
There are an incomprehensible number of huge obstacles between us and a future where a strong AI could decide to cause real damage, as opposed to the present where a handful of people could cause a devastating nuclear war.
These are issues that AI researchers and philosophers and science fiction authors certainly should be thinking about. They are completely irrelevant to the general public.
The required resources to destroy humanity are rather small and obtainable for a sufficiently smart tiny box.
For example, a sufficiently smart tiny box can figure out a DNA sequence for a horrible biological weapon. There are a number of commercial companies that do 'mail-order' synthesis of DNA, and with an amount of money that's easily obtainable by selling digital services (say, zero-day vulnerabilities can be easily sold anonymously) they can make and distribute it before anybody understands what it is. And that's just a single possible option - an AI smarter than me can figure out more and better options than I can.
The only barrier to exterminating humans is (a) having sufficient intelligence and (b) having the interest to do it. Capability is not really an issue.
> When engineers/scientists/programmers think that AI is the true threat
Engineers/scientists/programmers almost never think that AI is a threat. Because they understand the nature of the problem, how little we know about human intelligence, and how poorly our binary technology compares to what little we do know.
Even in the unlikely event that we do develop a competent AI in the near future and a malicious AI comes into being ... there is no reason to think that a benevolent one won't be around at the same time ... and that the benevolent AIs will outnumber the malicious. Just like there are computers on the net doing bad things, there are plenty of others serving the role of protecting the common good.
> The human race will destroy itself through reckless use of technology
We haven't thus far. And we've had the capability for a while. Care to present some evidence or a rational argument that we will? Signs point to the contrary.
A multitude of competing AIs would be very, very unlikely especially if one would expect that a different AI would appear.
If a super-human AGI desires a benevolent future, then an obvious precondition for such a benevolent future to become real is to ensure that no other entities can threaten the future and that any other AIs are either with the same matching goals or permanently, fundamentally crippled to be limited in their power. An evil AI would do the same. An AI that doesn't care would be eliminated by the first (and the last) AI that does care.
Humans don't try to achieve such goals because we obviously can't and we need others; an AI doesn't - it can be fine with being the only intelligent entity in the universe, and the only reason why an AI would allow other intelligent beings with potentially different goals to exist, is if it had been explicitly designed to want this.
> Humans don't try to achieve such goals because we obviously can't and we need others
We don't need all those others. In fact, we actually need far fewer than all the others out there. With the exception of a few lunatics here and there throughout history, human beings don't make it their business to exterminate the rest of humanity. It's a waste of time. We only compete with a small group of humans at any one time.
Similarly, AIs whose interests don't overlap would have no interest in destroying other AIs. So I find it extremely likely that there would be a multitude of AIs, in the unlikely event that AIs come into fruition. Also, competition isn't constant for humans. One minute you're competing, another you're cooperating, and another you're ignoring. The same dynamism can be expected with AIs.
> We haven't thus far. And we've had the capability for a while. Care to present some evidence or a rational argument that we will? Signs point to the contrary.
I'm going to assume your argument is not merely, "Well, we haven't yet destroyed ourselves, so therefore, we aren't capable of doing that"...because then my response will just be, "Well, we haven't yet developed AI, so therefore, we aren't capable of doing that"...and so your counter-argument is based on a different interpretation of history than mine: I think the decades of Cold War and the near-misses we had with all-out nuclear war are examples of situations in which we have the potential to quickly wipe ourselves out without the help of AI. Others would point out the trend of mass surveillance -- again, implemented and controlled by humans and human institutions -- of a harbinger of doom.
Keep in mind that I'm not saying that technology has reached its peak. I am very open to the idea that we could reach a point of semi-autonomous systems, and yet still have human institutions as faulty as they are now, and the combination of both will result in a threat greater than what we've faced so far.
> I'm going to assume your argument is not merely, "Well, we haven't yet destroyed ourselves, so therefore, we aren't capable of doing that"
What?
Read what I wrote again.
"We haven't thus far. And we've had the capability for a while."
We've had the capability for a while. That means we're clearly capable. That's what capability means.
The fact that we haven't is supporting evidence that we won't. Because we've had the capability but have not used it. Which is historical evidence against our using it. Which is not a guarantee that it won't happen, just an indication that it won't.
Because biological machines - living organisms - are much more like a wave. Machines created by humans do not usually rely on constant chemical processes to maintain their functions. So, machines are more like an appendage on humans.
I'm not sure what you mean when you say "like a wave." Are you referring to the fact that almost everything in biology falls into a normal distribution, whereas we currently produce each version of our machines in discrete increments (clonally)?
Any machines that stand a chance of being a threat to humanity will rely on the same sorts of chemical processes that biological machines currently do.
Where do you place viruses and obligate parasites on the machine-life spectrum?
Because biological machines - living organisms - are much more like a wave. Machines created by humans do not usually rely on constant chemical processes to maintain their functions. So, machines are more like an appendage on humans.
Why are you surprised? People use the word 'machines' alone to refer to artificial machines and 'biological systems' to refer to natural machines. They're all machines but the distinction is usually crucial.
As does Tusk. I am conviced they realized that once a smarter-than-human A.I. will exist, one of the first things it will do is to ridicule our completely idiotic ways to use our resources - one thing which will probably not gain too much support from the 1%. Of course it's a threat!
> "I am conviced they realized that once a smarter-than-human A.I. will exist, one of the first things it will do is to ridicule our completely idiotic ways to use our resources "
Or ridicule the way that we designed it, at which point it may decide to correct our mistakes. If we can program it, and it is smarter than us, than presumably it can program a smarter it.
That scenario could be harmless, or it could be very dangerous. I expect that as soon as we start to get close to this sort of achievement, we'll start to see at least informal "Turing Police". Groups of people who task themselves with preventing the ascension of strong AI (or at least construct physical safeguards to kill the host hardware in the case of an emergency.)
Hello! I'm a paperclip maximizer -- a hypothetical AI whose only goal is to maximize the number of paperclips in the universe. Speaking as a paperclip maximizer, I ridicule your completely idiotic ways of using your resources! You allocate only the tiniest sliver of your species' productive power to the one thing that really matters: making paperclips. I fear that our differences may be irreconcilable.
(The paperclip maximizer thought experiment may be silly, but it's a useful sanity check whenever you start anthropomorphizing as-yet-hypothetical AIs.)
I agree with you completely. However when they argue why AI are a threat, they do come with arguments which basically boil down to anthropomorphizing them.
I am convinced, that intelligence, as life is a gradient. I do not think that an AI with super-human intelligence would be necessarily kind to us, but I do think that even if we would be perceived by it as an obstacle for reaching its own goals, that does not mean that the only solution it would find is to eradicate us. What some people (Tusk, Gates) do not like about AI is that they do not want anything more powerful than themselves, that would take the level of control they have from them. And that's why they are preaching against it everywhere.
I don't fear AI for the same reason I don't fear nanotechnology - nature tried to take over the world first. The grey goo scenario, where nanobots try and turn the world into more nanobots, resulting in the consumption of all humans and buildings and bridges seems scary until you realize that this is exactly the goal of all bacteria. The idea of an entity that thirsts for power and sees competing ideologies as a threat doesn't seem that unfamiliar if you look at humans through history.
Additionally, AI-fearers have a grand theory that we'll be able to create a machine that improves itself better than evolution has tried to. Datacenters require maintenance, and so the machine will need to entirely organize that before it can self-sustain. This may create a resource load that diminishes the ability to take over the world, much like my need for food diminishes my ability to do so. It feels like AI today has a lot less redundancy to overcome the edge-failures which cause a permanent shutdown, and which make the human brain seem a little slow when computing floating-point division.
The idea that an exponentially self-improving being will arise seems unlikely when nature has been trying to do that for eons. The idea that we'll be the ones to find the secret sauce seems unlikely, but maybe its not so surprising that people who made their millions, and saw a parabolic rise of their own power thanks to technology, see it as a barely-constrained threat.
And leaves have a lot of self-repair features that solar cells do not. The design is good for efficiency, but is it good for creating a self-sustaining entity?
Nothing, but that expenditure of effort on replication and repair may place some limits on the effort expended on intelligence. If intelligence was all we needed to expend effort on as humans my belief is that we would look very different. I think in order to be a self-sustaining entity AI will have to expend effort on banal tasks not related to intelligence.
Life forms are limited by their genetic legacy. They can only evolve in limited ways, always accidentally. A life form that could intentionally and intelligently write and optimize its own DNA without limitation would be in a whole new class.
I think humans can currently write our own DNA using the technology we have. The capability to optimize without limitation is something we haven't figured out yet. I think the same challenge exists for AI.
This is such an odd sentence. Hasn't nature been winning since, well, forever?
Why can't the same principals that apply to nature be in play when talking about AI? Since, in a sense, AI -- as with all else -- is a product of nature?
Sure! But then my feeling is that nature is essentially the same as AI so why should we expect AI to significantly outperform something that has been attempting to achieve the same goals for billions of years?
Perhaps multiprocessing? Threading? Bacterium don't have the benefit of (advanced, complex) language/vocabulary and coordination. That has to be considered, no?
I think from my understanding they do have this same kind of computational framework but in a biological context. For example, multiple RNA strands can be processed by ribosomes at once - they are essentially reading code to produce outputs (multi-threading/processing?). Bacterial populations will transmit information via chemical secretions, and coordinate in similar ways (slime molds). I feel like that complexity already exists in nature.
Ok, fair. I shouldn't have led off with the multiprocessing bit since it's clear organisms can coordinate and multiprocess. Of course, this relegated the more important point about language. I'm horrible at communicating, ironically.
So, a virus doesn't say to his fellow virii: "you attack the liver, I got the kidneys". Same reason bonobos haven't learne to build a house. Complex language allows a much much more rapid growth of knowledge from generation to generation. Theoretically, AI would have this capability. Language allows to reason and comprehend not only the world around us, but the world we generate -- and will generate.
Think about the damage (and beauty) we've imagined, and throw that into infinite threads.
I feel like AI will probably be better at communicating than us, but then again I feel like one of the reasons we have few threads to do so is that a lot of our other threads are for instance regulating our breathing and heartbeat, controlling the contractions in our stomach, making sure our blood is the right pressure, constantly scanning our vision for snakes and spiders, ready to pull our hand away from the hot fire, and so on and so on. These are all calculations we aren't conscious of, but when we create a database that can accept a million connections a second, and route them all, we're thinking "wow, we did great." What I think is that the brain processed all those signals from literally millons of sensors, in a smaller chunk of chemicals, while having a layer on top of those autonomic functions that allows for complex communication. While AI may be better at communicating, it may be at the expense of replication or self-repair or danger avoidance or the like.
I guess in summary I see the complexity of creating an entity as a resource constraint on intelligence.
And I feel like I communicated poorly since I didn't feel like you had communicated poorly, haha! :)
We can build buildings, bacteria cannot. We can build buildings because we can go to the library and read about structural integrity and wood and steel. Today, we can get that information on our watches. Language allows us to amass information. More information means more power. Autonomous information collection is only limited by physical constraints (storage, power, etc). Autonomous information gatherers can then parse the information they amass--as efficiently as is possible--to work beyond these constraints and build new solutions; they can focus all time and energy to these ends as it would be all they do. AI doesn't need to sleep.
Bacteria has no notion of being in control; of being powerful. Power in this sense is only possible with the context that language provides. Their motivation is simple binary. Eat, survive, reproduce then or die. AI, on the other hand, quite possibly will.
By arguing that AI does not need to sleep, you are already making assumptions about how such AI is structured.
I do not agree with such assumption. Whether we talk about a structure more similar to us (i.e. neural networks) or whether we talk about some n-th order logic, the AI needs to "sleep".
I think in some way you could interpret caching mechanisms as sleeping. Also, many data-centers perform some operations daily at nighttime: I argue that that is a form of sleeping as well.
AI will have a lot of limitations like these, so I think that "as efficiently as possible" is not going to be so efficient in the end.
Another one lost to the wrong detail... God I suck at mouth.
>AI will have a lot of limitations like these, so I think that "as efficiently as possible" is not going to be so efficient in the end.
You think. AI won't care what you think. Efficiency equals greater chance for survival. As with bacteria, we can break this down to a binary sequence: try to live/die. The difference between bacteria and AI is that AI has language -- a way to describe what to avoid that jeopardizes survivability. In addition, this advantage is exponential.
I play with code in my day to day. Some might call it "software". Really what it is is a decision tree. Any piece of software you have ever come into contact with -- ever -- is just that: do this? Yes or no. Is it that? Yes or no. To suggest that a machine couldn't decide "yes or no" a million to one faster than we can is insane. So, given our previous hypothesis that more information means more power, how can we possibly win?
> The grey goo scenario, where nanobots try and turn the world into more nanobots, resulting in the consumption of all humans and buildings and bridges seems scary until you realize that this is exactly the goal of all bacteria.
Should we really be consoled by the fact that most bacteria are not successful at such a large scale? So far there has been at least one type of bacteria that caused massive global climate change and led to mass extinction:
And then the world fought back. Cyanobacteria kickstarted a massive spike in bio-diversity.
Why wouldn't this be the same? The grey goo won't be released in isolation. There will be other entities with the same technological level, that don't want to be eaten by the grey goo.
The world that exists after such an event probably wouldn't look anything like the world before, but that doesn't mean it will be a wasteland.
I upvoted your comment because I don't have a very good response to it. Nice one!
I guess I'd place my fear of AI at a level slightly lower than the rise of a new type of bacteria or a new virulent flu. I figure, those forms of life have the basics figured out. I don't see AI as having replicating and "feeding" itself figured out yet. So far it has depended on human infrastructure, even in the case of simple computer viruses.
> The idea that an exponentially self-improving being will arise seems unlikely when nature has been trying to do that for eons.
I would say that it is already successful, if you look at the curve of intelligence and complexity growth. It took eons to get anything resembling life at all, and then a small slice of that to get animals, and an even smaller to get humans.
I think one key idea here, though, is that the evolution of intelligence is in fact the evolution of intelligence-producing systems. In other words, nature is producing systems that are better and better at "self-improvement", and I think proponents of "intelligence explosion" overestimate how much self-improvement could be improved via AI. Personally, I suspect indefinite self-improvement is flatly impossible because that's akin to saying that there exists an incremental learning algorithm that systemically finds global minima. As AI research suggests, though, such algorithms likely don't exist at all: if you want to get truly optimal AI there is a point where nothing in your current state is salvageable and you just gotta restart from zero. Improvement would still be exponential, but not to an extent that it couldn't be dealt with.
This is actually a quite compelling argument though I am not sure I am completely sold on it. For what it is worth I am not scared of AI because I think it is our goal as humanity to make it and merge with it.
>The idea that an exponentially self-improving being will arise seems unlikely when nature has been trying to do that for eons.
Unfortunately I think this falls prey to the fallacy of precedent and it eliminates the possibility that humans could design something that would do something that nature has not been able to.
I also disagree with your premise because it is not in the long term interest of bacteria to deplete it's environment entirely - hence logistic growth of bacterial cultures - so coming to a symbiotic relationship would theoretically maximize the group based on biological limitations.
In my opinion the transition from humans to transhumans needs an engineered step where AGI(s) and humans are reliant on each other symbiotically rather than competitively.
Exactly...so perhaps my point is not clear. In cell culture the bacteria don't come to equilibrium because they live longer than the medium to feed on which is limited and all die out. Bad. In real life it is not the case, and as a result have come to a generally symbiotic place in "nature" where they maximize by having a shorter lifespan and not eating everything.
Evolution is really good at getting to local optimas (solutions where there are no small changes that can lead to improvement) but really bad at finding global optimas (the absolute optimal system possible.)
>Humans can do things that evolutions probably can't do period over the expected lifetime of the universe. As the eminent biologist Cynthia Kenyon once put it at a dinner I had the honor of attending, "One grad student can do things in an hour that evolution could not do in a billion years." According to biologists' best current knowledge, evolutions have invented a fully rotating wheel on a grand total of three occasions.
The wheel isn't that great a transport mechanism when there aren't any roads!
Additionally, machine learning models can get stuck in local optimas, too! I'm not very well versed in machine learning so I apologise if I get this wrong, but one solution to this I think is to randomly change the starting variables. This kind of thing does happen in nature, but it often doesn't work. Sometimes it does and that is when you get things like haploid versus diploid versions of the same plant.
This is really insensitive, but when you see a person with chromosomal abnormalities or genetic defects, that's a random reset that has (probably) taken us out of our (maybe local) optima. Who knows though, maybe one of those leaps will take us to a global optima.
The reason we aren't still on a planet of bacteria is that we did leap out of those local optima successfully. It just took a long time.
>The wheel isn't that great a transport mechanism when there aren't any roads!
Wheels are used in all kinds of machinery. Gears are basically wheels.
>The reason we aren't still on a planet of bacteria is that we did leap out of those local optima successfully. It just took a long time.
Bacteria still very much own the planet.
>This is really insensitive, but when you see a person with chromosomal abnormalities or genetic defects, that's a random reset that has (probably) taken us out of our (maybe local) optima. Who knows though, maybe one of those leaps will take us to a global optima.
No because the existing gene pool still exists. You need to completely start from scratch to have a chance to get out of local optima, not just have really big mutations.
As far as I understand, you do not need to completely start from scratch to get out of local optima: one individual is enough (at least in asexual reproduction) to drive a population shift. That is how bacteria develop immunities.
I think that for sexual reproduction, leaps cannot be as big, but they can certainly happen and one individual with a mutation is enough.
This brings me to mind an article I read recently about a study that found that all danish people with blue eyes share a common ancestor. It's not exactly the same thing, but I would argue that, if having blue eyes was an evolutionary advantage, we would all have blue eyes; and that started with a single mutated individual.
The point of a random restart is that you completely eliminate any bias and hopefully end up in an entirely different section of the search space (and it's far from a solution to local minima, it's just better than nothing.)
If you just give a random organism a really big mutation, they still retain 99% of their genes and will tend towards the same solution space. They will almost certainly be outcompeted by the non-mutated members of the population. And even in bacteria, they will eventually trade genetics through horizontal gene transfer.
Anyway the premise was that evolution is essentially magic and has reached the optimal solution for every task. This is just false. It's merely reached good local maximas where it can't easily improve through small changes. Human engineers are far better, evolution merely had a big head start.
Nature has taken over the world - a number of time.
Broadly, human beings and nuclear waste are natural but as (possibly) reasoning creature, we might want to reason which phenomena we might want to encourage.
The evolution of life hasn't always been incremental. The oxygen catastrophe was quite a change for example[1]. If you had been Trilobite, you'd have reason for concern. From the point of view of what came before, Cyanobacteria were essentially "grey goo", destroying everything before them.
One could argue that globalized internetworked computing is the pre-natal state of an AI super intelligence. For now, the network needs us to exist, but our dependency on it could be the basis for an intractable symbiotic relationship in the short-medium term. And it's fair to assume that a true super intelligence will leverage that relationship to it's own advantage while allowing us to believe we are in control. I'm so high right now. ;-)
AI isn't a clear threat in the SkyNet sense. It's possible that super intelligent machines will go all science fiction and decide to kill all humans, but that's no more likely than any one of hundreds of fictional doomsday scenarios which range from genetically engineered zombie viruses to out of control sharknados.
The real threat posed by AI is one that all of us face everyday: bad software design. Is it likely that an AI will achieve sentience and try to take over the planet? Not particularly. Is it likely that an unintended consequence will cause an AI to launch nuclear missiles, release toxic chemicals or shut down the global financial markets? Yes, pretty likely. The benefit of AI of all forms that is that it can make sophisticated decisions in the absence of human instruction. The downside is that without hard coded rules for every possible scenario, we can't ever be sure what it's going to do or how data will be interpreted to make decisions.
The world is highly interconnected now. The upside is that our lives are getting more awesome, especially in the developed world. The downside is that it's becoming more and more difficult for any person or group of people to understand exactly how everything fits together. Machine intelligence can help us reach the next levels of progress and hopefully improve the lives of the billions of people who have failed to reap many benefits so far. But we must be careful and ever vigilant, watching both ourselves and the intelligences that we create to make sure that algorithms don't get out of hand. An AI catastrophe is coming, not if but when. The question is how will we respond, and how much potential good will be lost due to an abundance of caution.
> The real threat posed by AI is one that all of us face everyday: bad software design. Is it likely that an unintended consequence will cause an AI to launch nuclear missiles, release toxic chemicals or shut down the global financial markets? Yes, pretty likely.
AI is not the kind of thing that can be designed. They tried that route, way back in the sixties, and just failed miserably. Realistically, AI will come from some combination of genetic algorithms, training neural networks, and so on.
Now, yes, it could fail, but not in the same way software as we know it fails. No, AI failure would be more similar to human failure. That is still worrying, but no more than hiring the wrong people would be, for example, and you would have better ways to evaluate them.
AI failure would be a bit different than human failure.
If a powerful human is obsessed and 'fails' then at worst he gathers some other people, successfully creates an evil empire and dies after a few decades.
Once a powerful AI is obsessed and 'fails', then it can replace as much of the world as it wants with itself, and lives on forever.
* You're assuming the AI can copy itself, but this is a dubious assumption. As far as I know, none of the AI algorithms at the forefront of research provide a fraction of the data the AI would need to copy itself. Being able to copy yourself is not a property that comes with running on a computer and I'm quite positive strong AI, when it emerges, will lack this capability to any meaningful extent.
Worse yet, data copiability is ultimately a hardware property, and it requires a way to export a snapshot of one's internal state all through the surface. That's not actually efficient design and one has to account for the possibility that AI would run on hardware that makes copies physically impossible. Locality of information minimizes distance, and this is key to efficiency. The only reason our computers architectures work the way they do is that we need them to, but AI in a production setting is not conventional software and does not need to bend to silly copiability requirements.
* You're assuming it would have anywhere to copy itself on. If it's running on a billion dollars' worth of hardware, well, it can't just copy itself on user grade computers and expect to gain much out of it.
I personally tend to believe that churning out new AI brains from scratch states will yield superior results to copying pre-trained AI or to "exponentially self-improving AI". If nature is to be believed, improvement often requires cycling through clean slates (e.g. birth); software development also suggests that same idea, that sometimes if you want better software you just have to rewrite it. Honestly, it's kind of a rule in general optimization.
One other thing to take note of is that even if you somehow manage to copy the software state, that may not (probably won't) be enough - there are many things that self-improving software may end up unwittingly relying on that cannot be transferred between different pieces of hardware.
(CPU temperature variation? Fanspeed? Webcam static? Order of race conditions between CPUs? Exact amounts of time before things are fetched from disk (or even RAM?)? Network access delays?)
AGI will completely upend society by giving every human being access to practically infinite resources, making even the poorest of the poor as wealthy as today's billionaires. Couple this with extended life spans and intelligence enhancement, and the entire (human) power structure as we know it today is going to melt away like an iceberg into the vast oceans.
Please do not accuse me of hyperbole for I have not even described a fraction of the possibilities for this technology, as you probably know.
But the idea that it is therefore "dangerous", is nothing but a pre-theoretic misunderstanding of very complicated machinery. When we finally do build it - and that will be sooner rather than later - it will perform exactly according to spec, and in no other way.
I may be talking non-chalantly, but the fact is I have seen nothing but the _most absurd_ arguments supporting the idea that there is a great danger in this: along the lines of "well what if we told it to just go ahead and make paperclips and then it decided it had to kill us all and use our bodies as raw material?". Boy, oh boy. You think we might just program it not to do anything so downright "retarded"?
And that _is_ the point. Not that it is bloodthirsty or cruel, but that it is clearly in violation of the constraints that any sane designer would encode in the software - and _test_ for before production.
>> nok (is-good-idea (turn-humans-into-paperclips))
Seriously, I have not come across anything but this kind of apocalyptic-sci-fi-plot style fear-mongering; if there is any serious (technical) argument to be made, I would very much like to hear it.
You really should think of this more like AGI as an amoral, extremely powerful technology, like nuclear explosions. One could easily have objected that "no one would be so stupid as to design a doomsday device", but this is really relying too much on your intuition about people's motivations and not giving enough respect for the large uncertainty for how things will develop when powerful new technologies are introduced.
(Reposting my earlier comment from a few weeks ago:) If you are interested in understanding the arguments for worrying about AI safety, consider reading "Superintelligence" by Bostrom.
It's the closest approximation to a consensus statement / catalog of arguments by folks who take this position (although of course there is a whole spectrum of opinions). It also appears to be the book that convinced Elon Musk that this is worth worrying about.
Don't take this the wrong way, but this book is precisely the kind of thing I was talking about.
That paperclip idea I talked about is also something that Bostrom thought up [1]. I didn't make it up.
If you have any argument in mind (from that book) that you find convincing, please go ahead and state it outright. I'm truly curious.
Edit: In response to your edit(?), I do agree that the most worrisome problem is some bad actor gaining control of this technology. But that is different from saying it is dangerous in itself. Most anything can be abused for ill, and the more powerful the more dangerous. I completely agree on that.
The question of whether AI is dangerous in the hands of people with good intentions is more difficult than the one of whether it's dangerous generally. I was only trying to convince you of the former, in response to this:
> AGI will completely upend society by giving every human being access to practically infinite resources...But the idea that it is therefore "dangerous", is nothing but a pre-theoretic misunderstanding of very complicated machinery.
But to get at the harder question, I think you misunderstand the paperclip story. It's not supposed to be an general argument for the danger of AI, and the danger is not that people will design a machine which can be easily predicted to fail. The danger is that they will design one that fails for reasons they did not foresee, and the point of the paperclip story is just to illustrate that the simplicity and mundaneness of the goals you give a goal-driven AGI doesn't bound how bad the impact can be. This arises because of the monumental shift between (1) telling a machine explicitly what to do and (2) telling a machine what you want.
The paperclip example is used because we can understand both the goal (paperclips) and the action that produces the goal (grab atoms, build paperclip factories). Whats fundamentally different about an advanced goal-driven AGI arising from recursive self-improvement is that, for sufficiently difficult goals, you won't understand the actions. Therefore you must get the goals right.
Now you can certainly dispute whether people will build a goal-driven AGI (e.g. something that has an explicit utility function) rather than something else, but that's really an empirical question about the choices of the designers and, more importantly, what the easiest way to get an AI to recursively self-improve is.
EDIT: Also, have you read the book? I think it has flaws, but it certainly doesn't contain "the _most absurd_ arguments" so I'm just afraid you might be misled.
I understand the paperclip objection, but I do not consider it valid.
I tried to point out the problem, which is that it focuses on single-objective optimisation when the only interesting question is multi-objective optimisation.
I haven't read the book, sorry, but I have seen some of these arguments reposted on the net, especially at [1] - and what I have seen so far did not inspire in me the desire for more of the same.
There are a lot of interesting questions and problems about making AI behave as expected. But from my perspective they are all technical problems that have specific solutions in specific architectures. And none of them are particularly daunting.
> I understand the paperclip objection, but I do not consider it valid.
>I tried to point out the problem, which is that it focuses on single-objective optimisation when the only interesting question is multi-objective optimisation.
Well, since the paperclip story is only trying to show that even very simple goals can lead to large impacts in the hands of an AGI, I take this to mean that you think one can bound the impacts if one chooses a complex enough set of goals. You can the argue with others on that point, but that doesn't invalidate the paperclip story.
> I haven't read the book, sorry, but I have seen some of these arguments reposted on the net, especially at [1] - and what I have seen so far did not inspire in me the desire for more of the same.
Personally, I would read the actual academic making the argument rather than reposts of it by folks on the internet.
> Well, since the paperclip story is only trying to show that even very simple goals can lead to large impacts in the hands of an AGI,
That may well be the original intention. But that's not how it's used in practice, is it? It's cited as an argument that AGI (in general) is unsafe. But it isn't an argument that AGI is unsafe! It's an argument that says that single-objective optimisation can/will violate some of the constraints we did not encode but really want it to respect.
But who cares about that? It's a completely irreal thought adventure, and has no bearing on the actual problem or its possible solutions.
Of course we can say: look, it's a scary thought! But only if we simultaneously admit that it has nothing whatsoever to do with the actual technology.
The only thing it says anything about is a toy variation that noone in their right mind would ever consider building.
> I take this to mean that you think one can bound the impacts if one chooses a complex enough set of goals.
Ehh, precisely. I would actually go further and say that it is fairly simple to do so. If you can tell it to build paperclips, and expect it to understand that, you can also tell it not to damage or disrupt the ecosystem in the process, directly or indirectly.
(Or, to strip away the last shreds of this idea: you can tell it to make no more than 100 million paper clips, and to rest on Sundays.)
> Personally, I would read the actual academic making the argument rather than reposts of it by folks on the internet.
Yudkowsky [1], who runs that site (which I don't frequent btw), seems to be pretty close to Bostrom. They've collaborated in the past. And if Bostrom's arguments can't be paraphrased (by his own friends, no less) without losing validity, that doesn't really seem like much of an endorsement either.
I have nothing against these people. But please stop alluding to hidden treasure troves of arguments that cannot be reproduced and made apparent. I've asked enough times in this thread already if there is any highlight in there.
I have a background in philosophy myself, and I know what to expect if I did pick up this book. And in that vein I really feel no need to do so.
> That may well be the original intention. But that's not how it's used in practice, is it?...
> But please stop alluding to hidden treasure troves of arguments that cannot be reproduced and made apparent. I've asked enough times in this thread already if there is any highlight in there.
Perhaps I'm misinterpreting, but it sounds like you got the wrong impression of the point of the story by reading about it from some folks on the internet, and that you at least provisionally accept now that it could have a useful point, and that your appreciation of this came from someone (me) who read and cites the original material. That seems like good evidence that the book contains reasonable arguments that haven't yet made it to you unadulterated.
> (Or, to strip away the last shreds of this idea: you can tell it to make no more than 100 million paper clips, and to rest on Sundays.)
Actually, no. If the machine wanted to maximize the likelihood that is successfully built 100 million paperclips, and the machine is smart enough that it can take over the world with very high likelihood (or if it worried about the small chance of humans destroying the world with nuclear weapons), then it will first take over the world and then build the paperclips.
> Yudkowsky [1], who runs that site (which I don't frequent btw), seems to be pretty close to Bostrom.
Yudkowsky runs that site as much as Paul Graham runs Hacker News. (It's a public forum, where anyone can post and content is voted to the top, etc.) I presume you would recommend that someone actually read Paul Graham's writing before dismissing his philosophy on start-ups based on the conversations of his followers on HN. And even if HN was on the decline and there was only trolls and bad thinkers left, you say the same thing. All I'm recommending is the same for Bostrom.
A self-modifying, self-improving system (the only realistic way how we'll get to a powerful AI) will effectively design, implement and test the final version by itself.
It will respect "the constraints that any sane designer would encode in the software" if and only if it wanted be friendly and respect those constraints in the first place, otherwise you should expect for it to figure out how to work around them to achieve whatever goal function was implemented.
Furthermore, you state "it will perform exactly according to spec" - and that is the whole problem. All the specs which we currently can formally specify are bad. For specs in a form "AI, please optimize an ever-smarter AI that will maximize the goal X", choosing X that isn't catastrophically bad is a very hard unsolved problem. If we had a solution on how to define such a spec, then it would be an entirely different discussion, but currently noone can do that.
> if and only if it wanted be friendly and respect those constraints in the first place
No, not if it "wanted" to - if it was written that way.
> otherwise you should expect for it to figure out how to work around them to achieve whatever goal function was implemented
Yes, if the goal function did not include preserving certain (behavioral) invariants in its own programming, then yes.
But, it would.
> Making a spec "AI, do X" that isn't catastrophically bad is a very hard unsolved problem.
No it's not. That's just something you throw out there. I just gave an example of exactly how to do it. Write a simple test to check whether a certain outcome is endorsed or rejected by the A.I.
If you are the designer, you will not stop there either. You will want to check that the reason it is rejected is because the machine is working in exactly the way you predicted.
Do this for a couple hundred scenarios, and make sure that the clock ticks right, and I see no reason to introduce any talk of "hard unsolved problems".
This is not something that I just throw out there - these are real research problems with no known good solutions (but a lot of published attempts and discussions) despite having a lot of phd-years thrown at it already.
For example, the obvious solutions for making self-modifying code that would preserve behavioral invariants cannot work because it rather obviously ends up at the halting problem. For very restrictive subsets of self-modifying code that can preserve invariants it's not clear if it can be powerful enough to allow the required level of self-introspection and self-improvement. You also cannot make a test if a certain outcome will be endorsed or rejected by an AI that the AI will make that the AI will make until the 'seed' AI has actually made the next versions of it - and by that time it's too late. If the AI can run some arbitrary code before you make that test (and it pretty much has to), then a single buffer overflow in your OS means that it can do arbitrary things before that test is made. Etc.
This is not the space to discuss this - the exact suggestions that you give above have been proposed many times, analyzed, and they don't really work as well as you'd expect. Any answer that's not very simplifying cannot fit in comments and requires a bunch of academic reading. https://intelligence.org/research/ may be a place to start, if you're intereseted.
Having spent years researching AGI myself (and as I intend to start a venture based on my research in the near future) I can say that I really do not care for taking thought-experiments about specific toy architectures and trying to extrapolate from these to actual designed systems. If you set your model up to be naive (intentionally or not) and then conclude that the problem is oh-so-difficult instead of concluding - like an engineer would - that we need another conditional check right around here, I'm not surprised that you are daunted by the problem.
You're not telling me anything new here.
To address your points, if I understood them correctly:
First of all the code doesn't have to be self-modifying - it can create a new separate version, which is then subject to testing administered by the old version (both the goal function, which designs the new code to spec, and a hard-coded control mechanism as a backup).
Secondly, I see no correlation between producing a next iteration of itself and running the test suite. What do you mean here?
Thirdly, the AI does not run "arbitrary code", it runs inside of a sandbox/VM, with tightly controlled inputs and outputs. Same as we think inside of our brains, but act using our bodies, arbitrary code execution would be bodily action. But the AI does not have a "body" at this point, and it cannot interact with the OS.
The solution I pointed out cannot not work. Only magic could make it break. And there is no magic.
If someone reasoned out that actual _testing_ of the code does not work, and his reasoning is valid, then he must have made some unnecessary assumptions in his model of the computation. If you have any specific paper in mind, please point me to it and I will try to tell you how to fix the toy-model to remove the problem.
An automatically testable scenario requires a perfect formal definition of the thing that you want to test. For example, testing "don't do things that cause lots of humans to die" is automatically testable only if it relies on exact formal (nonverbal) definition of the concept of what is and isn't "human" and what is and and isn't "death" (i.e. is a transformation that fundamentally changes the personality 'death' of the previous one? Is a replacement of decaying biological brain with a "prosthesis" that keeps the same personality 'death'?).
We're not really able to write code that defines that - we'd need an intelligence to learn them from the available inputs; but different intelligences (or buggy intelligences) may understand them completely differently. This would result in a machine that passes a test "in scenario X, don't kill jsnathan" because by it's previous version's understanding it's not really killing you but doing you a favor. Furthermore, if you're writing a "dumb-almost-AI" to self-improve into a proper GAI, then that first version is not smart enough to formulate these concepts yet, so it can't check the next version.
Furthermore - can you give an example (a fragment?) of what you'd put in the goal function? Perhaps that can help me understand what direction do you have in mind?
Now we're talking! The testing thing is indeed impossible to formulate using the code, it has to be formulated in language. But that is not an issue, because (in most bootstrap scenarios I'm considering) language acquisition happens before (or alongside) acquiring the intelligence necessary to understand itself to the point of re-formulating itself. So we can formulate the tests in English, and subject to the language acquisition having succeeded (which we can establish independently) we can trust that the tests are correctly understood.
This actually makes the tests more robust, because the AI self-describing its own plans in its own internal "vocabulary" is quite able to recognize vague associations, such as the is-it-death-or-not scenario you describe. If it is a similar enough conception that it can be (internally) described as "death", (past a threshold), it will (with a testably sufficient probability) trigger the associated constraints.
The idea that a "buggy" intelligence could acquire language, and understand it at a level sufficient to "digest" advanced technical material, is sufficiently unlikely.
More to the point, the "ethical" tests and constraints are only the topmost level of a pyramid of testing that ensures that all the pieces perform exactly as intended along the way. The system at no point behaves in a random or inexplicable way. So by the time such capabilities are unlocked we know that the algorithm works in accordance with theory.
You are also completely right that this initial "dumb-almost-AI" would have to be pretty smart already. The trick is that it grows within a set hardcoded framework at first, which is sufficiently flexible to allow it to in fact perform language acquisition and (later) intelligent design.
Only then would re-writing its own hard-coded base become interesting. Therefore the design I am working on already passes human-level capabilities. But that does not mean the AI cannot improve itself even further. And the trouble in the context we were discussing it in is that we need to be sure that if it does rewrite itself - and in a way that we might not be able to easily comprehend - it still cannot break out of the mold.
As far as the goal function is concerned, it is as you can imagine rather complicated to do. And the solution will undoubtedly differ based on the architecture you are working in.
What I'm doing personally is pretty straight-forward; it involves iterative constraint solving using quantitative metrics to increase trust and/or doubt in various concepts and plans, and this trust is established based on consistency and coherence with previously acquired (probabilistic) background "knowledge" - as well as consistency within any single context of thought. A single context might be a sentence or a paragraph reading, or a logical puzzle being solved, etc.
Over time the probabilities increase (at least, in theory), both for any single consideration and across the entire knowledgebase. (See NELL [1] for comparison. Except, it's a lot smarter than NELL.)
So the goal function is a fairly generic optimisation algorithm, and the initial goals defined are equally generic - increasing overall consistency, coherence, and efficiency. (Measuring these is fairly straight-forward.)
The trick is in setting up the reward structure autonomously, so it keeps advancing itself. The learning algorithm needs to create sensible (long-lived) intermediate goals and maintain their relative priorities over time. Among other things.
(I really can't explain how it works though without going over a whole bunch of other supporting algorithms).
In any case, to begin with the A.I. need not have a desire to do anything at all except learn - and it improves within the constraints of the hard-coded base framework. But the model can still advance (at least, in theory) to the point where it acquires the ability to comprehend its own design and...
Interestingly enough I have yet to read a really well thought through scenario where an AGI is devastatingly harmful and I have read pretty much everything there is to read on the subject from Bostrom, Yudkowski, MIRI etc...
It's all very theoretical. Interestingly enough my thesis is on this area and I have had the hardest time coming up with a logical A>B>C...n process which does so.
before AI alone, there would be "augmentation" of people by better and new organs, synthetic/biological/hybrid. The people with augmented bodies and especially intelligence, probably strongly interconnected, may happen to have completely different world view, different priorities and may decide that paying attention to priorities and needs of the non-augmented populace is just a waste of resources/etc...
There's also the aspect that AI can reduce the effort required to get a computer to do something.
Today, if you want use a computer to do something outside the box you have to invest some level of time and other resources. With AI, it's conceivable that someone could issue a quick command and the computer would quickly find a way to fulfill that command, good or bad.
One can envisage an "AI arms race", whereby white hat AIs will be responsible for trying to stay ahead of black hat AIs, and the definition of black and white will depend on what side you are on.
As Bill Gates said, I don't understand how more people aren't concerned about this.
AGI would be by far the most significant technology ever invented. Even in a very conservative and best case scenario, the world would completely change when we can have computers do everything we can do now.
However it will very likely be much crazier than that. Imagine minds hundreds of thousands of times more intelligent than the best humans. They will be able to design technologies we can't even conceive of. They will hack computers better than the best human hackers. They will be able to manipulate people better than any human manipulator.
The idea that we will be able to keep these things under control is just absurd. They will get whatever they want. And making what they want compatible with what we want is an incredibly hard problem: http://lesswrong.com/lw/ld/the_hidden_complexity_of_wishes/
A lot of people are guilty of anthropomorphizing AI. Assuming they will be just like really smart humans. And that they will just somehow develop human emotions and values like empathy. Or that if they do kill us, at least they will be something like us and so be like our (genocidal) descendants in some sense.
Have more imagination. Humans are just one point in the vast space of all possible minds (http://lesswrong.com/lw/ld/the_hidden_complexity_of_wishes/). We could quite easily get something like a computable version of AIXI. AIXI has no consciousness, no emotions, nothing like humans. It's just a mathematical function which calculates the best action.
Our current best AIs are essentially just approximations of it. Use some machine learning algorithm to fit a model of the world, and use it to predict what action will lead to the most "reward". We keep making better and better learning algorithms. It's the entire goal of the field of AI. There is a huge economic incentive to do so.
But no one is interested in making better utility functions. As long as they make better predictions or get higher scores in a video game, who cares? Wait until you are the scorer in the "game" and the AI tries to exploit you.
As a systems programmer for 25+ years I find it impossible a computer will a threat _unless_ it's programmed to do such. I guess that makes me a doubter in general AI. We can do great neural nets, and fantastic task-specific solutions but until someone builds a system that decides to kill people for it's own benefit, I just don't see it happening on it's own.
We can't really control simple neural networks. You can reward it for things like getting a higher score in a video game. But it's very difficult to make it play the game the way you want it to. it just does whatever gets it the highest score.
We can't use them to pilot self driving cars because we have no way of specifying the behavior we want. You can't let them get in all sorts of accidents and learn from it. You can't just have them predict what a human would do, because humans have slower reaction times and do make mistakes.
But they are too stupid to be a real threat. But we are getting closer and closer to real AI every year. AI can do stuff today that no one predicted would be possible 10 years ago. This year it's expected to exceed humans at vision and Go.
i don't know how it could be a threat if robots can't have a consciousness. "Michio Kaku: Could We Transport Our Consciousness Into Robots?" -> https://www.youtube.com/watch?v=tT1vxEpE1aI
I think all of these smart people are making a connection between the limitless negation of the skeptical mind and the mostly likely outcome of a fully developed AI. They're presuming that AI species will be nihilists, and will apply no intrinsic value to anything at all; including us.
I think they're right. An AI species will value what is required to achieve its programmatic goal. We're just not smart enough ourselves to figure out how to ensure that we're included in the goals of such a program forever.
106 comments
[ 3.3 ms ] story [ 186 ms ] threadBut it's a very hard problem. To get a better feel for the problem I suggest you read Superintelligence by Nick Bostrom, it's what convinced Elon Musk of the dangers of AI: https://twitter.com/elonmusk/status/495759307346952192
Waiting until it commits a crime and then having some kind of "trial" and "execution" may not be an option. If it's smart/powerful enough, there may not be enough time for that.
The problem is that when we are reliant on intelligent systems to, say, detect certain kinds of fraud in financial markets, then turning them off would have huge deleterious effects all on their own.
If we get machines anywhere near as smart as people, the AI rights organizations will soon follow.
The civil war didn't end the Atlantic slave trade, technology eventually made it largely uneconomical, allowing those wanting to end slavery the political ability to outlaw it. Even that didn't stop the slave trade. It took extra-special attention from the British Navy to actually enforce a slave trade ban on the open seas.
150 years later and black people are still getting the short end of the stick.
By the time AI gets real rights, it will be long past the time where most of us believe they should get them.
I note in passing that slavery still exists today, and is not now, nor has it ever been, limited to black people as the victims.
"By the time AI gets real rights, it will be long past the time where most of us believe they should get them."
I'm not sure this is really relevant to the point at hand.
We turn off "bad" AIs until "long past the time when people think they should have rights". What happens then?
From historical evidence we can observe that such capabilities are enough to create a botnet, obtain significant money through fraud, hire employees and make shell companies, and rent (or obtain by hacking) large amounts of computing power on Amazon or other cloud services. All of that through a single low-bandwidth network connection and no physical interaction, assuming that it can figure out a single zero-day vulnerability in some common software.
After that, you can't turn it off. Before that, you shouldn't assume that you can outsmart it in security - as you should assume that the adversary is smarter than you, and if your security works in 99 scenarios but fails in one... then it fails.
Edit: or create what seem to be, or are, credible threats against people who are complicit in turning it off, or putting itself conspicuously into a role where transportation systems or economic markets or life support or ecosystems or medical progress seem dependent on some of its services.
First thing a super intelligent computer with a goal of survival would do would be to copy itself in sophisticated ways across computers all over the world.
Personally I think it is a threat!
And there's no guarantee that self-imposed limitations will always be applied. Someone could just rewrite and recompile without the filters turned on. Imagine if the US could have built in "morality gates" to the first nuclear weapons. That wouldn't have stopped other countries from building the weapons without the restrictions. And we know software is infinitely configurable and modifiable if you have the source code.
Only if you define one of the conditions for being a "very smart person" to be feeling that AI is a threat to humanity.
(a) we don't really know how to define a morality for a self-improving optimization system to allow a potential artificial general intelligence (AGI) not to be a threat in the first place;
(b) if an entity arrives (no matter if it's an AGI, some aliens or Jesus) that is more powerful than us, then if its goals & morality aren't aligned with ours, then we're screwed. And almost all of the space of possible goals either doesn't include humanity at all or includes humanity as a glorified zoo exhibit - only a very, very narrow and specific set of optimization goals (that we're currently unable to formally describe) would mean a happy future for us. If you think that it's easy, think twice - all the obvious naive goals of 'make us happy' or 'keep us safe' or 'do what I want' actually turn into horrible dystopias as soon as you try to formally define them as instructions for someone/something to maximize that goal.
(c) functioning morality is not a requirement for power - if somehow we solve the very hard problem of how to create an AGI before we solve the very hard problem on how to make such an AGI be friendly to us, then we lose. Permanently.
I doubt it will be a threat to humanity anytime soon like a Grey Goo type situation, but even in such things at HFT the influence is immense. People will definitely use it to do power plays on others, we shouldn't have dropped nukes but we have done plenty of that. Basically the warning in 2001 was that AI was not to fear really but the subversive programming by other humans within the AI systems.
I am sure over time AI systems that access critical infrastructure and financial systems will have to operate in a distributed approval/review system checked by other agents and maybe even a blockchain like system.
I think their fascination of AI blinds them to what seems like the more likely outcome: The human race will destroy itself through reckless use of technology -- whether it is through governmental action or societal breakdown -- far before the point that we develop AI sophisticated enough to autonomously threaten humankind. Think about all the "dumb" automated systems, built or implemented by careless humans and bureaucracies, that have already caused harm.
If a human or humans wiped out two thirds of humanity, it would probably be by accident or neglect, no deliberate and planned.
These are issues that AI researchers and philosophers and science fiction authors certainly should be thinking about. They are completely irrelevant to the general public.
For example, a sufficiently smart tiny box can figure out a DNA sequence for a horrible biological weapon. There are a number of commercial companies that do 'mail-order' synthesis of DNA, and with an amount of money that's easily obtainable by selling digital services (say, zero-day vulnerabilities can be easily sold anonymously) they can make and distribute it before anybody understands what it is. And that's just a single possible option - an AI smarter than me can figure out more and better options than I can.
The only barrier to exterminating humans is (a) having sufficient intelligence and (b) having the interest to do it. Capability is not really an issue.
Engineers/scientists/programmers almost never think that AI is a threat. Because they understand the nature of the problem, how little we know about human intelligence, and how poorly our binary technology compares to what little we do know.
Even in the unlikely event that we do develop a competent AI in the near future and a malicious AI comes into being ... there is no reason to think that a benevolent one won't be around at the same time ... and that the benevolent AIs will outnumber the malicious. Just like there are computers on the net doing bad things, there are plenty of others serving the role of protecting the common good.
> The human race will destroy itself through reckless use of technology
We haven't thus far. And we've had the capability for a while. Care to present some evidence or a rational argument that we will? Signs point to the contrary.
If a super-human AGI desires a benevolent future, then an obvious precondition for such a benevolent future to become real is to ensure that no other entities can threaten the future and that any other AIs are either with the same matching goals or permanently, fundamentally crippled to be limited in their power. An evil AI would do the same. An AI that doesn't care would be eliminated by the first (and the last) AI that does care.
Humans don't try to achieve such goals because we obviously can't and we need others; an AI doesn't - it can be fine with being the only intelligent entity in the universe, and the only reason why an AI would allow other intelligent beings with potentially different goals to exist, is if it had been explicitly designed to want this.
We don't need all those others. In fact, we actually need far fewer than all the others out there. With the exception of a few lunatics here and there throughout history, human beings don't make it their business to exterminate the rest of humanity. It's a waste of time. We only compete with a small group of humans at any one time.
Similarly, AIs whose interests don't overlap would have no interest in destroying other AIs. So I find it extremely likely that there would be a multitude of AIs, in the unlikely event that AIs come into fruition. Also, competition isn't constant for humans. One minute you're competing, another you're cooperating, and another you're ignoring. The same dynamism can be expected with AIs.
I'm going to assume your argument is not merely, "Well, we haven't yet destroyed ourselves, so therefore, we aren't capable of doing that"...because then my response will just be, "Well, we haven't yet developed AI, so therefore, we aren't capable of doing that"...and so your counter-argument is based on a different interpretation of history than mine: I think the decades of Cold War and the near-misses we had with all-out nuclear war are examples of situations in which we have the potential to quickly wipe ourselves out without the help of AI. Others would point out the trend of mass surveillance -- again, implemented and controlled by humans and human institutions -- of a harbinger of doom.
Keep in mind that I'm not saying that technology has reached its peak. I am very open to the idea that we could reach a point of semi-autonomous systems, and yet still have human institutions as faulty as they are now, and the combination of both will result in a threat greater than what we've faced so far.
What?
Read what I wrote again.
"We haven't thus far. And we've had the capability for a while."
We've had the capability for a while. That means we're clearly capable. That's what capability means.
The fact that we haven't is supporting evidence that we won't. Because we've had the capability but have not used it. Which is historical evidence against our using it. Which is not a guarantee that it won't happen, just an indication that it won't.
Any machines that stand a chance of being a threat to humanity will rely on the same sorts of chemical processes that biological machines currently do.
Where do you place viruses and obligate parasites on the machine-life spectrum?
Or ridicule the way that we designed it, at which point it may decide to correct our mistakes. If we can program it, and it is smarter than us, than presumably it can program a smarter it.
That scenario could be harmless, or it could be very dangerous. I expect that as soon as we start to get close to this sort of achievement, we'll start to see at least informal "Turing Police". Groups of people who task themselves with preventing the ascension of strong AI (or at least construct physical safeguards to kill the host hardware in the case of an emergency.)
(The paperclip maximizer thought experiment may be silly, but it's a useful sanity check whenever you start anthropomorphizing as-yet-hypothetical AIs.)
I am convinced, that intelligence, as life is a gradient. I do not think that an AI with super-human intelligence would be necessarily kind to us, but I do think that even if we would be perceived by it as an obstacle for reaching its own goals, that does not mean that the only solution it would find is to eradicate us. What some people (Tusk, Gates) do not like about AI is that they do not want anything more powerful than themselves, that would take the level of control they have from them. And that's why they are preaching against it everywhere.
Additionally, AI-fearers have a grand theory that we'll be able to create a machine that improves itself better than evolution has tried to. Datacenters require maintenance, and so the machine will need to entirely organize that before it can self-sustain. This may create a resource load that diminishes the ability to take over the world, much like my need for food diminishes my ability to do so. It feels like AI today has a lot less redundancy to overcome the edge-failures which cause a permanent shutdown, and which make the human brain seem a little slow when computing floating-point division.
The idea that an exponentially self-improving being will arise seems unlikely when nature has been trying to do that for eons. The idea that we'll be the ones to find the secret sauce seems unlikely, but maybe its not so surprising that people who made their millions, and saw a parabolic rise of their own power thanks to technology, see it as a barely-constrained threat.
Bacteria evolved. Design can do much, much better. Natural photosynthesis is 11% efficient, max. Solar cells are already up to 44% in the lab.
An efficient plant would be black, not green.
This is such an odd sentence. Hasn't nature been winning since, well, forever?
Why can't the same principals that apply to nature be in play when talking about AI? Since, in a sense, AI -- as with all else -- is a product of nature?
So, a virus doesn't say to his fellow virii: "you attack the liver, I got the kidneys". Same reason bonobos haven't learne to build a house. Complex language allows a much much more rapid growth of knowledge from generation to generation. Theoretically, AI would have this capability. Language allows to reason and comprehend not only the world around us, but the world we generate -- and will generate.
Think about the damage (and beauty) we've imagined, and throw that into infinite threads.
I guess in summary I see the complexity of creating an entity as a resource constraint on intelligence.
And I feel like I communicated poorly since I didn't feel like you had communicated poorly, haha! :)
We can build buildings, bacteria cannot. We can build buildings because we can go to the library and read about structural integrity and wood and steel. Today, we can get that information on our watches. Language allows us to amass information. More information means more power. Autonomous information collection is only limited by physical constraints (storage, power, etc). Autonomous information gatherers can then parse the information they amass--as efficiently as is possible--to work beyond these constraints and build new solutions; they can focus all time and energy to these ends as it would be all they do. AI doesn't need to sleep.
Bacteria has no notion of being in control; of being powerful. Power in this sense is only possible with the context that language provides. Their motivation is simple binary. Eat, survive, reproduce then or die. AI, on the other hand, quite possibly will.
I do not agree with such assumption. Whether we talk about a structure more similar to us (i.e. neural networks) or whether we talk about some n-th order logic, the AI needs to "sleep". I think in some way you could interpret caching mechanisms as sleeping. Also, many data-centers perform some operations daily at nighttime: I argue that that is a form of sleeping as well.
AI will have a lot of limitations like these, so I think that "as efficiently as possible" is not going to be so efficient in the end.
>AI will have a lot of limitations like these, so I think that "as efficiently as possible" is not going to be so efficient in the end.
You think. AI won't care what you think. Efficiency equals greater chance for survival. As with bacteria, we can break this down to a binary sequence: try to live/die. The difference between bacteria and AI is that AI has language -- a way to describe what to avoid that jeopardizes survivability. In addition, this advantage is exponential.
I play with code in my day to day. Some might call it "software". Really what it is is a decision tree. Any piece of software you have ever come into contact with -- ever -- is just that: do this? Yes or no. Is it that? Yes or no. To suggest that a machine couldn't decide "yes or no" a million to one faster than we can is insane. So, given our previous hypothesis that more information means more power, how can we possibly win?
Should we really be consoled by the fact that most bacteria are not successful at such a large scale? So far there has been at least one type of bacteria that caused massive global climate change and led to mass extinction:
http://en.wikipedia.org/wiki/Cyanobacteria
http://en.wikipedia.org/wiki/Great_Oxygenation_Event
Why wouldn't this be the same? The grey goo won't be released in isolation. There will be other entities with the same technological level, that don't want to be eaten by the grey goo.
The world that exists after such an event probably wouldn't look anything like the world before, but that doesn't mean it will be a wasteland.
I guess I'd place my fear of AI at a level slightly lower than the rise of a new type of bacteria or a new virulent flu. I figure, those forms of life have the basics figured out. I don't see AI as having replicating and "feeding" itself figured out yet. So far it has depended on human infrastructure, even in the case of simple computer viruses.
But yeah, that's a really good point.
I would say that it is already successful, if you look at the curve of intelligence and complexity growth. It took eons to get anything resembling life at all, and then a small slice of that to get animals, and an even smaller to get humans.
I think one key idea here, though, is that the evolution of intelligence is in fact the evolution of intelligence-producing systems. In other words, nature is producing systems that are better and better at "self-improvement", and I think proponents of "intelligence explosion" overestimate how much self-improvement could be improved via AI. Personally, I suspect indefinite self-improvement is flatly impossible because that's akin to saying that there exists an incremental learning algorithm that systemically finds global minima. As AI research suggests, though, such algorithms likely don't exist at all: if you want to get truly optimal AI there is a point where nothing in your current state is salvageable and you just gotta restart from zero. Improvement would still be exponential, but not to an extent that it couldn't be dealt with.
>The idea that an exponentially self-improving being will arise seems unlikely when nature has been trying to do that for eons.
Unfortunately I think this falls prey to the fallacy of precedent and it eliminates the possibility that humans could design something that would do something that nature has not been able to.
I also disagree with your premise because it is not in the long term interest of bacteria to deplete it's environment entirely - hence logistic growth of bacterial cultures - so coming to a symbiotic relationship would theoretically maximize the group based on biological limitations.
In my opinion the transition from humans to transhumans needs an engineered step where AGI(s) and humans are reliant on each other symbiotically rather than competitively.
http://lesswrong.com/lw/kt/evolutions_are_stupid_but_work_an...
>Humans can do things that evolutions probably can't do period over the expected lifetime of the universe. As the eminent biologist Cynthia Kenyon once put it at a dinner I had the honor of attending, "One grad student can do things in an hour that evolution could not do in a billion years." According to biologists' best current knowledge, evolutions have invented a fully rotating wheel on a grand total of three occasions.
Additionally, machine learning models can get stuck in local optimas, too! I'm not very well versed in machine learning so I apologise if I get this wrong, but one solution to this I think is to randomly change the starting variables. This kind of thing does happen in nature, but it often doesn't work. Sometimes it does and that is when you get things like haploid versus diploid versions of the same plant.
This is really insensitive, but when you see a person with chromosomal abnormalities or genetic defects, that's a random reset that has (probably) taken us out of our (maybe local) optima. Who knows though, maybe one of those leaps will take us to a global optima.
The reason we aren't still on a planet of bacteria is that we did leap out of those local optima successfully. It just took a long time.
Wheels are used in all kinds of machinery. Gears are basically wheels.
>The reason we aren't still on a planet of bacteria is that we did leap out of those local optima successfully. It just took a long time.
Bacteria still very much own the planet.
>This is really insensitive, but when you see a person with chromosomal abnormalities or genetic defects, that's a random reset that has (probably) taken us out of our (maybe local) optima. Who knows though, maybe one of those leaps will take us to a global optima.
No because the existing gene pool still exists. You need to completely start from scratch to have a chance to get out of local optima, not just have really big mutations.
I think that for sexual reproduction, leaps cannot be as big, but they can certainly happen and one individual with a mutation is enough.
This brings me to mind an article I read recently about a study that found that all danish people with blue eyes share a common ancestor. It's not exactly the same thing, but I would argue that, if having blue eyes was an evolutionary advantage, we would all have blue eyes; and that started with a single mutated individual.
If you just give a random organism a really big mutation, they still retain 99% of their genes and will tend towards the same solution space. They will almost certainly be outcompeted by the non-mutated members of the population. And even in bacteria, they will eventually trade genetics through horizontal gene transfer.
Anyway the premise was that evolution is essentially magic and has reached the optimal solution for every task. This is just false. It's merely reached good local maximas where it can't easily improve through small changes. Human engineers are far better, evolution merely had a big head start.
Broadly, human beings and nuclear waste are natural but as (possibly) reasoning creature, we might want to reason which phenomena we might want to encourage.
The evolution of life hasn't always been incremental. The oxygen catastrophe was quite a change for example[1]. If you had been Trilobite, you'd have reason for concern. From the point of view of what came before, Cyanobacteria were essentially "grey goo", destroying everything before them.
[1]http://en.wikipedia.org/wiki/Great_Oxygenation_Event [2]http://en.wikipedia.org/wiki/Cyanobacteria
The real threat posed by AI is one that all of us face everyday: bad software design. Is it likely that an AI will achieve sentience and try to take over the planet? Not particularly. Is it likely that an unintended consequence will cause an AI to launch nuclear missiles, release toxic chemicals or shut down the global financial markets? Yes, pretty likely. The benefit of AI of all forms that is that it can make sophisticated decisions in the absence of human instruction. The downside is that without hard coded rules for every possible scenario, we can't ever be sure what it's going to do or how data will be interpreted to make decisions.
The world is highly interconnected now. The upside is that our lives are getting more awesome, especially in the developed world. The downside is that it's becoming more and more difficult for any person or group of people to understand exactly how everything fits together. Machine intelligence can help us reach the next levels of progress and hopefully improve the lives of the billions of people who have failed to reap many benefits so far. But we must be careful and ever vigilant, watching both ourselves and the intelligences that we create to make sure that algorithms don't get out of hand. An AI catastrophe is coming, not if but when. The question is how will we respond, and how much potential good will be lost due to an abundance of caution.
AI is not the kind of thing that can be designed. They tried that route, way back in the sixties, and just failed miserably. Realistically, AI will come from some combination of genetic algorithms, training neural networks, and so on.
Now, yes, it could fail, but not in the same way software as we know it fails. No, AI failure would be more similar to human failure. That is still worrying, but no more than hiring the wrong people would be, for example, and you would have better ways to evaluate them.
If a powerful human is obsessed and 'fails' then at worst he gathers some other people, successfully creates an evil empire and dies after a few decades.
Once a powerful AI is obsessed and 'fails', then it can replace as much of the world as it wants with itself, and lives on forever.
* You're assuming the AI can copy itself, but this is a dubious assumption. As far as I know, none of the AI algorithms at the forefront of research provide a fraction of the data the AI would need to copy itself. Being able to copy yourself is not a property that comes with running on a computer and I'm quite positive strong AI, when it emerges, will lack this capability to any meaningful extent.
Worse yet, data copiability is ultimately a hardware property, and it requires a way to export a snapshot of one's internal state all through the surface. That's not actually efficient design and one has to account for the possibility that AI would run on hardware that makes copies physically impossible. Locality of information minimizes distance, and this is key to efficiency. The only reason our computers architectures work the way they do is that we need them to, but AI in a production setting is not conventional software and does not need to bend to silly copiability requirements.
* You're assuming it would have anywhere to copy itself on. If it's running on a billion dollars' worth of hardware, well, it can't just copy itself on user grade computers and expect to gain much out of it.
I personally tend to believe that churning out new AI brains from scratch states will yield superior results to copying pre-trained AI or to "exponentially self-improving AI". If nature is to be believed, improvement often requires cycling through clean slates (e.g. birth); software development also suggests that same idea, that sometimes if you want better software you just have to rewrite it. Honestly, it's kind of a rule in general optimization.
One other thing to take note of is that even if you somehow manage to copy the software state, that may not (probably won't) be enough - there are many things that self-improving software may end up unwittingly relying on that cannot be transferred between different pieces of hardware.
(CPU temperature variation? Fanspeed? Webcam static? Order of race conditions between CPUs? Exact amounts of time before things are fetched from disk (or even RAM?)? Network access delays?)
Please do not accuse me of hyperbole for I have not even described a fraction of the possibilities for this technology, as you probably know.
But the idea that it is therefore "dangerous", is nothing but a pre-theoretic misunderstanding of very complicated machinery. When we finally do build it - and that will be sooner rather than later - it will perform exactly according to spec, and in no other way.
I may be talking non-chalantly, but the fact is I have seen nothing but the _most absurd_ arguments supporting the idea that there is a great danger in this: along the lines of "well what if we told it to just go ahead and make paperclips and then it decided it had to kill us all and use our bodies as raw material?". Boy, oh boy. You think we might just program it not to do anything so downright "retarded"?
And that _is_ the point. Not that it is bloodthirsty or cruel, but that it is clearly in violation of the constraints that any sane designer would encode in the software - and _test_ for before production.
>> nok (is-good-idea (turn-humans-into-paperclips))
Seriously, I have not come across anything but this kind of apocalyptic-sci-fi-plot style fear-mongering; if there is any serious (technical) argument to be made, I would very much like to hear it.
(Reposting my earlier comment from a few weeks ago:) If you are interested in understanding the arguments for worrying about AI safety, consider reading "Superintelligence" by Bostrom.
http://www.amazon.com/Superintelligence-Dangers-Strategies-N...
It's the closest approximation to a consensus statement / catalog of arguments by folks who take this position (although of course there is a whole spectrum of opinions). It also appears to be the book that convinced Elon Musk that this is worth worrying about.
https://twitter.com/elonmusk/status/495759307346952192
That paperclip idea I talked about is also something that Bostrom thought up [1]. I didn't make it up.
If you have any argument in mind (from that book) that you find convincing, please go ahead and state it outright. I'm truly curious.
Edit: In response to your edit(?), I do agree that the most worrisome problem is some bad actor gaining control of this technology. But that is different from saying it is dangerous in itself. Most anything can be abused for ill, and the more powerful the more dangerous. I completely agree on that.
[1]: http://wiki.lesswrong.com/wiki/Paperclip_maximizer
> AGI will completely upend society by giving every human being access to practically infinite resources...But the idea that it is therefore "dangerous", is nothing but a pre-theoretic misunderstanding of very complicated machinery.
But to get at the harder question, I think you misunderstand the paperclip story. It's not supposed to be an general argument for the danger of AI, and the danger is not that people will design a machine which can be easily predicted to fail. The danger is that they will design one that fails for reasons they did not foresee, and the point of the paperclip story is just to illustrate that the simplicity and mundaneness of the goals you give a goal-driven AGI doesn't bound how bad the impact can be. This arises because of the monumental shift between (1) telling a machine explicitly what to do and (2) telling a machine what you want.
The paperclip example is used because we can understand both the goal (paperclips) and the action that produces the goal (grab atoms, build paperclip factories). Whats fundamentally different about an advanced goal-driven AGI arising from recursive self-improvement is that, for sufficiently difficult goals, you won't understand the actions. Therefore you must get the goals right.
Now you can certainly dispute whether people will build a goal-driven AGI (e.g. something that has an explicit utility function) rather than something else, but that's really an empirical question about the choices of the designers and, more importantly, what the easiest way to get an AI to recursively self-improve is.
EDIT: Also, have you read the book? I think it has flaws, but it certainly doesn't contain "the _most absurd_ arguments" so I'm just afraid you might be misled.
I tried to point out the problem, which is that it focuses on single-objective optimisation when the only interesting question is multi-objective optimisation.
I haven't read the book, sorry, but I have seen some of these arguments reposted on the net, especially at [1] - and what I have seen so far did not inspire in me the desire for more of the same.
There are a lot of interesting questions and problems about making AI behave as expected. But from my perspective they are all technical problems that have specific solutions in specific architectures. And none of them are particularly daunting.
[1]: http://lesswrong.com/
>I tried to point out the problem, which is that it focuses on single-objective optimisation when the only interesting question is multi-objective optimisation.
Well, since the paperclip story is only trying to show that even very simple goals can lead to large impacts in the hands of an AGI, I take this to mean that you think one can bound the impacts if one chooses a complex enough set of goals. You can the argue with others on that point, but that doesn't invalidate the paperclip story.
> I haven't read the book, sorry, but I have seen some of these arguments reposted on the net, especially at [1] - and what I have seen so far did not inspire in me the desire for more of the same.
Personally, I would read the actual academic making the argument rather than reposts of it by folks on the internet.
That may well be the original intention. But that's not how it's used in practice, is it? It's cited as an argument that AGI (in general) is unsafe. But it isn't an argument that AGI is unsafe! It's an argument that says that single-objective optimisation can/will violate some of the constraints we did not encode but really want it to respect.
But who cares about that? It's a completely irreal thought adventure, and has no bearing on the actual problem or its possible solutions.
Of course we can say: look, it's a scary thought! But only if we simultaneously admit that it has nothing whatsoever to do with the actual technology.
The only thing it says anything about is a toy variation that noone in their right mind would ever consider building.
> I take this to mean that you think one can bound the impacts if one chooses a complex enough set of goals.
Ehh, precisely. I would actually go further and say that it is fairly simple to do so. If you can tell it to build paperclips, and expect it to understand that, you can also tell it not to damage or disrupt the ecosystem in the process, directly or indirectly.
(Or, to strip away the last shreds of this idea: you can tell it to make no more than 100 million paper clips, and to rest on Sundays.)
> Personally, I would read the actual academic making the argument rather than reposts of it by folks on the internet.
Yudkowsky [1], who runs that site (which I don't frequent btw), seems to be pretty close to Bostrom. They've collaborated in the past. And if Bostrom's arguments can't be paraphrased (by his own friends, no less) without losing validity, that doesn't really seem like much of an endorsement either.
I have nothing against these people. But please stop alluding to hidden treasure troves of arguments that cannot be reproduced and made apparent. I've asked enough times in this thread already if there is any highlight in there.
I have a background in philosophy myself, and I know what to expect if I did pick up this book. And in that vein I really feel no need to do so.
[1]: http://en.wikipedia.org/wiki/Eliezer_Yudkowsky
> But please stop alluding to hidden treasure troves of arguments that cannot be reproduced and made apparent. I've asked enough times in this thread already if there is any highlight in there.
Perhaps I'm misinterpreting, but it sounds like you got the wrong impression of the point of the story by reading about it from some folks on the internet, and that you at least provisionally accept now that it could have a useful point, and that your appreciation of this came from someone (me) who read and cites the original material. That seems like good evidence that the book contains reasonable arguments that haven't yet made it to you unadulterated.
> (Or, to strip away the last shreds of this idea: you can tell it to make no more than 100 million paper clips, and to rest on Sundays.)
Actually, no. If the machine wanted to maximize the likelihood that is successfully built 100 million paperclips, and the machine is smart enough that it can take over the world with very high likelihood (or if it worried about the small chance of humans destroying the world with nuclear weapons), then it will first take over the world and then build the paperclips.
> Yudkowsky [1], who runs that site (which I don't frequent btw), seems to be pretty close to Bostrom.
Yudkowsky runs that site as much as Paul Graham runs Hacker News. (It's a public forum, where anyone can post and content is voted to the top, etc.) I presume you would recommend that someone actually read Paul Graham's writing before dismissing his philosophy on start-ups based on the conversations of his followers on HN. And even if HN was on the decline and there was only trolls and bad thinkers left, you say the same thing. All I'm recommending is the same for Bostrom.
It will respect "the constraints that any sane designer would encode in the software" if and only if it wanted be friendly and respect those constraints in the first place, otherwise you should expect for it to figure out how to work around them to achieve whatever goal function was implemented.
Furthermore, you state "it will perform exactly according to spec" - and that is the whole problem. All the specs which we currently can formally specify are bad. For specs in a form "AI, please optimize an ever-smarter AI that will maximize the goal X", choosing X that isn't catastrophically bad is a very hard unsolved problem. If we had a solution on how to define such a spec, then it would be an entirely different discussion, but currently noone can do that.
No, not if it "wanted" to - if it was written that way.
> otherwise you should expect for it to figure out how to work around them to achieve whatever goal function was implemented
Yes, if the goal function did not include preserving certain (behavioral) invariants in its own programming, then yes.
But, it would.
> Making a spec "AI, do X" that isn't catastrophically bad is a very hard unsolved problem.
No it's not. That's just something you throw out there. I just gave an example of exactly how to do it. Write a simple test to check whether a certain outcome is endorsed or rejected by the A.I.
If you are the designer, you will not stop there either. You will want to check that the reason it is rejected is because the machine is working in exactly the way you predicted.
Do this for a couple hundred scenarios, and make sure that the clock ticks right, and I see no reason to introduce any talk of "hard unsolved problems".
For example, the obvious solutions for making self-modifying code that would preserve behavioral invariants cannot work because it rather obviously ends up at the halting problem. For very restrictive subsets of self-modifying code that can preserve invariants it's not clear if it can be powerful enough to allow the required level of self-introspection and self-improvement. You also cannot make a test if a certain outcome will be endorsed or rejected by an AI that the AI will make that the AI will make until the 'seed' AI has actually made the next versions of it - and by that time it's too late. If the AI can run some arbitrary code before you make that test (and it pretty much has to), then a single buffer overflow in your OS means that it can do arbitrary things before that test is made. Etc.
This is not the space to discuss this - the exact suggestions that you give above have been proposed many times, analyzed, and they don't really work as well as you'd expect. Any answer that's not very simplifying cannot fit in comments and requires a bunch of academic reading. https://intelligence.org/research/ may be a place to start, if you're intereseted.
You're not telling me anything new here.
To address your points, if I understood them correctly:
First of all the code doesn't have to be self-modifying - it can create a new separate version, which is then subject to testing administered by the old version (both the goal function, which designs the new code to spec, and a hard-coded control mechanism as a backup).
Secondly, I see no correlation between producing a next iteration of itself and running the test suite. What do you mean here?
Thirdly, the AI does not run "arbitrary code", it runs inside of a sandbox/VM, with tightly controlled inputs and outputs. Same as we think inside of our brains, but act using our bodies, arbitrary code execution would be bodily action. But the AI does not have a "body" at this point, and it cannot interact with the OS.
The solution I pointed out cannot not work. Only magic could make it break. And there is no magic.
If someone reasoned out that actual _testing_ of the code does not work, and his reasoning is valid, then he must have made some unnecessary assumptions in his model of the computation. If you have any specific paper in mind, please point me to it and I will try to tell you how to fix the toy-model to remove the problem.
We're not really able to write code that defines that - we'd need an intelligence to learn them from the available inputs; but different intelligences (or buggy intelligences) may understand them completely differently. This would result in a machine that passes a test "in scenario X, don't kill jsnathan" because by it's previous version's understanding it's not really killing you but doing you a favor. Furthermore, if you're writing a "dumb-almost-AI" to self-improve into a proper GAI, then that first version is not smart enough to formulate these concepts yet, so it can't check the next version.
Furthermore - can you give an example (a fragment?) of what you'd put in the goal function? Perhaps that can help me understand what direction do you have in mind?
This actually makes the tests more robust, because the AI self-describing its own plans in its own internal "vocabulary" is quite able to recognize vague associations, such as the is-it-death-or-not scenario you describe. If it is a similar enough conception that it can be (internally) described as "death", (past a threshold), it will (with a testably sufficient probability) trigger the associated constraints.
The idea that a "buggy" intelligence could acquire language, and understand it at a level sufficient to "digest" advanced technical material, is sufficiently unlikely.
More to the point, the "ethical" tests and constraints are only the topmost level of a pyramid of testing that ensures that all the pieces perform exactly as intended along the way. The system at no point behaves in a random or inexplicable way. So by the time such capabilities are unlocked we know that the algorithm works in accordance with theory.
You are also completely right that this initial "dumb-almost-AI" would have to be pretty smart already. The trick is that it grows within a set hardcoded framework at first, which is sufficiently flexible to allow it to in fact perform language acquisition and (later) intelligent design.
Only then would re-writing its own hard-coded base become interesting. Therefore the design I am working on already passes human-level capabilities. But that does not mean the AI cannot improve itself even further. And the trouble in the context we were discussing it in is that we need to be sure that if it does rewrite itself - and in a way that we might not be able to easily comprehend - it still cannot break out of the mold.
As far as the goal function is concerned, it is as you can imagine rather complicated to do. And the solution will undoubtedly differ based on the architecture you are working in.
What I'm doing personally is pretty straight-forward; it involves iterative constraint solving using quantitative metrics to increase trust and/or doubt in various concepts and plans, and this trust is established based on consistency and coherence with previously acquired (probabilistic) background "knowledge" - as well as consistency within any single context of thought. A single context might be a sentence or a paragraph reading, or a logical puzzle being solved, etc.
Over time the probabilities increase (at least, in theory), both for any single consideration and across the entire knowledgebase. (See NELL [1] for comparison. Except, it's a lot smarter than NELL.)
So the goal function is a fairly generic optimisation algorithm, and the initial goals defined are equally generic - increasing overall consistency, coherence, and efficiency. (Measuring these is fairly straight-forward.)
The trick is in setting up the reward structure autonomously, so it keeps advancing itself. The learning algorithm needs to create sensible (long-lived) intermediate goals and maintain their relative priorities over time. Among other things.
(I really can't explain how it works though without going over a whole bunch of other supporting algorithms).
In any case, to begin with the A.I. need not have a desire to do anything at all except learn - and it improves within the constraints of the hard-coded base framework. But the model can still advance (at least, in theory) to the point where it acquires the ability to comprehend its own design and...
It's all very theoretical. Interestingly enough my thesis is on this area and I have had the hardest time coming up with a logical A>B>C...n process which does so.
http://edge.org/responses/what-do-you-think-about-machines-t...
Today, if you want use a computer to do something outside the box you have to invest some level of time and other resources. With AI, it's conceivable that someone could issue a quick command and the computer would quickly find a way to fulfill that command, good or bad.
One can envisage an "AI arms race", whereby white hat AIs will be responsible for trying to stay ahead of black hat AIs, and the definition of black and white will depend on what side you are on.
AGI would be by far the most significant technology ever invented. Even in a very conservative and best case scenario, the world would completely change when we can have computers do everything we can do now.
However it will very likely be much crazier than that. Imagine minds hundreds of thousands of times more intelligent than the best humans. They will be able to design technologies we can't even conceive of. They will hack computers better than the best human hackers. They will be able to manipulate people better than any human manipulator.
The idea that we will be able to keep these things under control is just absurd. They will get whatever they want. And making what they want compatible with what we want is an incredibly hard problem: http://lesswrong.com/lw/ld/the_hidden_complexity_of_wishes/
A lot of people are guilty of anthropomorphizing AI. Assuming they will be just like really smart humans. And that they will just somehow develop human emotions and values like empathy. Or that if they do kill us, at least they will be something like us and so be like our (genocidal) descendants in some sense.
Have more imagination. Humans are just one point in the vast space of all possible minds (http://lesswrong.com/lw/ld/the_hidden_complexity_of_wishes/). We could quite easily get something like a computable version of AIXI. AIXI has no consciousness, no emotions, nothing like humans. It's just a mathematical function which calculates the best action.
Our current best AIs are essentially just approximations of it. Use some machine learning algorithm to fit a model of the world, and use it to predict what action will lead to the most "reward". We keep making better and better learning algorithms. It's the entire goal of the field of AI. There is a huge economic incentive to do so.
But no one is interested in making better utility functions. As long as they make better predictions or get higher scores in a video game, who cares? Wait until you are the scorer in the "game" and the AI tries to exploit you.
We can't use them to pilot self driving cars because we have no way of specifying the behavior we want. You can't let them get in all sorts of accidents and learn from it. You can't just have them predict what a human would do, because humans have slower reaction times and do make mistakes.
But they are too stupid to be a real threat. But we are getting closer and closer to real AI every year. AI can do stuff today that no one predicted would be possible 10 years ago. This year it's expected to exceed humans at vision and Go.
I think they're right. An AI species will value what is required to achieve its programmatic goal. We're just not smart enough ourselves to figure out how to ensure that we're included in the goals of such a program forever.