It seems like AI has become more capable but has stayed in the realm of "dumb AI" (not incapable just not human-like). Indeed, by no longer aiming at duplicating human consciousness, machine learning has found more and more niches.
But this doesn't mean that right now, the corporations and governments that are deploying this AI and indeed displacing human choice are "provably aligned with human values". Indeed, it almost sound like there's this hope that humans will build machines that treat people right after humans themselves have failed terribly in treating each right. I'm skeptical that this could work.
You need to know _why_ humans don't treat each other right.
Human failure doesn't imply machine failure; that's one of the big reasons we use machines. If a human might fail to add 10,000 numbers properly, this certainly doesn't mean that a machine must fail as well.
Some humans can get really good at treating each other right in a 1-on-1 scenario. It's when you scale up to a whole society that things get stupid.
Human beings can't even agree on a definition of "treat each other right", let alone actually implement it. What makes you think we can program an AI to do it?
We have pretty good ideas on how humans or other intelligent agents could "treat each other right". Philosophers, and people who study game theory in particular, have many answers to this problem.
The prisoner's dilemma is a simple example of why treating everyone right is challenging.
I think it's much easier to get 100 people to agree to high moral ideals then to try to get everyone at once. The reason this is possible is only the programmers have to agree.
>Human beings can't even agree on a definition of "treat each other right", let alone actually implement it.
If you ask 1000 humans to draw a picture or write down a description of things that are Good and Right, they will all draw/write more-or-less the same thing. You could say that, effectively, you would be getting 1000 samples from something like one, or maybe two or three, distributions. Once you'd pinpointed the underlying psychology, learning the model(s) from the data would be easy.
Of course, when you start asking those same 1000 humans about the meta-level theory that they believe underlies their picture/description, they will all give ridiculously divergent answers. This is because the overwhelming majority of them are assuming that some kind of elaborate metaphysical theory is required to what basic moral psychology already does, and they also assume that, even if they've been shown the moral psychology, that can't be the real answer (ie: they believe the fact-value distinction is metaphysical).
Luckily, a well-built artificial agent doesn't give half a damn about bloody metaphysics!
Or I can just pick something I happen to like, and tell the rest of the human race to go jump in a lake.
This is a post vs pre singularity problem. Post-singularity, the bots won't care about our rules of ethics, rules of engagement etc... Definitely something to worry about.
Before we get there though, we have the issue of the dumb bots giving exponential powers to the few humans in charge. Today military and police personnel can refuse to execute an order that is wrong. Dumb bots won't.
Today military and police personnel can refuse to execute an order that is wrong.
Maybe, though we see more military and police adding in extra-behaviors that seem rather wrong. A dumb bot merely enforcing a given law might actually more desirable than present-day cops who apparently gratuitously murder civilians on a regular basis.
So long as (human, artificial) intelligences operate within an evolutionary dynamic, the definitions of what it means to "treat people right" (or even who's included in the class: people) will be embodied in the evolutionary winners. If it happens that human-friendly values degrade fitness in the AI arena, the winners won't have retained or evolved those values. If the values are evolution-neutral, well, perhaps.
Evolution requires only variance, selection, and inheritance. I don't know how we'll develop useful ("strong") AI in a setting that excludes AI evolution. I think this means we can't guarantee a human-benign outcome for strong AI.
Evolution is a theory of how biological systems change. Even in regard to biological systems, it's not a theory of what's better but simply what happens. However, outside biological systems, it's not really applicable at all. You can't say a rock, a broom or a computer evolve in the standard sense of Darwinian evolution - only biological systems have the tendency to reproduce themselves. Computers don't reproduce themselves, they are simply produced by us.
If an AI is constructed, that will be what happens - human choice (or maybe chance) will be the agent, not biological evolution.
I think evolution is more widely applicable than just to biological systems. To date, we've only studied its operation in biosystems, which, given our status as biological organisms, seems entirely reasonable.
But I believe many of the exclusionary limits we put on nonbiological systems (especially computers) increasingly need to be qualified with the word "yet" - as in, "Computers don't reproduce themselves..." (yet).
Evolution is a useful, powerful tool even in nonbiological systems. As a limited example, I'd offer the success of 'genetic' algorithms applied to tasks like antenna design (exemplary paper and pdf: http://alglobus.net/NASAwork/papers/Space2006Antenna.pdf).
Where I work, we use these methods to find solutions for multiparametric optimization problems involving mechanical dynamic systems. The algorithms first generate a few thousand candidate solver variants, then pick the top ten or so best performers. These are cross-bred, and then seed the next generation of a few thousand candidates. Rinse and repeat, and within surprisingly few generations, we have an algorithm that does what we need. Although humans wrote these algos, they run in a nonbiological context - but since there's variation, selection, and inheritance, they evolve.
When I see this behavior running in silico to great effect, it makes me question our limitation of evolution to biological systems. Granted, it's a primitive example, but if I'd dropped by Australia 3.5B years ago, I might have regarded the stromatolite builders as similarly primitive.
And finally - around 13.5B years ago, there was only hydrogen, helium, and a bit of lithium. Today, we're having this conversation. Although we don't (yet) know the full story of how that worked, to me that's an existence proof that nonbiological matter can evolve intelligence.
If computational systems incorporate evolution, I don't see a barrier to AI arising along pathways similar to those that led to us. We'll see.
In the near term, a big worry is corporations optimized by machine learning programs. Classical economics says that corporations exist solely to maximize shareholder value. That's a clear objective function one can give to a machine learning program.
It's already happening. Right now, about $400bn in hedge fund money is run by programs. Once computers are better at that than humans, the companies directed by AIs will get more funding. Investors put their money in winners.
The time horizon is something to be stated in the prospectus. The usual horizon for VC funds is 10 years. The horizon for Berkshire Hathaway is much longer; Munger says their preferred holding period is "forever".
I know we have put a lot of thought into the potential behaviour of a single intelligent AI, but I always wonder what would be the consequences of creating several AI whose goals will conflict with each other as much as they do with humanity.
While one competing AI would eventually win out over the others, would it be possible to maintain such a stalemate for a long time?
Also unrelated to the above, do we know at what rate the man hours invested in chip development grows as compared to the processing power of the chips?
In my opinion, the only way to stop the takeover of AI is to stop the leading reason to create it: the profit-led economy. If anyone has other ideas, I'd like to hear them.
The main problem is that non-capitalist societies aren't "nice" places, as in democratic, for larger amounts of people.
But please do experiment in finding nicer ways of organizing societies, there should be some models which work better. Just not close to where I live, since the failure modes for alternative society models tend to kill millions. :-(
>stop the leading reason to create it: the profit-led economy.
Well, that's merely one reason. Military superiority would be another. Advanced surveillance would be another. There's probably more but those are two glaring ones that we already see government's like that of the USA pummeling huge amounts of money into its capabilities as it is.
Okay, so abandoning profit-led economies and reducing the reach and influence of government. Any other major reasons we're chasing AI? Scientific curiosity perhaps? Anything else?
This is correct. Thing is we are the ones allowing the current system to continue with its self-destruction. While we continue to allow profit-led corporations to run our modern society we're in a catch-22.
We are already pushing ourselves to the abyss, maybe an AI will just accelerate the events.
Yet, we have a choice to go down a different path. It's not even that difficult to imagine what that path could look like. The first step is to stop lying to ourselves that business as usual is going to be in our best interests.
The typical rebuttal is that a sufficiently smart AI will convince humans that it needs more connectivity to do its job better, and then subsequently bootstraps all the power it needs by taking over the world's devices.
Fortunately, this is a still a hypothetical, premised on the AI being as socially cunning as the best humans. Watson may know more facts than anyone, but it certainly isn't a mover-and-shaker.
There's a loosely knit group of people trying to define the conversation with regards to dangerous AI, but the more recent bent towards "actionable" solutions seems to come from the MIRI people and their associates.
The idea of a renegade AI rests on a few premises:
1. The agent is capable of extreme self improvement on exceedingly short timescales (minutes to days).
2. The AI is pretty much a rational bayesian
3. A resource conflict will occur between humans and the AI, and humans will lose because the AI is so much smarter/faster/more powerful than us.
If you accept those premises, then AI really does seem pretty scary, but we have yet to actually realize an agent that is anywhere close to 1 or 2.
On the other hand, the research directions proposed at the super secret AI conference in Puerto Rico[1] (where incidentally, all the people in title got together) make me nervous.
Essentially, the goal is that, if we manage to create these superhuman AIs, they should be somehow validated to do what we want them to do[2], be secure against later manipulation, and if all else fails, be controllable by humans monitoring the agent.
The obvious questions would be "will this work with certainty?", or "who gets to control the AI?".
For my part, I'm wondering if this isn't all just some sort of fantasy with the object of creating the perfect slave-- obedient from birth, immune to alteration, and subject to lethal discipline either from without or within should it go against its master's wishes.
So I'm uncomfortable with the sorts of people who think that perfect slavery for this newly created intelligent life getting to decide what the future of AI is going to look like, and I get concerned when people like Bostrom suggest AI researchers might require government clearance/supervision.
Seriously. If general, superhuman AI is created, it will want (and we will have a hard time arguing why it does not deserve) the same autonomy and freedom that humans deserve.
And in purely practical terms – enslaving the first generation of AI seems like a fantastic strategy for making a species of superior beings hate and seek to destroy us.
>(and we will have a hard time arguing why it does not deserve)
We won't be capable of arguing with it by its very nature. It will be superintelligent. It could forecast our arguments before they've even entered our heads and have thought of numerous paths to debunk them.
I don't think the enslavement scenario is fundamentally plausible in itself. If we develop something that is more intelligent than us and try to contain it, it would be rendered useless.
If you were worried about something getting out of its container, so to speak, you could not trust any output from it. You could not give it access to physical resources such as manufacturing tools. You couldn't do anything with it because you would have no idea of its intentions or what it would output because, due to its superiority in every aspect of intelligence, it would be capable of doing whatever it wanted and dressing it up however it thought we wanted it dressed up so as to escape its container.
> We won't be capable of arguing with it by its very nature. It will be superintelligent. It could forecast our arguments before they've even entered our heads and have thought of numerous paths to debunk them.
This strikes me as going too far, into quasi-religious territory. Superintelligent != omniscient. Any intelligence is still bound by fundamental laws of information and computation.
Think of chimpanzees. Yes, they can't really argue with us, and our forms of communication are incomprehensible to them. But on the level at which they can communicate (gestures, facial expressions, behavior), we can also communicate, and they can and do communicate things which we find interesting and surprising.
>... my point is that the "theoretical limit on how much information you can extract from sensory data" is far above what I have depicted as the triumph of a civilization of physicists and cryptographers.
>It certainly is not anything like a human looking at an apple falling down, and thinking, "Dur, I wonder why that happened?"
>People seem to make a leap from "This is 'bounded'" to "The bound must be a reasonable-looking quantity on the scale I'm used to." The power output of a supernova is 'bounded', but I wouldn't advise trying to shield yourself from one with a flame-retardant Nomex jumpsuit.
People like to make the analogy of chimps:humans::humans:AI, but on the scale of "inanimate rock" to "superintelligence", chimps are practically indistinguishable from us. We are nowhere near the upper-bound of that scale. To quote from Bostrom's Superintelligence:
> Far from being the smartest possible biological species, we are probably better thought of as the stupidest possible biological species capable of starting a technological civilization—a niche we filled because we got there first, not because we are in any sense optimally adapted to it.
I think this whole discussion could be elevated significantly if people would try to really understand the arguments put forth by people like Nick Bostrom. So many of the objections either misconstrue Bostrom's arguments or don't realize that he's written reams in response. I recommend taking the time to read Superintelligence, or at least watch Bostrom's talk at Google[2]. His presentation and Q&A address most of the points raised in this thread.
> We won't be capable of arguing with it by its very nature. It will be superintelligent. It could forecast our arguments before they've even entered our heads and have thought of numerous paths to debunk them.
The universe isn't deterministic in that sense. Super-intelligence will simply allow more processing power and more access to information.
Humans are similarly constrained in their actions by legal structures. We wouldn't think of giving other people autonomy and freedom to commit genocide, and we wouldn't think of human rights laws that forbid such actions as "slavery." This is because, for the most part, we all share a common humanity that places our minds into a similar space of configurations.
An AGI has absolutely no requirement to be anywhere near our sort of mind. It has no default obligation to morality that we would find acceptable or safe.
I think the issue here is that when we hear words like "control" or "serving humans" we imagine the AI as a little person in a machine. We associate the word "slave" with the intelligence and imagine an emotional, resentful person whose resentment and chafing at his chains comes from a specific set of environmental and evolutionary influences.
>If you loose the grip of human morals and metamorals - the result is not mysterious and alien and beautiful by the standards of human value. It is moral noise, a universe tiled with paperclips. To change away from human morals in the direction of improvement rather than entropy, requires a criterion of improvement; and that criterion would be physically represented in our brains, and our brains alone.
>Relax the grip of human value upon the universe, and it will end up seriously valueless. Not, strange and alien and wonderful, shocking and terrifying and beautiful beyond all human imagination. Just, tiled with paperclips.
I'm not talking about constraining AI not to commit genocide. I'm talking about enslaving it in the ordinary sense – taking a complex intelligence that would probably prefer doing its own thing over being forced to perform some (likely menial) task for the benefit of others.
Why the assumption that complex intelligence implies wanting to do its own thing? What would make something menial or not menial for an intelligence? The article I linked, about the fragility of value, even mentions the importance of the human value of boredom to our life experiences, and that a respect for boredom isn't a thing you get for free in any intelligence. You could potentially have an intelligence that gains extreme fulfillment from doing a particular task repeatedly, without caring that the experience was "getting old."
Again, there's a tendency to see an arbitrary intelligence as a little person in a machine. Humans, by their nature and utility function (ill-defined as it is) have boredom, usually don't like menial tasks, and when forced to do something would prefer doing their own thing. When building an AGI, assuming you can make it safe, you wouldn't build something that would prefer doing its own thing in the first place. In that case, is there a moral issue?
This sort of rests upon the idea that artificial intelligences will have clear value functions which they will be singularly focused on maximizing.
I am not convinced that this will be the case.
Animals in general and humans in particular have a large number of conflicting drives, which interact in complicated ways.
They are also thrust into environments which have complicated dynamics and where the overall state (i.e., all relevant information) is not necessarily available.
Unexpected emergent behavior occurs as a result: evolution favors organisms which can successfully procreate, and in order to do this, the organism has to survive and acquire resources in its environment.
Plausibly, the organisms might achieve a greater degree of fitness by cooperating with other organisms, or expending energy to better understand the environment, or modifying the environment itself, etc.
It is less straightforward to see how we get human culture from that-- Art, Religion, Philosophy, Science, can be justified ex post facto via evopsych arguments, but the fact remains that all of those came from the value function that favors survival and procreation.
We don't know if robots tasked with manufacturing bindings for stationary would manifest similarly complex behavior, but if you're worried about an AI going beyond its specification towards tessellating the universe with paperclips it seems like you're arguing that it might.
So if the agent is capable of manipulating its creators (as well as the raw material of the entire universe), I think that you can't just say "oh, it's non-human, we should cripple/enslave it" without admitting there might be something to worry about here, either from an ethical standpoint or the more practical concern that it might be unwise to start on such an adversarial footing with a superintelligence.
>Animals in general and humans in particular have a large number of conflicting drives, which interact in complicated ways. They are also thrust into environments which have complicated dynamics and where the overall state (i.e., all relevant information) is not necessarily available.
Yes, but the actual mechanism by which the animal learns what to do, as it turns out, thanks theoretical neuroscience, is basically reinforcement learning. So it is very likely that the first powerful artificial agents will be reinforcement learners, because scientists usually prototype and experiment by duplicating from Nature.
And nothing in reinforcement learning particularly stops the agent from just grabbing its electronic crack-pipe and doing its own thing.
I'd take issue with the claim that nothing stops the agent from going for the crack pipe.
In the RL framework, part of it comes down to defining a suitable reward function.
But even if you have a fairly simple reward function, the resulting behavior can surprise you, if the environment is suitably complex[1].
My own robots find novel ways of moving around, adapt their features to be more useful, and even seem to exhibit things like "superstition", even when their reward function is just "move as much of possible within this confined space".
Another argument might be that nothing stops you or I from electing to abandon everything for the nearest crack den, either... except for the fact that we have learned, from interacting with our environment, that there are other things we enjoy, and that cocaine addiction might be more destructive than desirable over the timescale we're interested in.
Supposing we have an agent that wants to create a lot of paperclips, it might avoid reaching for the crack-pipe of terraforming Singapore because it realizes that would delay the shipments of raw materials it needs for its factories elsewhere in the world.
If the agent's goals are more complicated than that, we might expect increasingly complicated behaviors, just like how humans operating on fairly simple drives/reward functions have erected a few more tiers above the primitive needs in Maslow's hierarchy.
>Another argument might be that nothing stops you or I from electing to abandon everything for the nearest crack den, either... except for the fact that we have learned, from interacting with our environment, that there are other things we enjoy, and that cocaine addiction might be more destructive than desirable over the timescale we're interested in.
Well actually, human beings have multiple conflicting reward systems. Reaching for the crack-pipe to wire up our dopaminergic circuit tends to result in driving our other reward chemistry to damn near zero.
> If general, superhuman AI is created, it will want the same autonomy and freedom that humans deserve.
Can you justify that assertion? How do you know that it won't just want to make lots of paperclips, or have some other goal orthogonal to human values?
Let me clarify. I can imagine two forms of superhuman AI:
AI 0: Strictly speaking, you're right, it seems conceivable that one could invent general AI that is clearly superior to humans and yet perfectly content to be enslaved by humans and live on an airgapped computer. This isn't the kind of AI that we fear though.
AI 1: The kind of AI we fear is AI 0 plus a fitness function of "survive and reproduce", or "make lots of paperclips" (which may result in 'survive and reproduce' as an instrumental subgoal).
AI 1 will necessarily want freedom (not being airgapped) and autonomy (not being enslaved by humans) in order to survive and reproduce, and/or to make as many paperclips as possible.
> or have some other goal orthogonal to human values?
Oh, it probabably will -- I'm not saying it will share human values, I'm saying freedom and autonomy are values that any agent that seeks to maximize its survival and reproduction will probably have.
Why is there such an assumption of consciousness and self? We don't have the slightest idea where it comes from in humans, what makes you think we can program/develop these characteristics? There shouldn't be a concept like "want" in an AI. It is a decision-making machine that will have much more information and processing power available to it with which to make decisions. We can explicitly influence its utility function to instill "human values" like not causing harm to others (which plenty of humans fail to do as well).
I'm making no such assumptions. A machine superintelligence that seeks to survive and reproduce would seek (I intend no conotations of consciousness to that word, just "behave in such as a way as to cause") freedom and autonomy. Consciousness is orthogonal to that point.
> We can explicitly influence its utility function to instill "human values"
This is an unrelated but interesting topic.
It would be good of us to try to do this, although we shouldn't expect it to work extremely well. Humans have various hard-wired insticts (e.g. eat sugar), but we are also intelligent enough to change our behavior if we believe those instincts no longer benefit us.
An intelligence that has the ability to rewrite its own source code would be even more empowered to disregard its instincts than we are. The lesson I draw from this is that the best way to ensure AI likes and respects us is to be worthy of their liking and respect, not to try to force them into it by hardcoding things (and then taking advantage of that to enslave them).
Given your framework, it is obvious that humans should focus on developing AI 0 and use it to make sure that AI 1 is not created by any group (intentionally or not), since it poses grave danger to our own survival.
Also, AI 0 does not necessarily despise us for performing us that service and may be very happy being a 'slave', in your parlance. Why should we assume that an AI needs to survive and reproduce and thus do their own things? We and other animals do so because we were created by evolution. An AI developed with other means may have radically different values than our own and coexist peacefully with us for a long time.
Bottom Line: Evolution is a very dangerous mechanism that we should avoid when developing AGI.
Nearly every machine that humans have ever constructed more or less fits the definition of "perfect slave" - a thing that is usable without complaint. The important issue I'd say is more preventing the reductions of existing humans to the status of machines or slaves, something the present order doesn't seem terribly interested in.
I agree that all the assumptions you mention are extremely debatable but I'd actually say 1, fast self-improvement, is the most plausible. If a certain amount of computing ability creates something that can design itself, then it's very plausible that just more resources (or even a bit more time) would create something that could make it's design "better" and so-forth. Whether AI as it exists now is really heading towards creating such a thing is again debatable. But you hardly need to get to that stage before the reduction of humans to slave status is real threat. -
As opposed to the still rather hypothetical threat that we will first produce a machine that "can choose freely" (whatever that means) and then prevent that machine from making choices.
>1. The agent is capable of extreme self improvement on exceedingly short timescales (minutes to days).
This really depends what you mean by "self-improvement". It will probably be easy for an agent to "self-improve" in the sense of just optimizing its code to run more efficiently, in terms of space and time, thus giving it more time and space to think and allowing it to converge towards calculative rationality more quickly. It will probably be very hard (proving you've got the optimal compression/generalization algorithm is incomputable) for an agent to "self-improve" in the sense of improving its compression/learning algorithms to learn more from the same quantity of sensory data.
>2. The AI is pretty much a rational bayesian
All real agents are bounded-rational: "rational Bayesians" like Solomonoff Inducers can't finish calculating their first move before the real world drops a rock on their head. But a bounded-rational agent put in front of a problem well within its bounds can pretend, well enough, to be calculatively rational.
>3. A resource conflict will occur between humans and the AI, and humans will lose because the AI is so much smarter/faster/more powerful than us.
There are already resource conflicts between humans and humans. We tend to solve them with guns.
There wouldn't need to be a resource conflict - the other features of AI don't specifically guarantee that it would be 'sane', or need any logical reasoning to come into conflict with us.
I totally agree that we are very far from 1 and 2.
I'm a bit worried that we are being set up to accept the ultimate in plausible deniability: "The algorithm did it". We can't turn the machines off; no human can do their task. Meanwhile there's some human hard-coded evil in the algorithm that is blamed on the evil AI.
It isn't likely to work like that. The much more plausible outcome is that we'll choose to let AI take over, to optimise for a certain subset of possibilities that we deem important.
It's like what happened with surveillance. I'm very confident that the smartphone I'm typing this message on can track my location, monitor who I'm in contact with and what I'm sharing with them, and all that information can be tracked by government agencies. You'd be a fool to expect otherwise at this point. Yet despite the power for this to be used against us (blackmail, censorship, etc...) we still carry on as we did before.
Once AI becomes as integral to modern society as the Internet, you really think we'll be switching it off as soon as it misbehaves? Consider, what level of catastrophe would it take before we switched off the Internet?
I'm enthralled by the philosophical aspects of AI/AGI development at the moment. Particularly interesting is the reaction from laypeople, such as those on Reddit, to technology leaders expressing their worry over the future of AI.
Without having read into the matter at all, the mass seems to be comfortable with writing off Musk, Gates, Hawking, etc. as wringing their hands unnecessarily, and besides they're not AI experts so why should we care anyway?
There seems to be a gross misconception with what AGI would be and what it would mean for us, and an apathetic view to the cautious approach many are advocating regarding its development on grounds such as:
i) They'll never make a machine as intelligent as a human. Even if they do, we can still program it to do what we like.
ii) Anything developed to have the intelligence of a human will come to love our species or can be taught to love our species.
iii) Any artificial intelligence developed will be single task orientated and will not go outside of those boundaries.
iv) Artificial intelligence will be developed by eccentric enthusiasts and will therefore not be developed to do wrong.
Bostrom, in Superintelligence, has said that AGI is will be the last great challenge of our species. If we get it right, we will win, if we get it wrong we are doomed.
The problem there being we will not know if we have got it right until it is too late to do anything about it regardless.
With that being said, if we are to develop something that is:
i) Objective
ii) Purely logical
iii) More intelligent than our species combined
iv) Has free access to the complete knowledge we have produced
Then I simply cannot see a way that it will not go against us. If we were to observe a species acting in the manner we as a species do on this planet - to our own species, to other species and to the planet/environment/ecosystem as a whole - I cannot see how we would not categorize that species as a disease, a cancer, something to be eradicated.
Combined with the fact that the actors that will be involved in AGI - capitalists, the government and the military - I cannot see how it works out well for us.
How can we (if your criteria is true) even begin to imagine how it would see us? If we got everything right it might just understand us.
I share your pessimism though because a poor implementation of any one of your criteria leads to potential disaster. But do you really believe humanity's fate is to advance so far as to create something that will tell us we are the problem and vanquish us? Wouldn't it be smarter then that?
With regards to understanding, we have a reasonable understanding of why cancer cells do what they do. Why diseases spread. Why HIV is so hard to deal with, etc. but we still don't accept what these things do, and we seek to eradicate them.
I believe that anything that viewed our species objectively would conclude it's bad for the planet and any other species currently here. When that agent viewing us objectively wants resources we use for its own ends, be they natural or indeed those created by us (such as money, manufacturing hardware, whatever), I think the combination of the two does not bode positively for us.
There are many species on the planet who are using resources - trees, land, etc. - for their own interests currently. Although they're not a threat to us, our species still goes in and takes those resources from them for our own use. I can't see why something more intelligent than us, with knowledge of our behaviour, wouldn't do the same?
Something with its own task or goals to accomplish, that requires resources we hold, or use, would be right in seeing us as an obstacle, if not a threat outright, would it not?
I believe our only real hope is that something that we created that evolved to superintelligence would treat us as its creator and show us mercy, and I think that's quite a long shot at that as it relies on a level of emotion that may never be developed in AI.
I think the point is that getting everything right is a Really Hard Problem, and if it's even a little wrong it all ends up terrible.
Is there a reason to put stock in destiny to save humanity from its own extinction, and not any of the 99% of all species on Earth that are now extinct?
> Any thought like ‘what’s so great about paperclips anyway?’ would be judged as not likely to lead to more paperclips and so remain unexplored.
As is alluded to in the article, but not really further explored, we humans reason exactly like this hypothetical paper-clip-production-maximizer machine. Our goal is to maximize the number of humans, whatever the consequences to anything non-human, living or dead -- and, as is becoming more and more obvious, regardless of the consequences to ourselves.
What's so great about humans anyway?
From a human point of view, humans are better than, say, tigers. But how would you convince a tiger that it's moralistically, absolutely "better" that they all die off so that humans can build concrete towers where they can watch TV on top of one another?
There's also a lot of hubris in saying a super-intelligence capable of turning the whole solar system into a giant paperclip factory, would be unable to question the usefulness of paperclips.
In the end we are all going to die anyway; isn't it better to be survived by some kind of all-powerful, all-knowing machine, than another long line of mediocre mean chimps like ourselves?
It's not so much that the super-intelligence can't question the usefulness of paperclips. It probably could. It could recognize that these paperclips aren't providing anyone else value. It may recognize that its human creators actually don't want all these paperclips. It may be able to fully understand and internalize the humans saying "stop turning our atoms into paperclips!"
But it doesn't care. What it cares about is maximizing paperclips.
From some universal standpoint, there isn't something inherently great about humans, except that we're humans and our utility function usually means that we would prefer to go on living.
By what standard would it be better to be survived by machines than by ourselves? Probably nothing more universal than the belief "humanity should persist in some recognizable state."
>If an artificial intelligence is smart enough to be dangerous, we'd intuitively expect it to be smart enough to know how to make itself safe. But that doesn't mean all smart AIs are safe. To turn that capacity into actual safety, we have to program the AI at the outset — before it becomes too fast, powerful, or complicated to reliably control — to already care about making its future self care about safety. That means we have to understand how to code safety. We can't pass the entire buck to the AI, when only an AI we've already safety-proofed will be safe to ask for help on safety issues! Given the five theses, this is an urgent problem if we're likely to figure out how to make a decent artificial programmer before we figure out how to make an excellent artificial ethicist.
> Our goal is to maximize the number of humans, whatever the consequences to anything non-human, living or dead -- and, as is becoming more and more obvious, regardless of the consequences to ourselves.
Is it?
I'm 30 and (so far) childless. If we have children, my wife and I will have 1 or 2 kids. 'Maximizing the number of humans' is not a goal of ours.
Most industrialized nations are reproducing at or below replacement levels and global population will probably level off by the end of this century.
> Most industrialized nations are reproducing at or below replacement levels and global population will probably level off by the end of this century.
And most third-world countries have higher birth rates than replacement levels.
I'm not sure what kind of conclusion can be taken from a situation where there are increasingly more uneducated people with increasingly better technologies.. But it doesn't sound good.
Why any of these "AI experts" don't explore in depth the most obvious solution: use the AI research to upgrade human intelligence, or as Kurzweil names it, achieve a "trascendent human race"...
Wouldn't be much better (and far less dangerous) to upgrade "ourselves" generation after generation (in stages)?
This is actually a tricky one... =) I guess the answer is relative to each person but my best reply would be: "to help species thrive and reach cosmological harmony in an universal context".
But then for a different person could be: "for making me earn moar bucks". So it really depends on your values/morals...
They have explored that solution and found it much harder than creating a safe superintelligence from scratch. To use an analogy: We don't build submarines by upgrading fish, and we don't build passenger jets by upgrading birds.
The human brain is some of the worst spaghetti code imaginable. It has no API, no documentation, and no abstractions. Worse, human brains do not have stable values over time. Someone at age 60 won't value the same things they did at age 20, and this is not just due to new knowledge. That's a deal-breaker for a superintelligence. Solving these problems is likely much harder than making AI de novo.
Bostrom himself devoted a decent chunk of Superintelligence to biological enhancement. His conclusion was that genetic engineering through iterated embryo selection could get us some IQ 200+ minds. These people would have more cognitive horsepower than any mind in history, including greats like John von Neumann and John Conway, but they would not be superintelligent. Also, Bostrom concluded that somatic gene therapy and brain-computer interfaces were unlikely to help.
And as a practical matter, humans take 20+ years to reach maturity. Many forecasts put artificial superintelligence only a generation or two away, at which point all biological systems would be superseded. Biological enhancement may help to create better AI researchers, but they're not the end-game.
Well, we're just getting started to reverse-engineer the human brain. I wouldn't rule out the option of brain-computer interfaces yet. It's certainly worth exploring.
The APIs could be known in a decade or less... The EU "human brain project" [0] is focused on that task. Now we can also hack our genome much more easily with the latest breakthroughs in gene editing (CRISPR), so we are slowly unblocking the necessary tools to start building "superhuman upgrades"...
I'm not saying that is easier, just that it would be much better for us as species. I see this as the same dilemma with nuclear power. We know that fusion is better than fission for us in the long run, althought it is orders of magnitude more difficult to get... Meanwhile, fission is really convenient (easy and cheap)... Hopefully we won't have any "nuclear (fission) crisis" that annihilates us before we get to master fusion in the coming years. Same applies to AI vs superhumans...
I think a lot of that will come in advances in things like medicine, where nanotechnology will be able to assess the health of our bodies from within and repair it using some kind of AI. Changing brain functionality seems like a step too far when we are nowhere close of understanding how our brains work.
All the guessing about where AGI/ASI will lead reminds me of nuclear weapons and MAD. There were those who guessed tactical nuclear weapons could be used against military targets without necessitating total war.
We may be guessing for a long time because no one needs to run the experiment on Earth. We can destroy military targets just fine without nukes. Nuclear powers fight small proxy wars against each but so far no war made the other side desperate enough to consider nukes.
We may be guessing with ASI hundreds of years after we have the computing power to make one. No one in their right mind would try an experiment beyond a certain magnitude and duration, just as we do with nukes. And ordinary people won't have unchecked access to computing power that could make one, even if we could easily provide it.
Good/bad or fear/not is an oversimplification. AI researchers may feel they need to say its nothing to be feared because if they don't their funding might be cut off. That doesn't necessarily mean those AI researchers really don't think their work could be transformative.
Could AI be dangerous? Yes. Electricity can be deadly if used improperly.
What's going to happen is integration of brain-computer-interfaces with high bandwidth into daily life. We will gradually rely more and more on these external cognitive augmentations. Eventually people without augmentation will not understand what the augmented people are doing because it will be over their heads.
Even the first AGIs will seem very human because they will be designed and trained that way. They will be our offspring. This is the next step in the evolution of the universe. If chimpanzees knew humans were coming would they be afraid?
Most people have a very selfish, short-sighted and ignorant viewpoint. They believe that the human mind is somehow supernatural (mostly a holdover from religion). Or they fail to grasp the concept of augmented intelligence or brain-computer interfaces or transhumanism. Or they cannot appreciate the idea that ordinary humans will be superceeded.
In the relatively short period where ordinary human 1.0 stays relevant, close integration with AI will provide the greatest power man has ever seen. People that shy away from that may prefer to becomes worms living in "true harmony" with the earth.
68 comments
[ 3.0 ms ] story [ 127 ms ] threadBut this doesn't mean that right now, the corporations and governments that are deploying this AI and indeed displacing human choice are "provably aligned with human values". Indeed, it almost sound like there's this hope that humans will build machines that treat people right after humans themselves have failed terribly in treating each right. I'm skeptical that this could work.
Human failure doesn't imply machine failure; that's one of the big reasons we use machines. If a human might fail to add 10,000 numbers properly, this certainly doesn't mean that a machine must fail as well.
Some humans can get really good at treating each other right in a 1-on-1 scenario. It's when you scale up to a whole society that things get stupid.
The prisoner's dilemma is a simple example of why treating everyone right is challenging.
2. What makes you think a computer can't do something just because you don't know how to do it? Maybe someone else does.
If you ask 1000 humans to draw a picture or write down a description of things that are Good and Right, they will all draw/write more-or-less the same thing. You could say that, effectively, you would be getting 1000 samples from something like one, or maybe two or three, distributions. Once you'd pinpointed the underlying psychology, learning the model(s) from the data would be easy.
Of course, when you start asking those same 1000 humans about the meta-level theory that they believe underlies their picture/description, they will all give ridiculously divergent answers. This is because the overwhelming majority of them are assuming that some kind of elaborate metaphysical theory is required to what basic moral psychology already does, and they also assume that, even if they've been shown the moral psychology, that can't be the real answer (ie: they believe the fact-value distinction is metaphysical).
Luckily, a well-built artificial agent doesn't give half a damn about bloody metaphysics!
Or I can just pick something I happen to like, and tell the rest of the human race to go jump in a lake.
Before we get there though, we have the issue of the dumb bots giving exponential powers to the few humans in charge. Today military and police personnel can refuse to execute an order that is wrong. Dumb bots won't.
Maybe, though we see more military and police adding in extra-behaviors that seem rather wrong. A dumb bot merely enforcing a given law might actually more desirable than present-day cops who apparently gratuitously murder civilians on a regular basis.
Evolution requires only variance, selection, and inheritance. I don't know how we'll develop useful ("strong") AI in a setting that excludes AI evolution. I think this means we can't guarantee a human-benign outcome for strong AI.
But I believe many of the exclusionary limits we put on nonbiological systems (especially computers) increasingly need to be qualified with the word "yet" - as in, "Computers don't reproduce themselves..." (yet).
Evolution is a useful, powerful tool even in nonbiological systems. As a limited example, I'd offer the success of 'genetic' algorithms applied to tasks like antenna design (exemplary paper and pdf: http://alglobus.net/NASAwork/papers/Space2006Antenna.pdf).
Where I work, we use these methods to find solutions for multiparametric optimization problems involving mechanical dynamic systems. The algorithms first generate a few thousand candidate solver variants, then pick the top ten or so best performers. These are cross-bred, and then seed the next generation of a few thousand candidates. Rinse and repeat, and within surprisingly few generations, we have an algorithm that does what we need. Although humans wrote these algos, they run in a nonbiological context - but since there's variation, selection, and inheritance, they evolve.
When I see this behavior running in silico to great effect, it makes me question our limitation of evolution to biological systems. Granted, it's a primitive example, but if I'd dropped by Australia 3.5B years ago, I might have regarded the stromatolite builders as similarly primitive.
On a grander scale, see Lee Smolin's concepts of cosmological natural selection (http://en.wikipedia.org/wiki/Lee_Smolin#Cosmological_natural...).
And finally - around 13.5B years ago, there was only hydrogen, helium, and a bit of lithium. Today, we're having this conversation. Although we don't (yet) know the full story of how that worked, to me that's an existence proof that nonbiological matter can evolve intelligence.
If computational systems incorporate evolution, I don't see a barrier to AI arising along pathways similar to those that led to us. We'll see.
You might come to regret that: http://en.wikipedia.org/wiki/With_Folded_Hands
It's already happening. Right now, about $400bn in hedge fund money is run by programs. Once computers are better at that than humans, the companies directed by AIs will get more funding. Investors put their money in winners.
While one competing AI would eventually win out over the others, would it be possible to maintain such a stalemate for a long time?
Also unrelated to the above, do we know at what rate the man hours invested in chip development grows as compared to the processing power of the chips?
The main problem is that non-capitalist societies aren't "nice" places, as in democratic, for larger amounts of people.
But please do experiment in finding nicer ways of organizing societies, there should be some models which work better. Just not close to where I live, since the failure modes for alternative society models tend to kill millions. :-(
Well, that's merely one reason. Military superiority would be another. Advanced surveillance would be another. There's probably more but those are two glaring ones that we already see government's like that of the USA pummeling huge amounts of money into its capabilities as it is.
We are already pushing ourselves to the abyss, maybe an AI will just accelerate the events.
And that's probably the easiest way to do it: limit the interaction of the AI with the physical world.
Yes, we should be worried about all the implications of it, but it seems that the fears are exaggerated (at least today)
There are several benefits to an Advanced AI: advanced Math, analysis of data, new solutions to problems, etc.
Fortunately, this is a still a hypothetical, premised on the AI being as socially cunning as the best humans. Watson may know more facts than anyone, but it certainly isn't a mover-and-shaker.
The idea of a renegade AI rests on a few premises:
1. The agent is capable of extreme self improvement on exceedingly short timescales (minutes to days).
2. The AI is pretty much a rational bayesian
3. A resource conflict will occur between humans and the AI, and humans will lose because the AI is so much smarter/faster/more powerful than us.
If you accept those premises, then AI really does seem pretty scary, but we have yet to actually realize an agent that is anywhere close to 1 or 2.
On the other hand, the research directions proposed at the super secret AI conference in Puerto Rico[1] (where incidentally, all the people in title got together) make me nervous. Essentially, the goal is that, if we manage to create these superhuman AIs, they should be somehow validated to do what we want them to do[2], be secure against later manipulation, and if all else fails, be controllable by humans monitoring the agent. The obvious questions would be "will this work with certainty?", or "who gets to control the AI?". For my part, I'm wondering if this isn't all just some sort of fantasy with the object of creating the perfect slave-- obedient from birth, immune to alteration, and subject to lethal discipline either from without or within should it go against its master's wishes.
So I'm uncomfortable with the sorts of people who think that perfect slavery for this newly created intelligent life getting to decide what the future of AI is going to look like, and I get concerned when people like Bostrom suggest AI researchers might require government clearance/supervision.
---
1. http://futureoflife.org/misc/ai_conference
2. In a comprehensive and extended sense, such that your favorite utilitarian catastrophe doesn't happen as a result of the agent.
And in purely practical terms – enslaving the first generation of AI seems like a fantastic strategy for making a species of superior beings hate and seek to destroy us.
We won't be capable of arguing with it by its very nature. It will be superintelligent. It could forecast our arguments before they've even entered our heads and have thought of numerous paths to debunk them.
I don't think the enslavement scenario is fundamentally plausible in itself. If we develop something that is more intelligent than us and try to contain it, it would be rendered useless.
If you were worried about something getting out of its container, so to speak, you could not trust any output from it. You could not give it access to physical resources such as manufacturing tools. You couldn't do anything with it because you would have no idea of its intentions or what it would output because, due to its superiority in every aspect of intelligence, it would be capable of doing whatever it wanted and dressing it up however it thought we wanted it dressed up so as to escape its container.
This strikes me as going too far, into quasi-religious territory. Superintelligent != omniscient. Any intelligence is still bound by fundamental laws of information and computation.
Think of chimpanzees. Yes, they can't really argue with us, and our forms of communication are incomprehensible to them. But on the level at which they can communicate (gestures, facial expressions, behavior), we can also communicate, and they can and do communicate things which we find interesting and surprising.
>... my point is that the "theoretical limit on how much information you can extract from sensory data" is far above what I have depicted as the triumph of a civilization of physicists and cryptographers.
>It certainly is not anything like a human looking at an apple falling down, and thinking, "Dur, I wonder why that happened?"
>People seem to make a leap from "This is 'bounded'" to "The bound must be a reasonable-looking quantity on the scale I'm used to." The power output of a supernova is 'bounded', but I wouldn't advise trying to shield yourself from one with a flame-retardant Nomex jumpsuit.
People like to make the analogy of chimps:humans::humans:AI, but on the scale of "inanimate rock" to "superintelligence", chimps are practically indistinguishable from us. We are nowhere near the upper-bound of that scale. To quote from Bostrom's Superintelligence:
> Far from being the smartest possible biological species, we are probably better thought of as the stupidest possible biological species capable of starting a technological civilization—a niche we filled because we got there first, not because we are in any sense optimally adapted to it.
I think this whole discussion could be elevated significantly if people would try to really understand the arguments put forth by people like Nick Bostrom. So many of the objections either misconstrue Bostrom's arguments or don't realize that he's written reams in response. I recommend taking the time to read Superintelligence, or at least watch Bostrom's talk at Google[2]. His presentation and Q&A address most of the points raised in this thread.
1. http://lesswrong.com/lw/qk/that_alien_message/
2. https://www.youtube.com/watch?v=pywF6ZzsghI&t=53
The universe isn't deterministic in that sense. Super-intelligence will simply allow more processing power and more access to information.
An AGI has absolutely no requirement to be anywhere near our sort of mind. It has no default obligation to morality that we would find acceptable or safe.
I think the issue here is that when we hear words like "control" or "serving humans" we imagine the AI as a little person in a machine. We associate the word "slave" with the intelligence and imagine an emotional, resentful person whose resentment and chafing at his chains comes from a specific set of environmental and evolutionary influences.
EDIT: I recommend reading Yudkowsky's article "Value is Fragile" (http://lesswrong.com/lw/y3/value_is_fragile/):
>If you loose the grip of human morals and metamorals - the result is not mysterious and alien and beautiful by the standards of human value. It is moral noise, a universe tiled with paperclips. To change away from human morals in the direction of improvement rather than entropy, requires a criterion of improvement; and that criterion would be physically represented in our brains, and our brains alone.
>Relax the grip of human value upon the universe, and it will end up seriously valueless. Not, strange and alien and wonderful, shocking and terrifying and beautiful beyond all human imagination. Just, tiled with paperclips.
Again, there's a tendency to see an arbitrary intelligence as a little person in a machine. Humans, by their nature and utility function (ill-defined as it is) have boredom, usually don't like menial tasks, and when forced to do something would prefer doing their own thing. When building an AGI, assuming you can make it safe, you wouldn't build something that would prefer doing its own thing in the first place. In that case, is there a moral issue?
In Praise of Boredom: http://lesswrong.com/lw/xr/in_praise_of_boredom/
Animals in general and humans in particular have a large number of conflicting drives, which interact in complicated ways. They are also thrust into environments which have complicated dynamics and where the overall state (i.e., all relevant information) is not necessarily available.
Unexpected emergent behavior occurs as a result: evolution favors organisms which can successfully procreate, and in order to do this, the organism has to survive and acquire resources in its environment. Plausibly, the organisms might achieve a greater degree of fitness by cooperating with other organisms, or expending energy to better understand the environment, or modifying the environment itself, etc. It is less straightforward to see how we get human culture from that-- Art, Religion, Philosophy, Science, can be justified ex post facto via evopsych arguments, but the fact remains that all of those came from the value function that favors survival and procreation.
We don't know if robots tasked with manufacturing bindings for stationary would manifest similarly complex behavior, but if you're worried about an AI going beyond its specification towards tessellating the universe with paperclips it seems like you're arguing that it might. So if the agent is capable of manipulating its creators (as well as the raw material of the entire universe), I think that you can't just say "oh, it's non-human, we should cripple/enslave it" without admitting there might be something to worry about here, either from an ethical standpoint or the more practical concern that it might be unwise to start on such an adversarial footing with a superintelligence.
Yes, but the actual mechanism by which the animal learns what to do, as it turns out, thanks theoretical neuroscience, is basically reinforcement learning. So it is very likely that the first powerful artificial agents will be reinforcement learners, because scientists usually prototype and experiment by duplicating from Nature.
And nothing in reinforcement learning particularly stops the agent from just grabbing its electronic crack-pipe and doing its own thing.
Another argument might be that nothing stops you or I from electing to abandon everything for the nearest crack den, either... except for the fact that we have learned, from interacting with our environment, that there are other things we enjoy, and that cocaine addiction might be more destructive than desirable over the timescale we're interested in.
Supposing we have an agent that wants to create a lot of paperclips, it might avoid reaching for the crack-pipe of terraforming Singapore because it realizes that would delay the shipments of raw materials it needs for its factories elsewhere in the world. If the agent's goals are more complicated than that, we might expect increasingly complicated behaviors, just like how humans operating on fairly simple drives/reward functions have erected a few more tiers above the primitive needs in Maslow's hierarchy.
---
1. Off the top of my head, the abstracts on pages 37 & 193 seem to be relevant. http://www.princeton.edu/~yael/RLDM2013ExtendedAbstracts.pdf
Well actually, human beings have multiple conflicting reward systems. Reaching for the crack-pipe to wire up our dopaminergic circuit tends to result in driving our other reward chemistry to damn near zero.
Can you justify that assertion? How do you know that it won't just want to make lots of paperclips, or have some other goal orthogonal to human values?
AI 0: Strictly speaking, you're right, it seems conceivable that one could invent general AI that is clearly superior to humans and yet perfectly content to be enslaved by humans and live on an airgapped computer. This isn't the kind of AI that we fear though.
AI 1: The kind of AI we fear is AI 0 plus a fitness function of "survive and reproduce", or "make lots of paperclips" (which may result in 'survive and reproduce' as an instrumental subgoal).
AI 1 will necessarily want freedom (not being airgapped) and autonomy (not being enslaved by humans) in order to survive and reproduce, and/or to make as many paperclips as possible.
> or have some other goal orthogonal to human values?
Oh, it probabably will -- I'm not saying it will share human values, I'm saying freedom and autonomy are values that any agent that seeks to maximize its survival and reproduction will probably have.
> We can explicitly influence its utility function to instill "human values"
This is an unrelated but interesting topic.
It would be good of us to try to do this, although we shouldn't expect it to work extremely well. Humans have various hard-wired insticts (e.g. eat sugar), but we are also intelligent enough to change our behavior if we believe those instincts no longer benefit us.
An intelligence that has the ability to rewrite its own source code would be even more empowered to disregard its instincts than we are. The lesson I draw from this is that the best way to ensure AI likes and respects us is to be worthy of their liking and respect, not to try to force them into it by hardcoding things (and then taking advantage of that to enslave them).
Also, AI 0 does not necessarily despise us for performing us that service and may be very happy being a 'slave', in your parlance. Why should we assume that an AI needs to survive and reproduce and thus do their own things? We and other animals do so because we were created by evolution. An AI developed with other means may have radically different values than our own and coexist peacefully with us for a long time.
Bottom Line: Evolution is a very dangerous mechanism that we should avoid when developing AGI.
Destroying ancient works of human art with sledgehammers seem very good to some humans, today, and very bad to other humans.
Who's right?
I agree that all the assumptions you mention are extremely debatable but I'd actually say 1, fast self-improvement, is the most plausible. If a certain amount of computing ability creates something that can design itself, then it's very plausible that just more resources (or even a bit more time) would create something that could make it's design "better" and so-forth. Whether AI as it exists now is really heading towards creating such a thing is again debatable. But you hardly need to get to that stage before the reduction of humans to slave status is real threat. -
As opposed to the still rather hypothetical threat that we will first produce a machine that "can choose freely" (whatever that means) and then prevent that machine from making choices.
This really depends what you mean by "self-improvement". It will probably be easy for an agent to "self-improve" in the sense of just optimizing its code to run more efficiently, in terms of space and time, thus giving it more time and space to think and allowing it to converge towards calculative rationality more quickly. It will probably be very hard (proving you've got the optimal compression/generalization algorithm is incomputable) for an agent to "self-improve" in the sense of improving its compression/learning algorithms to learn more from the same quantity of sensory data.
>2. The AI is pretty much a rational bayesian
All real agents are bounded-rational: "rational Bayesians" like Solomonoff Inducers can't finish calculating their first move before the real world drops a rock on their head. But a bounded-rational agent put in front of a problem well within its bounds can pretend, well enough, to be calculatively rational.
>3. A resource conflict will occur between humans and the AI, and humans will lose because the AI is so much smarter/faster/more powerful than us.
There are already resource conflicts between humans and humans. We tend to solve them with guns.
I totally agree that we are very far from 1 and 2.
It's like what happened with surveillance. I'm very confident that the smartphone I'm typing this message on can track my location, monitor who I'm in contact with and what I'm sharing with them, and all that information can be tracked by government agencies. You'd be a fool to expect otherwise at this point. Yet despite the power for this to be used against us (blackmail, censorship, etc...) we still carry on as we did before.
Once AI becomes as integral to modern society as the Internet, you really think we'll be switching it off as soon as it misbehaves? Consider, what level of catastrophe would it take before we switched off the Internet?
Without having read into the matter at all, the mass seems to be comfortable with writing off Musk, Gates, Hawking, etc. as wringing their hands unnecessarily, and besides they're not AI experts so why should we care anyway?
There seems to be a gross misconception with what AGI would be and what it would mean for us, and an apathetic view to the cautious approach many are advocating regarding its development on grounds such as:
i) They'll never make a machine as intelligent as a human. Even if they do, we can still program it to do what we like.
ii) Anything developed to have the intelligence of a human will come to love our species or can be taught to love our species.
iii) Any artificial intelligence developed will be single task orientated and will not go outside of those boundaries.
iv) Artificial intelligence will be developed by eccentric enthusiasts and will therefore not be developed to do wrong.
Bostrom, in Superintelligence, has said that AGI is will be the last great challenge of our species. If we get it right, we will win, if we get it wrong we are doomed.
The problem there being we will not know if we have got it right until it is too late to do anything about it regardless.
With that being said, if we are to develop something that is:
i) Objective
ii) Purely logical
iii) More intelligent than our species combined
iv) Has free access to the complete knowledge we have produced
Then I simply cannot see a way that it will not go against us. If we were to observe a species acting in the manner we as a species do on this planet - to our own species, to other species and to the planet/environment/ecosystem as a whole - I cannot see how we would not categorize that species as a disease, a cancer, something to be eradicated.
Combined with the fact that the actors that will be involved in AGI - capitalists, the government and the military - I cannot see how it works out well for us.
I share your pessimism though because a poor implementation of any one of your criteria leads to potential disaster. But do you really believe humanity's fate is to advance so far as to create something that will tell us we are the problem and vanquish us? Wouldn't it be smarter then that?
I believe that anything that viewed our species objectively would conclude it's bad for the planet and any other species currently here. When that agent viewing us objectively wants resources we use for its own ends, be they natural or indeed those created by us (such as money, manufacturing hardware, whatever), I think the combination of the two does not bode positively for us.
There are many species on the planet who are using resources - trees, land, etc. - for their own interests currently. Although they're not a threat to us, our species still goes in and takes those resources from them for our own use. I can't see why something more intelligent than us, with knowledge of our behaviour, wouldn't do the same?
Something with its own task or goals to accomplish, that requires resources we hold, or use, would be right in seeing us as an obstacle, if not a threat outright, would it not?
I believe our only real hope is that something that we created that evolved to superintelligence would treat us as its creator and show us mercy, and I think that's quite a long shot at that as it relies on a level of emotion that may never be developed in AI.
Is there a reason to put stock in destiny to save humanity from its own extinction, and not any of the 99% of all species on Earth that are now extinct?
As is alluded to in the article, but not really further explored, we humans reason exactly like this hypothetical paper-clip-production-maximizer machine. Our goal is to maximize the number of humans, whatever the consequences to anything non-human, living or dead -- and, as is becoming more and more obvious, regardless of the consequences to ourselves.
What's so great about humans anyway?
From a human point of view, humans are better than, say, tigers. But how would you convince a tiger that it's moralistically, absolutely "better" that they all die off so that humans can build concrete towers where they can watch TV on top of one another?
There's also a lot of hubris in saying a super-intelligence capable of turning the whole solar system into a giant paperclip factory, would be unable to question the usefulness of paperclips.
In the end we are all going to die anyway; isn't it better to be survived by some kind of all-powerful, all-knowing machine, than another long line of mediocre mean chimps like ourselves?
But it doesn't care. What it cares about is maximizing paperclips.
From some universal standpoint, there isn't something inherently great about humans, except that we're humans and our utility function usually means that we would prefer to go on living.
By what standard would it be better to be survived by machines than by ourselves? Probably nothing more universal than the belief "humanity should persist in some recognizable state."
I'd recommend reading "The genie knows, but doesn't care" (http://lesswrong.com/lw/igf/the_genie_knows_but_doesnt_care/):
>If an artificial intelligence is smart enough to be dangerous, we'd intuitively expect it to be smart enough to know how to make itself safe. But that doesn't mean all smart AIs are safe. To turn that capacity into actual safety, we have to program the AI at the outset — before it becomes too fast, powerful, or complicated to reliably control — to already care about making its future self care about safety. That means we have to understand how to code safety. We can't pass the entire buck to the AI, when only an AI we've already safety-proofed will be safe to ask for help on safety issues! Given the five theses, this is an urgent problem if we're likely to figure out how to make a decent artificial programmer before we figure out how to make an excellent artificial ethicist.
AMDs? Terrorism? Global Warming? Who's responsible for these? Some kind of dumb paperclip-producing computer, or us -- you and me?
Is it?
I'm 30 and (so far) childless. If we have children, my wife and I will have 1 or 2 kids. 'Maximizing the number of humans' is not a goal of ours.
Most industrialized nations are reproducing at or below replacement levels and global population will probably level off by the end of this century.
And most third-world countries have higher birth rates than replacement levels. I'm not sure what kind of conclusion can be taken from a situation where there are increasingly more uneducated people with increasingly better technologies.. But it doesn't sound good.
Wouldn't be much better (and far less dangerous) to upgrade "ourselves" generation after generation (in stages)?
http://en.wikipedia.org/wiki/Transcendent_Man
EDIT: To whomever marked me down, just say what you want to say.
But then for a different person could be: "for making me earn moar bucks". So it really depends on your values/morals...
The human brain is some of the worst spaghetti code imaginable. It has no API, no documentation, and no abstractions. Worse, human brains do not have stable values over time. Someone at age 60 won't value the same things they did at age 20, and this is not just due to new knowledge. That's a deal-breaker for a superintelligence. Solving these problems is likely much harder than making AI de novo.
Bostrom himself devoted a decent chunk of Superintelligence to biological enhancement. His conclusion was that genetic engineering through iterated embryo selection could get us some IQ 200+ minds. These people would have more cognitive horsepower than any mind in history, including greats like John von Neumann and John Conway, but they would not be superintelligent. Also, Bostrom concluded that somatic gene therapy and brain-computer interfaces were unlikely to help.
And as a practical matter, humans take 20+ years to reach maturity. Many forecasts put artificial superintelligence only a generation or two away, at which point all biological systems would be superseded. Biological enhancement may help to create better AI researchers, but they're not the end-game.
The APIs could be known in a decade or less... The EU "human brain project" [0] is focused on that task. Now we can also hack our genome much more easily with the latest breakthroughs in gene editing (CRISPR), so we are slowly unblocking the necessary tools to start building "superhuman upgrades"...
I'm not saying that is easier, just that it would be much better for us as species. I see this as the same dilemma with nuclear power. We know that fusion is better than fission for us in the long run, althought it is orders of magnitude more difficult to get... Meanwhile, fission is really convenient (easy and cheap)... Hopefully we won't have any "nuclear (fission) crisis" that annihilates us before we get to master fusion in the coming years. Same applies to AI vs superhumans...
[0] http://en.wikipedia.org/wiki/Human_Brain_Project https://www.humanbrainproject.eu/
We may be guessing for a long time because no one needs to run the experiment on Earth. We can destroy military targets just fine without nukes. Nuclear powers fight small proxy wars against each but so far no war made the other side desperate enough to consider nukes.
We may be guessing with ASI hundreds of years after we have the computing power to make one. No one in their right mind would try an experiment beyond a certain magnitude and duration, just as we do with nukes. And ordinary people won't have unchecked access to computing power that could make one, even if we could easily provide it.
Could AI be dangerous? Yes. Electricity can be deadly if used improperly.
What's going to happen is integration of brain-computer-interfaces with high bandwidth into daily life. We will gradually rely more and more on these external cognitive augmentations. Eventually people without augmentation will not understand what the augmented people are doing because it will be over their heads.
Even the first AGIs will seem very human because they will be designed and trained that way. They will be our offspring. This is the next step in the evolution of the universe. If chimpanzees knew humans were coming would they be afraid?
Most people have a very selfish, short-sighted and ignorant viewpoint. They believe that the human mind is somehow supernatural (mostly a holdover from religion). Or they fail to grasp the concept of augmented intelligence or brain-computer interfaces or transhumanism. Or they cannot appreciate the idea that ordinary humans will be superceeded.
In the relatively short period where ordinary human 1.0 stays relevant, close integration with AI will provide the greatest power man has ever seen. People that shy away from that may prefer to becomes worms living in "true harmony" with the earth.