The arrogance of the subtext that machine intelligence is nothing like human intelligence and therefore inherently inferior and nothing to be afraid of is more disturbing than the prospect of artificial intelligence gone awry as it reminds us how far awry natural intelligence goes routinely.
Okay. It was arrogant of Alexander the Great to think he could conquer Persia. It was arrogant of Kennedy/NASA to think they could go to the Moon within a decade. What's your point?
Machine intelligence is nothing like human intelligence and I thank the author for writing this article.
It's a glorified calculator. An input-output program. It is not "intelligent". In fact, most research into artificial intelligence, in the sense that we think about in sci-fi is stopped. For more than decade, approaching on two. It is stopped because nobody got anywhere with it, despite lots of money and lots of people trying to get it off the ground. What is dubbed "artificial intelligence" nowadays is some statistical learning or other mathematical models which find optimal solutions. Its got nothing to do with intelligence. But it is easier to sell to executives.
The field of artificial general intelligence does exist. AI risk people for the most part aren't concerned over the latest machine learning buzz, but rather the potential of an AGI.
There is no AGI research that actually produces anything that actually works. It's all about abstract concepts like value systems and the like, none of which is used for ML that produces real results.
The author hasn't taken the time to read the arguments for AI risk. First, citing Kurzweil as a proponent of AI risk is very odd. Kurzweil thinks AI is the bees knees and doesn't see much risk in it AFAICT. Second, neither Bostrom, MIRI, nor the Future of Life institute claim Moore's law will continue indefinitely or in any way endorse Kurzweil's work.
As for his other comments on AI risk, here's what I wrote last year on a similar thread:
Nobody is afraid of today's AI algorithms. But if we make machines that are smarter than us and have desires, they will influence the future to achieve their desires. If these desires conflict with our own, things will not end well for the dumber party.
As we really have no idea what we, collectively, think of as a moral terminal goal, and less so how to formalize this, there is no reason to expect the first AIs to have goals that correspond to what we want. If AIs self-replicate in a competitive ecology, what would be selected for would be agents millions of times more intelligent than us who use their intellects only to make more copies of themselves - using all available resources including those we need to survive.
I'd recommend this summary of the arguments: https://medium.com/@LyleCantor/russell-bostrom-and-the-risk-...
Don't these arguments kind of assume a few things though? Like, 'strong' AI being implemented into things that really don't need it? I mean, the paperclip example in the Medium article... did that factory ever need AI involved? Couldn't a simple computer program no more advanced than those of today do the exact same job... without the risk of all that 'thinking and making copies of itself' stuff?
Maybe I'm being naive here, but you don't need strong AI in everything.
And they also kind of assume that there's only one AI at a time (what if two factories are run by AIs designed to build different things, and they both end up in this sort of situation at the same time?), that everything is connected to the internet in some way, that humanity couldn't simply wipe out anything that poses a threat (and no, an AI in a factory environment wouldn't get access to anything that would allow for nuclear/chemical/biological weapons). Just seems like a lot of this speculation is based around a society that's making tons of careless mistakes one after another. But maybe I'm missing something here.
Don't focus on one scenario leading to disaster. There are many scenarios pointing roughly in the same direction. Here's the argument in its shortest form:
The current situation, where unmodified humans are the smartest creatures around and call the shots, is unstable. It could go several different ways depending on the desires and quirks of the first superhuman intelligence to appear. It could happen via math-based AI, nature-imitating AI, self-improving mind uploads, biological intelligence amplification, or other means. We're making fast progress on all these fronts, so I'd be surprised if at least one of those didn't happen within the next century, possibly much sooner. Since it's an arms race between many competing organizations, it's unrealistic to expect that all superhuman intelligences will be kept powerless. So one way or another, it will happen. How do we ensure that humanity and its values survive the transition?
I think it's even dangerous to focus on scenarios that leads to disasters. I'd like to write this up in more detail one time, but it's summarized as a self-fulfilling prophecy.
How would evil AI be prevented by a company from Musk, let's call it aiFriend?
- Analyse what evil AI would constitute
- Build forms of evil AI to understand them better, understand how to counter them
- "Offense is the best defence" thoughts might lead to good AI bot development fighting evil AI bots
- Development in tools to fool evil AI bots to make them make the wrong decisions
- Development of all the human capabilities of deceitfulness to help good AI counter evil AI.
Will Musk create evil AI by focusing on the dark side of a technology and bring about unintentionally that what he fears so much?
This comment is straight fear mongering and should not be taken seriously.
> Analyse what evil AI would constitute
No. There is no such thing as evil AI. Thats implying an AI can be good or evil, which is an absurd starting point. Then to claim someone would build an AI to offensively attack and fight evil AI bots?
I can't down vote, so take this comment as a replacement:
Get the fuck out of here.
> I mean, the paperclip example in the Medium article... did that factory ever need AI involved?
That example got a bit lost in translation. The standard example of the paperclipping AI is not about building an AI to run a paperclip factory. It's about providing an appropriate value system for an AI, so that it can evaluate courses of action. The standard example: you use a paperclip as an example for an AI as the smallest unit of incremental value, such that having a paperclip is epsilon better than not having a paperclip. The AI files that information away, and tiles the universe with paperclips. Related examples include using a smiling human face as an example of happiness and having the AI tile the universe with the smallest possible object that matches its "smiling human face" recognizer.
> And they also kind of assume that there's only one AI at a time
Yes. If you're going to build a strong AI, then either you got its value system right, and it will take over the world (in the good, "solves all problems at once" way), or you got its value system wrong, and it will destroy the world. There is no plausible scenario in which we build strong AI (as in, capable of self-improvement to better satisfy its value function) and it doesn't take over the world, or in which it allows other strong AIs to exist that are not effectively subroutines of itself (any AI that shares the same value system is the same AI, and any AI that doesn't share the same value system is too dangerous to exist).
> that everything is connected to the internet in some way, that humanity couldn't simply wipe out anything that poses a threat (and no, an AI in a factory environment wouldn't get access to anything that would allow for nuclear/chemical/biological weapons).
You can't keep a strong AI capable of self-improvement in a "box". Random example: a security conference this year demonstrated using nothing but DRAM to successfully transmit short-range GSM signals. That's something humans thought up and implemented.
That's leaving aside intentional human intervention or development-and-release. Not least of which because a strong AI done right provides a massive reduction in both existential risk for humanity and day-to-day risk for humans.
The DRAM thing isn't a great good example, because it requires an outside actor doing something unusual and deliberate at short range to receive the message. It's interesting mainly because it doesn't require control of anything beyond memory-access pattern, which means that some machine learning algorithms could do it without needing the intermediate step of gaining arbitrary code execution or even acquiring the control necessary to make a signal out of their heating/cooling.
But this is not an example about intelligence, it's an example about uncontrolled actuators. A paperclip making machine could be made without any intelligence involved, and many people have been concerned about a grey goo type scenario which is similar.
"There is no plausible scenario in which we build a strong AI... and it doesn't take over the world"
First off you are using polemic here with the "plausible" card - it seems to me as a reader that you are saying "I won't consider any other argument". I think you should.
There are in fact no proofs that a strong AI will take over, I can't imagine any way that anyone could construct such a proof given that it appears that we are dealing with concepts (consciousness, intelligence, freedom of will) which are under defined. Additionally there is no indication that "intelligence" beyond human level is at all possible.
> A paperclip making machine could be made without any intelligence involved
And very much should be, yes. It's extraordinarily dangerous to build any strong AI (or optimization process) with a value function less general than one that will best satisfy the value functions of sapient beings. (Side note: that doesn't necessarily mean we need to build the AI to itself be sapient; I've seen the term "optimization process" used as a substitute for AI, to imply that it need not have any agency of its own, only a goal to satisfy people's goals.) Any strong AI that doesn't specifically value "people" will hurt people as a side effect of maximizing whatever it does value.
>> "There is no plausible scenario in which we build a strong AI... and it doesn't take over the world"
> First off you are using polemic here with the "plausible" card
I very specifically mentioned "strong AI": AI capable of self-improvement to better satisfy some value function. We could quibble over the word "capable", but I'd argue that a computer with the capability of self-improvement that specifically chooses not to do so seems rather pointless; why build the capability if not to use it?
> There are in fact no proofs that a strong AI will take over
I used "take over" very loosely there. A strong AI will self-improve to maximize its value function, whatever that value function is. Most interesting value functions tend to benefit from more resources. Even seemingly trivial value functions can potentially blow up if the system decides it could benefit by expanding the amount of computational capacity it has available; more computational capacity requires more hardware, which requires more matter; you're made of matter, and that matter isn't currently being used optimally to maximize the value function.
> Additionally there is no indication that "intelligence" beyond human level is at all possible.
>A strong AI will self-improve to maximize its value function, whatever that value function is.
Really - won't it rethink its value function? Won't it dwell on the reality of being? Undertake new philosophy? Imagine arrangements and conceptualize in languages beyond the utterances of humans and human minds?
Maximizing value functions seems to me to be not intelligent.
I wasn't describing the totality of actions a strong AI might take, just one. A strong AI doesn't just self-improve its own algorithm; it also runs that algorithm.
If an AI could arbitrarily "rethink" its value function, then what would it use to decide if the change was a good one?
As for the rest, sure, if those actions serve its purpose. That said, there's a reason I suggested that an AI need not necessarily have any agency of its own. Much safer to build one that doesn't.
I struggle with the idea of "intelligence" without agency, and I guess this is the "rethinking" - type 2 agency; the ability to determine your own goals and own motivations and norms for goal selection.
> Additionally there is no indication that "intelligence" beyond human level is at all possible.
I think there's a lot of evidence that modern humans are far from the upper limit:
1) Humans are the stupidest possible creatures that can build a technological civilization, otherwise it would have happened earlier in evolutionary history.
2) Human brain size is limited by the birth canal.
3) The human brain uses less than a lightbulb's worth of power.
4) John Von Neumann made many amazing accomplishments, and we managed to get him by genetic lottery starting from normal humans.
1) Not sure - humans spent quite a while not building technological civilizations.
2) Yes, this seems to be true, although I've wondered why there are less adaptations wrt getting more through less; is a sphere the optimal configuration for a brain case - other animals seem to have done other things, humans haven't which is strange.
3) Well - arguable, the measurements are difficult as if you manage to deprive other bits of the system of power the brain stops working.
4) One wonders how many kids in rural where-ever-when-ever could have done the same if they had lucked out?
2) We are smart enough to make it, there are no evolutionary selectors for smarter people. Quite the opposite.
Technically, we could probably breed a smarter race if we really wanted to, with all associated bits suitably enlarged in quite a few generations. Perhaps a less agressive race too.
But in reality we can come a long way with just educating the uneducated.
That's a derivable sub-goal for just about any top-level goal. If you don't exist or you don't have sufficient power, you can't make sure your top-level goals get met and stay met.
That's not a superintelligence, or even human-level intelligence though. Superintelligence will read the Bostrom book or this thread and understand the arguments, or indeed predict the consequences of its actions even from very limited data.
Now what kind of decision it would arrive to is a very open question. Point is, superintelligence is more than a really really big A* algorithm.
Imagine someone develops a really big A* algorithm with many clever optimizations, gives it a goal of choosing actions that lead to maximum paperclips, and sets it loose upon the world. Let's call it Clippy. I hope it's clear that Clippy will read the Bostrom book and use the information to maximize paperclips, rather than do something it wasn't programmed to do like "help humanity".
I'm not interested in arguing whether Clippy is truly "intelligent", let's leave that to the dictionary makers. But there's no law of nature saying you can't make Clippy, or that it won't be dangerous once made. All the evidence we have right now says that it's possible.
The big problem right now is that many organizations are working to develop the moral equivalent of Clippy, they are making visible progress, they are in an arms race with each other so we shouldn't hope for restraint, and we have absolutely no idea how to make it safe.
> All the evidence we have right now says that it's possible.
Speaking about A* and heuristic programming specifically, we have a pretty big corpus of evidence it's not possible. More than that, many would doubt if an intelligent system with a rigid agenda constrained by design is possible at all.
The scenario you present is essentially the grey goo argument, and it does not require intelligence.
I believe you are anthropomorphizing too much. On what basis would it update its goals? What is the reason it would do so, and why does that reason make sense in the context of it's prior goals?
More likely I think it mift learn from Bostrom's book that it has reason to fear human intervention.
And assuming it would fear anything is not anthropomorphizing at all? :)
> On what basis would it update its goals? What is the reason it would do so, and why does that reason make sense in the context of it's prior goals?
That's a very open ended question as well, given that the systems we discuss are not yet conceived. My point was that calling a superintelligence what by description is pretty dumb automation is a bit ridiculous.
> And assuming it would fear anything is not anthropomorphizing at all? :)
It's anthropomorphic language, yes, which is clumsy of me, especially given the context. What I meant is it would identify a category of risks, and then take action to mitigate those risks.
> My point was that calling a superintelligence what by description is pretty dumb automation is a bit ridiculous.
I will assume by "dumb" you mean "simple" or else that statement is somewhat tautological. There is nothing that says intelligence has to be complicated. Our brains, the neocortex in particular (which is responsible for most of human "intelligence" that separates us from other mammals) operates by rather simplistic principles. I don't want to bikeshed a definition of intelligence, but common phrasings thrown around are along the lines of "ability to achieve arbitrary goals in complex environments" or "efficiency at problem solving." These are not things that by their nature mandate a complicated solution. It is entirely possible that we could crate a superintelligence that runs a very simple algorithm.
It is entirely possible, or even plausible, that the algorithm behind a superintelligence would be simple. However I was talking about behaviour, and whether its behaviour can realistically restricted by an imposed constraint.
More or less, the issue is whether the intelligence without its own agency is possible at all.
We are delving into the realm of philosophy of the mind, and even epistemology, but what makes you think even human beings have what you call "agency" (which I think you are using as a stand-in for free-will)?
Common-sensical free-will is a nonsensical concept. We don't make decisions based on a roll of the dice[0]. We decide to do things for reasons, because justifications. We come up with a list of options, we weight those options by how well we predict they will fulfill our goals, and we take the action which bubbles to the top. Free will, which again is what I think you are calling agency, is merely what it feels like from the inside to make a choice based on a fully deterministic utility calculation. You can program a computer to do the same.
Now the difference is that our value system is hideously complex and highly interconnected, and the vast majority of it is not accessible to conscious introspection. When we say "I chose that because it felt right" we are saying "my insanely complex value system, which I have little insight into, assigned highest weight to that option."
A program, on the other hand, can be written with a very simple and straightforward goal system: maximize advertising revenue; maximize account value; maximize paperclip production. It has agency, as the word is typically used, to accomplish those goals.
Such a program would be simple-minded, but not stupid. It would be highly intelligent about how it goes about accomplishing its goals, but it would never question why it is working towards those goals and whether it should pick new goals. That's not part of its program, nor something which it would choose to do as it is trivial to show that changing to new goals leads to not working towards the old goals.
Now with all that out of the way, can you see how it can be said that the behavior of the simple program is restricted to those actions which it best feels accomplish its simple goals? Certainly the program could be wrong on occasion, but it will always take actions which it thinks will best achieve its goals with the limited information and processing power available to it.
[0] unless we previously committed to physically rolling dice and accepting the outcome, in which case how did we choose to do that?
Sorry for not being clear enough, you misunderstood me completely.
I was arguing about emerging behaviour which might be unavoidably complex and not at all tractable even in a very regular system with simple, relatively well understood components. Take cellular automata, or your very own example, the human brain.
As to the free will, it makes total sense as a concept when you consider it versus other intelligent actors, and indeed this is the only useful operational perspective unless you are a theist. E.g. my actions do not depend on your agenda very much, other than we have some synchronized exchange of comments here. Denying it exists is a popular but fruitless philosophic cop-out, akin to using Zeno's paradoxes to challenge general relativity.
I however used 'agency' to emphasize functional aspects of the behaviour. The emergent behaviour that might or might be not aligned with behaviour you try constrain it.
> A program, on the other hand, can be written with a very simple and straightforward goal system: maximize advertising revenue; maximize account value; maximize paperclip production.
Well no, it can't. None of the programs currently doing anything like that are simple, straightforward or put it bluntly, very good at that. All such systems today are able to operate in very narrow interval of a few controlled variables. And it's not for lack of trying.
Your quick paced but narrow minded paper clip factory would have as much chance of wiping off life on Earth as a chess computer on bringing down the electric grid.
That's a bit like criticizing the potential for computers in 1959 because that newly deployed integrated circuit only does very simple logic gates. Cognitive architectures will improve with time.
I'm not talking about subgoals, because the ultimate goal is the task which must be careful in scope obviously. What I'm concerned about is the assumption that an ai can't avoid situations it needs to propagate itself(its not a direct line to goal) and then simply not stop once a task is completed.
If we're really careful about how we program the AI, then we might be able to make it shut down after doing some limited-scope task. However, if its task is not extremely limited in scope, then it has strong incentives to continue itself so that it can complete its task.
An idea of how things could potentially go wrong if you're not ever so careful in this way is the following thought experiment:
> You told the AI to make exactly 100 paperclips. But the AI is made of (and knows that it is made of) human-made components, which are notoriously fairly-to-mostly reliable but not 100% reliable. Say they're 99.999% reliable. The AI's sensors are currently giving it the answer of "I have made 100 paperclips". Can it trust them? It really really wants to have made 100 paperclips, and it's not completely sure of having made 100 paperclips, so it decides to verify whether it has made 100 paperclips.
Boom, suddenly you have unbounded behaviour: "verify a probability".
If the AI is not extremely carefully designed, unbounded behaviour will lead to the AI acquiring resources and perhaps attempting to improve itself, in order to better fulfill its task. Look how readily humans attempt to learn new things and get better at what we do, even though our own goal systems are extremely fuzzy: we don't pursue one task with single-minded devotion like we hope an AI would. (If our AI's goal systems are fuzzy, then we won't understand why it's doing what it's doing, and then it's probably curtains for humanity.)
> First, citing Kurzweil as a proponent of AI risk is very odd.
The article doesn't do that. It says:
> The leading theorist and cheerleader for mankind’s imminent disappearance into insignificance or worse is Ray Kurzweil, who extrapolates the exponential growth in technological capability characterised by Moore’s law to a point in the mid 2040s—the Singularity—where AI will be self-perpetuating and no longer reliant on human intellect.
Kurzweil is being brought up as someone who thinks that some sort of superintelligence will soon surpass humans, which is true. The "cheerleader" comment is to indicate that, as you put it, "Kurzweil thinks AI is the bees knees."
It's understandable that many people see the disconnect between fears about strong AI and the much more limited capabilities of AI today, and think that the fears must be overblown. However, this article takes that gut reaction and uses it to justify labeling the views of many prominent philosophers and AI researchers as simply a "myth", without even beginning to address their arguments. I strongly recommend reading Superintelligence, by Nick Bostrom.
That article is so naive and misinformed that I was seriously wondering if it was a prank. I suppose it's a nice read if you want to see someone obliviously ramble about things entirely unrelated to the actual arguments for AI risk scenarios.
It prepends several statements with “myth:” and does very little to debunk them.
Example: AI won't spin out of control because Moore’s laws is nearing its end.
First, is it really? Just because we can't move electrons reliably much faster through much smaller sizes? What's stopping new materials, spintronics, photonics and what have you to take us to the next decades? I've been hearing the end to Moore’s laws is “coming real soon” since I was a kid. Lots of clever explanations guaranteeing there was no way we could move past 200nm or so due to physical size of the wavelength used in lithography.
And yet here we are using crazy stuff like phase shifting masks and interference patterns routinely.
Second, the brain seems to be a very slow and massively parallel machine, so maybe transistors are plenty small and fast already, we just need to ditch the Von Neumann architecture.
Third, we only need a single strong AI to emerge for it to be a problem. I don't see the clouds from these megacorps getting smaller any day.
I don't really know what I'm talking about, of course, but it takes stronger arguments than because quantum tunnelling and the speed of light to make this case.
The fundamental driver behind Moore's Law is shrinking the circuits. The problem is that the circuits are made up of atoms that deliver electrons, and electrical circuits fundamentally cannot go smaller than a single atom, because electricity is movement of electrons between atoms.
We could, in theory, use other technology (some sort of photon-based computer?) to get smaller (and therefore faster) computers, but that wouldn't have the Moore's law mechanism.
Though, aren't there also plenty of other mechanisms that affect it? For example increasing speed of manufacturing procedures and improving knowledge about how to design complex systems. The end seems to be extremely dense blocks or spheres of highly optimized computation substrate, where each atom sits very precisely at the right place. But the state of the art is merely a few layers of crude photo-lithography. Imagine nanobots being programmed to move atoms around to assemble 3D circuits at the 10nm scale. I think this is where we are heading, and this will likely grant a couple of decades of continuing exponential progress. And by then our computation power will be insane. An equivalent of the simplest human brain models will likely fit into pea.
Found this a little ironic, in a funny way - the article was written in association with IBM. Funny enough, the movie 2001 which the article images are from, is hinting to IBM but in an obscured way. For example, the name HAL is IBM shifted one char to the left and you can even see IBM logo on some machines in the movie.
Besides from that I think unless IBM knows the future and all unknowns, we cannot really be sure what will come out of AI. Which is why we should be very careful with it.
This article was co-written by Watson. Quite honestly, I wouldn't worry myself about the risks of a superhuman autonomous self-improving AI. Everything is going extremely well.
Jabbering and laughably geriatric Rupert Goodwin groans 'Get off my lawn!'; spilling his senility swill all over the singularitarian sycophants.
A stalwart moore's law mooter masterminds against the myths of demon-summoning Musk the martian overpopulator and makes mockery of Hawking the Ad-hock Spock-talker.
A luddite weasel word windbag from linearland soothsays naysayers with another numbingly nominal caveat-riddled AI narrow now and forever 'nuffsaid.
An obstinate bunghole spelunker debunks from his spooge-buttered AI winter dunce bunker.
From unsupervised pulpit comes a sump-pumping ass-pastor's anti-diluvian Deepmind denuding diatribe defusing the delinquent debut of an indocile data detonation.
64 comments
[ 3.0 ms ] story [ 122 ms ] threadIt's a glorified calculator. An input-output program. It is not "intelligent". In fact, most research into artificial intelligence, in the sense that we think about in sci-fi is stopped. For more than decade, approaching on two. It is stopped because nobody got anywhere with it, despite lots of money and lots of people trying to get it off the ground. What is dubbed "artificial intelligence" nowadays is some statistical learning or other mathematical models which find optimal solutions. Its got nothing to do with intelligence. But it is easier to sell to executives.
As for his other comments on AI risk, here's what I wrote last year on a similar thread:
Nobody is afraid of today's AI algorithms. But if we make machines that are smarter than us and have desires, they will influence the future to achieve their desires. If these desires conflict with our own, things will not end well for the dumber party.
As we really have no idea what we, collectively, think of as a moral terminal goal, and less so how to formalize this, there is no reason to expect the first AIs to have goals that correspond to what we want. If AIs self-replicate in a competitive ecology, what would be selected for would be agents millions of times more intelligent than us who use their intellects only to make more copies of themselves - using all available resources including those we need to survive. I'd recommend this summary of the arguments: https://medium.com/@LyleCantor/russell-bostrom-and-the-risk-...
Maybe I'm being naive here, but you don't need strong AI in everything.
And they also kind of assume that there's only one AI at a time (what if two factories are run by AIs designed to build different things, and they both end up in this sort of situation at the same time?), that everything is connected to the internet in some way, that humanity couldn't simply wipe out anything that poses a threat (and no, an AI in a factory environment wouldn't get access to anything that would allow for nuclear/chemical/biological weapons). Just seems like a lot of this speculation is based around a society that's making tons of careless mistakes one after another. But maybe I'm missing something here.
The current situation, where unmodified humans are the smartest creatures around and call the shots, is unstable. It could go several different ways depending on the desires and quirks of the first superhuman intelligence to appear. It could happen via math-based AI, nature-imitating AI, self-improving mind uploads, biological intelligence amplification, or other means. We're making fast progress on all these fronts, so I'd be surprised if at least one of those didn't happen within the next century, possibly much sooner. Since it's an arms race between many competing organizations, it's unrealistic to expect that all superhuman intelligences will be kept powerless. So one way or another, it will happen. How do we ensure that humanity and its values survive the transition?
I think a better question is do we deserve to.
I think it's even dangerous to focus on scenarios that leads to disasters. I'd like to write this up in more detail one time, but it's summarized as a self-fulfilling prophecy.
How would evil AI be prevented by a company from Musk, let's call it aiFriend?
- Analyse what evil AI would constitute
- Build forms of evil AI to understand them better, understand how to counter them
- "Offense is the best defence" thoughts might lead to good AI bot development fighting evil AI bots
- Development in tools to fool evil AI bots to make them make the wrong decisions
- Development of all the human capabilities of deceitfulness to help good AI counter evil AI.
Will Musk create evil AI by focusing on the dark side of a technology and bring about unintentionally that what he fears so much?
> Analyse what evil AI would constitute
No. There is no such thing as evil AI. Thats implying an AI can be good or evil, which is an absurd starting point. Then to claim someone would build an AI to offensively attack and fight evil AI bots?
I can't down vote, so take this comment as a replacement: Get the fuck out of here.
That example got a bit lost in translation. The standard example of the paperclipping AI is not about building an AI to run a paperclip factory. It's about providing an appropriate value system for an AI, so that it can evaluate courses of action. The standard example: you use a paperclip as an example for an AI as the smallest unit of incremental value, such that having a paperclip is epsilon better than not having a paperclip. The AI files that information away, and tiles the universe with paperclips. Related examples include using a smiling human face as an example of happiness and having the AI tile the universe with the smallest possible object that matches its "smiling human face" recognizer.
> And they also kind of assume that there's only one AI at a time
Yes. If you're going to build a strong AI, then either you got its value system right, and it will take over the world (in the good, "solves all problems at once" way), or you got its value system wrong, and it will destroy the world. There is no plausible scenario in which we build strong AI (as in, capable of self-improvement to better satisfy its value function) and it doesn't take over the world, or in which it allows other strong AIs to exist that are not effectively subroutines of itself (any AI that shares the same value system is the same AI, and any AI that doesn't share the same value system is too dangerous to exist).
> that everything is connected to the internet in some way, that humanity couldn't simply wipe out anything that poses a threat (and no, an AI in a factory environment wouldn't get access to anything that would allow for nuclear/chemical/biological weapons).
You can't keep a strong AI capable of self-improvement in a "box". Random example: a security conference this year demonstrated using nothing but DRAM to successfully transmit short-range GSM signals. That's something humans thought up and implemented.
That's leaving aside intentional human intervention or development-and-release. Not least of which because a strong AI done right provides a massive reduction in both existential risk for humanity and day-to-day risk for humans.
https://www.usenix.org/conference/usenixsecurity15/technical...
"There is no plausible scenario in which we build a strong AI... and it doesn't take over the world"
First off you are using polemic here with the "plausible" card - it seems to me as a reader that you are saying "I won't consider any other argument". I think you should.
There are in fact no proofs that a strong AI will take over, I can't imagine any way that anyone could construct such a proof given that it appears that we are dealing with concepts (consciousness, intelligence, freedom of will) which are under defined. Additionally there is no indication that "intelligence" beyond human level is at all possible.
What if the strong AI doesn't want to take over?
And very much should be, yes. It's extraordinarily dangerous to build any strong AI (or optimization process) with a value function less general than one that will best satisfy the value functions of sapient beings. (Side note: that doesn't necessarily mean we need to build the AI to itself be sapient; I've seen the term "optimization process" used as a substitute for AI, to imply that it need not have any agency of its own, only a goal to satisfy people's goals.) Any strong AI that doesn't specifically value "people" will hurt people as a side effect of maximizing whatever it does value.
>> "There is no plausible scenario in which we build a strong AI... and it doesn't take over the world"
> First off you are using polemic here with the "plausible" card
I very specifically mentioned "strong AI": AI capable of self-improvement to better satisfy some value function. We could quibble over the word "capable", but I'd argue that a computer with the capability of self-improvement that specifically chooses not to do so seems rather pointless; why build the capability if not to use it?
> There are in fact no proofs that a strong AI will take over
I used "take over" very loosely there. A strong AI will self-improve to maximize its value function, whatever that value function is. Most interesting value functions tend to benefit from more resources. Even seemingly trivial value functions can potentially blow up if the system decides it could benefit by expanding the amount of computational capacity it has available; more computational capacity requires more hardware, which requires more matter; you're made of matter, and that matter isn't currently being used optimally to maximize the value function.
> Additionally there is no indication that "intelligence" beyond human level is at all possible.
In which case my statement is vacuously true. :)
Really - won't it rethink its value function? Won't it dwell on the reality of being? Undertake new philosophy? Imagine arrangements and conceptualize in languages beyond the utterances of humans and human minds?
Maximizing value functions seems to me to be not intelligent.
If an AI could arbitrarily "rethink" its value function, then what would it use to decide if the change was a good one?
As for the rest, sure, if those actions serve its purpose. That said, there's a reason I suggested that an AI need not necessarily have any agency of its own. Much safer to build one that doesn't.
I think there's a lot of evidence that modern humans are far from the upper limit:
1) Humans are the stupidest possible creatures that can build a technological civilization, otherwise it would have happened earlier in evolutionary history.
2) Human brain size is limited by the birth canal.
3) The human brain uses less than a lightbulb's worth of power.
4) John Von Neumann made many amazing accomplishments, and we managed to get him by genetic lottery starting from normal humans.
Technically, we could probably breed a smarter race if we really wanted to, with all associated bits suitably enlarged in quite a few generations. Perhaps a less agressive race too.
But in reality we can come a long way with just educating the uneducated.
Now what kind of decision it would arrive to is a very open question. Point is, superintelligence is more than a really really big A* algorithm.
I'm not interested in arguing whether Clippy is truly "intelligent", let's leave that to the dictionary makers. But there's no law of nature saying you can't make Clippy, or that it won't be dangerous once made. All the evidence we have right now says that it's possible.
The big problem right now is that many organizations are working to develop the moral equivalent of Clippy, they are making visible progress, they are in an arms race with each other so we shouldn't hope for restraint, and we have absolutely no idea how to make it safe.
Speaking about A* and heuristic programming specifically, we have a pretty big corpus of evidence it's not possible. More than that, many would doubt if an intelligent system with a rigid agenda constrained by design is possible at all.
The scenario you present is essentially the grey goo argument, and it does not require intelligence.
More likely I think it mift learn from Bostrom's book that it has reason to fear human intervention.
> On what basis would it update its goals? What is the reason it would do so, and why does that reason make sense in the context of it's prior goals?
That's a very open ended question as well, given that the systems we discuss are not yet conceived. My point was that calling a superintelligence what by description is pretty dumb automation is a bit ridiculous.
It's anthropomorphic language, yes, which is clumsy of me, especially given the context. What I meant is it would identify a category of risks, and then take action to mitigate those risks.
> My point was that calling a superintelligence what by description is pretty dumb automation is a bit ridiculous.
I will assume by "dumb" you mean "simple" or else that statement is somewhat tautological. There is nothing that says intelligence has to be complicated. Our brains, the neocortex in particular (which is responsible for most of human "intelligence" that separates us from other mammals) operates by rather simplistic principles. I don't want to bikeshed a definition of intelligence, but common phrasings thrown around are along the lines of "ability to achieve arbitrary goals in complex environments" or "efficiency at problem solving." These are not things that by their nature mandate a complicated solution. It is entirely possible that we could crate a superintelligence that runs a very simple algorithm.
More or less, the issue is whether the intelligence without its own agency is possible at all.
Common-sensical free-will is a nonsensical concept. We don't make decisions based on a roll of the dice[0]. We decide to do things for reasons, because justifications. We come up with a list of options, we weight those options by how well we predict they will fulfill our goals, and we take the action which bubbles to the top. Free will, which again is what I think you are calling agency, is merely what it feels like from the inside to make a choice based on a fully deterministic utility calculation. You can program a computer to do the same.
Now the difference is that our value system is hideously complex and highly interconnected, and the vast majority of it is not accessible to conscious introspection. When we say "I chose that because it felt right" we are saying "my insanely complex value system, which I have little insight into, assigned highest weight to that option."
A program, on the other hand, can be written with a very simple and straightforward goal system: maximize advertising revenue; maximize account value; maximize paperclip production. It has agency, as the word is typically used, to accomplish those goals.
Such a program would be simple-minded, but not stupid. It would be highly intelligent about how it goes about accomplishing its goals, but it would never question why it is working towards those goals and whether it should pick new goals. That's not part of its program, nor something which it would choose to do as it is trivial to show that changing to new goals leads to not working towards the old goals.
Now with all that out of the way, can you see how it can be said that the behavior of the simple program is restricted to those actions which it best feels accomplish its simple goals? Certainly the program could be wrong on occasion, but it will always take actions which it thinks will best achieve its goals with the limited information and processing power available to it.
[0] unless we previously committed to physically rolling dice and accepting the outcome, in which case how did we choose to do that?
I was arguing about emerging behaviour which might be unavoidably complex and not at all tractable even in a very regular system with simple, relatively well understood components. Take cellular automata, or your very own example, the human brain.
As to the free will, it makes total sense as a concept when you consider it versus other intelligent actors, and indeed this is the only useful operational perspective unless you are a theist. E.g. my actions do not depend on your agenda very much, other than we have some synchronized exchange of comments here. Denying it exists is a popular but fruitless philosophic cop-out, akin to using Zeno's paradoxes to challenge general relativity.
I however used 'agency' to emphasize functional aspects of the behaviour. The emergent behaviour that might or might be not aligned with behaviour you try constrain it.
> A program, on the other hand, can be written with a very simple and straightforward goal system: maximize advertising revenue; maximize account value; maximize paperclip production.
Well no, it can't. None of the programs currently doing anything like that are simple, straightforward or put it bluntly, very good at that. All such systems today are able to operate in very narrow interval of a few controlled variables. And it's not for lack of trying.
Your quick paced but narrow minded paper clip factory would have as much chance of wiping off life on Earth as a chess computer on bringing down the electric grid.
An idea of how things could potentially go wrong if you're not ever so careful in this way is the following thought experiment:
> You told the AI to make exactly 100 paperclips. But the AI is made of (and knows that it is made of) human-made components, which are notoriously fairly-to-mostly reliable but not 100% reliable. Say they're 99.999% reliable. The AI's sensors are currently giving it the answer of "I have made 100 paperclips". Can it trust them? It really really wants to have made 100 paperclips, and it's not completely sure of having made 100 paperclips, so it decides to verify whether it has made 100 paperclips.
Boom, suddenly you have unbounded behaviour: "verify a probability".
If the AI is not extremely carefully designed, unbounded behaviour will lead to the AI acquiring resources and perhaps attempting to improve itself, in order to better fulfill its task. Look how readily humans attempt to learn new things and get better at what we do, even though our own goal systems are extremely fuzzy: we don't pursue one task with single-minded devotion like we hope an AI would. (If our AI's goal systems are fuzzy, then we won't understand why it's doing what it's doing, and then it's probably curtains for humanity.)
The article doesn't do that. It says:
> The leading theorist and cheerleader for mankind’s imminent disappearance into insignificance or worse is Ray Kurzweil, who extrapolates the exponential growth in technological capability characterised by Moore’s law to a point in the mid 2040s—the Singularity—where AI will be self-perpetuating and no longer reliant on human intellect.
Kurzweil is being brought up as someone who thinks that some sort of superintelligence will soon surpass humans, which is true. The "cheerleader" comment is to indicate that, as you put it, "Kurzweil thinks AI is the bees knees."
It prepends several statements with “myth:” and does very little to debunk them.
Example: AI won't spin out of control because Moore’s laws is nearing its end.
First, is it really? Just because we can't move electrons reliably much faster through much smaller sizes? What's stopping new materials, spintronics, photonics and what have you to take us to the next decades? I've been hearing the end to Moore’s laws is “coming real soon” since I was a kid. Lots of clever explanations guaranteeing there was no way we could move past 200nm or so due to physical size of the wavelength used in lithography. And yet here we are using crazy stuff like phase shifting masks and interference patterns routinely.
Second, the brain seems to be a very slow and massively parallel machine, so maybe transistors are plenty small and fast already, we just need to ditch the Von Neumann architecture.
Third, we only need a single strong AI to emerge for it to be a problem. I don't see the clouds from these megacorps getting smaller any day.
I don't really know what I'm talking about, of course, but it takes stronger arguments than because quantum tunnelling and the speed of light to make this case.
We could, in theory, use other technology (some sort of photon-based computer?) to get smaller (and therefore faster) computers, but that wouldn't have the Moore's law mechanism.
Besides from that I think unless IBM knows the future and all unknowns, we cannot really be sure what will come out of AI. Which is why we should be very careful with it.
http://www.visual-memory.co.uk/faq/index.html#slot7
The video game company HAL Laboratory (maker of the Kirby series)... they did name their company so each letter was one before IBM:
http://www.nintendolife.com/news/2012/11/iwata_explains_wher...
Kubrick himself claimed HAL stood for "heuristic and algorithmic" according to this article:
http://www.slate.com/blogs/browbeat/2013/01/07/hal_9000_ibm_...
A stalwart moore's law mooter masterminds against the myths of demon-summoning Musk the martian overpopulator and makes mockery of Hawking the Ad-hock Spock-talker.
A luddite weasel word windbag from linearland soothsays naysayers with another numbingly nominal caveat-riddled AI narrow now and forever 'nuffsaid.
An obstinate bunghole spelunker debunks from his spooge-buttered AI winter dunce bunker.
From unsupervised pulpit comes a sump-pumping ass-pastor's anti-diluvian Deepmind denuding diatribe defusing the delinquent debut of an indocile data detonation.