As the gentleman says, a lack of Autonomous Agents is not the thing standing between humanity and oblivion. The conspicuous surplus of super empowered agents (as in "I could leave this room and 80 million people would be dead 40 minutes later") is.
As for the Royal Society stuff. For the record it's not learning, and it's not the machines, I can say it again if anyone bothers to listen. Although given that it's not what will get news print or sell books or get on telly I am pretty sure that no one will.
Ho hum.
(edited 'cos I forgot a bracket, which was why I gave up c)
That's a false dilemma. It would be wise to defend against every existential threat in proportion to its likelihood. The threat of nuclear war is well known, and significant resources are spent on reducing its likelihood. Until very recently the same could not be said for the threat of unfriendly AI.
> That's a false dilemma. It would be wise to defend against every existential threat in proportion to its likelihood.
Lives saved versus dollars spent is a better metric. Having said that, the Singularity crowd tends to lump in all potential future lives into the saved category. The question of allocating resources to avoid existential threats very quickly turns into a Pascal's mugging in favor of AI safety (think of the trillions of future children).
> The threat of nuclear war is well known ... until very recently the same could not be said for the threat of unfriendly AI.
Enough nuclear weapons exist to kill all humans forever. Strong AI does not exist. Expert estimates range from decades to over a century. At the same time, the people making the most noise seem come from the side of avoiding existential threats and not AI research. I am (highly) skeptical they will produce anything of value. Let me rephrase. Would you expect any productive work in 1840 by philosophers concerning potential dooms day weapons yet to be created by physicists?
There doesn't exist anywhere near enough nuclear weapons to kill all humans. Even in the 1980s, people were predicting between 20 and 100 million immediate American deaths from a full-scale Soviet attack, and since then there are much fewer nuclear weapons. See e.g. the discussion in this comment thread: https://slatestarcodex.com/2015/10/31/ot32-when-hell-is-full...
It has been suggested that humans recovered from a population bottleneck of less than 30,000 individuals (the Toba catastrophy). A nuclear war would leave billions alive.
On the other hand, nuclear research could solve our energy needs for the foreseeable future. And AI could protect humanity against almost every possible existential risk.
Let's hope that the increased notice taken of AI, and fear thereof, doesn't suddenly shut down a major source of hope for the future.
I'm wondering how much Eliezer Yudkowsky and the LessWrong crowd were influenced by Bostrom or vice versa. This all sounds very familiar, but the dates seem earlier:
"Bostrom introduced the philosophical concept of 'existential risk' in 2002."
Bostrom brought up the orthogonality idea before Eliezer, who consequently (not immediately) changed his mind.
EDIT: I am unable to find the source. Might have been a halluzination, but I thought I remembered them having a discussion where Eliezer more or less still thought "let the AGI win, humanity be damned". Ah, my brain and I are not in a good relationship.
"Bostrom introduced the philosophical concept of 'existential risk' in 2002."
Oh, come on. This has been a theme in science fiction back to at least the 1940s. Jack Williamson's "With Folded Hands" (1947) is probably the clearest early writing on the subject.
"The car turned off the shining avenue, taking him back to the quiet splendor of his prison. His futile hands clenched and relaxed again, folded on his knees. There was nothing left to do."
Also i find it funny how much attention this theoretical problem receives, when we don't have a good explanations on what intelligence is and how it works.
Also all these stories seem to be projecting our own behavior on robots; Humans and apes like to be alpha males, because it gives them an evolutionary advantage; by extension it seems natural that Robots will also have the same drive for power; is that really true?
I think that the Robot will not have such an in built desire - if it is based on reasoning then the machine might actually be more reasonable. In other words there is enough room for both Humans and Robots.
Mr. Yudkowsky says: "Moore's Law of Mad Science: Every eighteen months, the minimum IQ necessary to destroy the world drops by one point." http://www.azquotes.com/quote/819025
The point is that our gadgets become more powerful and more potent as weapons; so as we have a good history of past abuses it is sort of easy to extrapolate to the future.
What is still don't understand: would intelligence be an inhibiting factor - like more reasonable Humans are supposedly less destructive; Maybe the same goes for machines.
I don't buy the argument about competition for resources - With enough effort you can always stretch it so that there is enough for everybody/everything.
Another interesting aspect: once upon a time people would become very agitated when discussing politics (that was when we still had ideological differences and when people thought that their stance does matter); In our time we have discussions about sci-fi instead of that.
"The first registered thoughts on the possibility of artificial, machine intelligence, becoming a risk to humanity stems from the late Industrial Revolution. It was Samuel Butler, in 1863, in his Darwin Among the Machines that suggested machines could eventually replace humans as the dominant agents on Earth"
Although he clearly didn't come up with the idea (there's the whole nuclear war thing, after all), perhaps it wasn't called "existential risk" before? Google ngram viewer show hits going back to the 1930's [1], but judging by the brief snippets, they seem to mostly be related to religion.
It has been posted many times, but none of the posts rose above the 'significant attention' threshold described in the FAQ. This one does, though, so it'll be treated as a dupe if it gets posted again.
Btw, we're working on a better approach to duplicate detection that will reduce the number of reposts and will more often give credit to the original submitter.
I'm not sure about that. A person can theoretically be scaled by adding another person to their cause, and there are a lot of people around. If the first brain in a box is simulated by a supercomputer that has only three others like it in existence, it's going to be much more difficult to scale it.
Of course, just adding another person doesn't necessarily make things go faster (mythical man-month and all that). But then again, just adding more hardware isn't going to necessarily make a brain in a box more powerful, either.
With AI you can optimize or change the underlying algorithms, or allocate more computing resources as necessary. You can't do that with humans—at least not in any literal sense, nor in any way that is remotely as efficient or effective.
The very nature of human intelligence is such that its architecture, scale and speed of computation are all bound within a relatively narrow range.
Humans don't scale. If you try to add more of them, you get things like governments and markets, i.e. lots of people who can't agree on shit. Not very efficient. Scaling a properly designed AI, on the other hand, could lead to something much more powerful - cognitive power of great many humans with a clear focus of an individual.
Human minds in meat don't scale. But upload, and make a bunch of copies, introducing more-or-less random variation. Run in parallel, with suitable aggregation and selection. Got that from Quantum Thief by Hannu Rajaniemi.
I'm not sure why you think humans don't scale. Do you think that the research done by the Manhattan Project was no more impressive than research conducted by a single individual? Or that a single individual is going to have as much success gaining political power as a political party?
Now as I said, "adding another person doesn't necessarily make things go faster." It can, but not always. But what evidence do we have that a human level AI would be able to scale easily? That seems to be a pretty huge assumption that gets glossed over.
>But what evidence do we have that a human level AI would be able to scale easily?
The fact it would probably be software, executing on a digital platform. These things typically lend themselves to optimization and scaling.
Seems to me that the chance we'll have just enough computing power to run human-level AI—without sufficient resources to go beyond that—is a small one. It is far more likely that we will either have too little, or more than ample computing power available.
What evidence do you have that human-level AI won't scale easily?
The best argument I can think of in your favor would be prohibitive computational complexity requirements, but the aforementioned dynamic still applies—it's unlikely that we'd have only just enough computing power to achieve human-level and not go beyond that.
We can look at efforts to stimulate the brain, which do not seem to scale easy. As I wrote in another comment, the K computer was able to simulate a seconds worth of brain activity. It doesn't seem to have scaled easily. I don't have experience with supercomputers, but I have a feeling that things are a bit more complicated than "just network it together with another supercomputer, and then everything will be twice as fast."
And that's just talking about a raw increase in computing power. We don't have any idea whether a human level intelligence program would even be able to put increased processing power to good use (and it could theoretically be detrimental; look at what happens to humans who have too many neuronal connections).
You shouldn't consider government inefficiency a flaw. That's by design. If you have a government or a brain that makes decisions quickly, you might not have time to backpedal on seemingly good ideas that turn out bad.
This analogy would be accurate if humans had brain-to-brain interfaces. AI's that collaborate will be able to do so much faster than humans with their slow politicking. But in before that, an AI doesn't even need to collaborate with other entities, as it can expand its computing resources at will.
I was thinking more in terms of cost. Adding a person to your cause requires a lot of money (the cost of them devoting their entire working time), and you only get so much extra benefit (efficiency losses due to communication and their own incentives). And the more effective the human, the more he would cost.
With an AI, you just dump that money directly into the brain by slapping on more compute power and storage. It'd be a direct increase in ability to its will. I'm not sure what exactly the raw costs would be (maybe it would still cost more than hiring a human), but at least it seems like it'd be a much more efficient use of each dollar.
The supercomputer that just simulated one second's worth of brain activity (K computer) cost $1.25 billion to design and build and $10 million a year to operate. Now, I don't have experience with supercomputers, but I doubt that you'd just be able to slap in some extra ram and get a big speed up.
It seems likely that the first simulated human level intelligences, at whatever point in the future they pop up, will at first be extremely costly and extremely slow. We don't really know if, say, connecting two supercomputers two simulate twice as many virtual neurons would make it more effective or if it would make it too unstable to do anything (how would a human brain with twice as many neurons react? It might just be mentally unstable and unproductive). But even a simple doubling experiment like that would probably take a long time to plan out and realize.
It isn't AI specifically. Any intelligence capable of taking over or shutting down a wide range of systems, or even just a few critical ones, needs to be closely monitored if there is more than a small chance of it causing massive harm.
Now, if that intelligence also far exceeds our own in certain aspects but is not empathetic enough, and if it were to have a keen understanding of how to control humans, and if it were much more able to quickly control and adapt software and hardware, then it would become a much bigger risk.
But, you're right- in theory, there could be a natural intelligence that could be the same level of risk as an artificial one. Most of us, including myself, are just not aware of one.
> Any intelligence capable of taking over or shutting down a wide range of systems,
That's a thing I don't care for about most science-fiction. The AI entity can just magically take over every technology it's connected to, because magic. No concern about the computational complexity of breaking the code-signing certificates on the affected computers. It's like everything on the Internet of Things is hopelessly and completely insecure!
... okay now that I type that out, maybe that's more realistic than I gave it credit for after all ... :b
Seems to me that some isolated artificially intelligent computer sitting on a deserted island somewhere isn't going to cause much trouble. Rather, it's the combination of a totally connected electronic world with an AI entity (and the notion that the AI entity would kinda sorta be a "native" of the interconnected electronic world) that could cause a lot of trouble.
Of course there are other reasons why having everything be "connected to the internet" (light switches, refrigerators, nuclear warheads) is probably not the wisest idea... would an AI with bad intentions really be any more harmful then a human being with bad intentions who had access to every interconnected thing?
At least at first, the most powerful entities will be enhanced humans aka transhumans aka human-AI hybrids. Indeed, we're already there, in the sense that we rely on machines so heavily, and in so many ways. Interfaces will improve, with greater integration, of course.
I've been reading Hannu Rajaniemi lately. He has in some ways a trippier take on this stuff than even Peter Watts does.
"the universe appears lifeless not because complex life is unusual but, rather, because it is always somehow thwarted before it becomes advanced enough to colonize space."
"Before the prospect of an intelligence explosion, we humans are like small children playing with a bomb."
A bomb is dumb, its explosion just raw chemistry. I would argue we're like small children playing with a sleeping parent: when it finally wakes up, it's going to have its own agenda for us.
You're going to build an amoral, powerful computer and it's going to do what? Slap the fork out of your hand every time you try to eat something?
We build machines using a highly human-dependent artificial selection process. That's not going to change because the number one feature being selected for is usefulness to humans. I'm confused about how that's going to result in an even modestly-powerful, fully integrated machine that is not demonstrably useful to us.
That's a bit like saying we should reconsider having sex because our babies might evolve grenades for hands, and that would be suicidal.
Humans are dumb too, but still manage to explode regularly - although not usually with intelligence.
I'm completely failing to understand why a hypothetical god-AI is being taken seriously as a threat to survival, but the many potential and actual horrors of the human world are being ignored here.
We're perfectly capable of destroying ourselves without needing to invent an AI to do it for us. Why is this not taken seriously as a problem?
I'm not really well read on this stuff, but do most people think that deep brain computer interfaces are farther off than self-improving artificial general intelligence?
I usually see arguments about harmful AGI based on the idea that AGI will be able to optimize itself and acquire computational resources to become powerful in a scary way.
I just sort of think that the human brain is already a tremendously powerful computational resource and augmenting it with weaker AI is going to be easier, cheaper, and ultimately, much powerful than AGI. But, of course that depends on the development of really powerful BCIs.
Why not develop AI systems that complement the already insanely sophisticated computational system of the human brain instead of trying to replace it? Or enable groups of humans to work together along with AI to solve really hard problems?
People often seem to think of AI as an arms race, but is it possible that systems that leverage human intelligence may have an insurmountable advantage?
I agree. Any system with potential for strong AI will be more immediately useful for augmentation. Although it's dated, I recommend Disappearing Through the Skylight O. B. Hardison, Jr.
47 comments
[ 4.7 ms ] story [ 124 ms ] threadAs the gentleman says, a lack of Autonomous Agents is not the thing standing between humanity and oblivion. The conspicuous surplus of super empowered agents (as in "I could leave this room and 80 million people would be dead 40 minutes later") is.
As for the Royal Society stuff. For the record it's not learning, and it's not the machines, I can say it again if anyone bothers to listen. Although given that it's not what will get news print or sell books or get on telly I am pretty sure that no one will.
Ho hum.
(edited 'cos I forgot a bracket, which was why I gave up c)
We can do both.
Lives saved versus dollars spent is a better metric. Having said that, the Singularity crowd tends to lump in all potential future lives into the saved category. The question of allocating resources to avoid existential threats very quickly turns into a Pascal's mugging in favor of AI safety (think of the trillions of future children).
> The threat of nuclear war is well known ... until very recently the same could not be said for the threat of unfriendly AI.
Enough nuclear weapons exist to kill all humans forever. Strong AI does not exist. Expert estimates range from decades to over a century. At the same time, the people making the most noise seem come from the side of avoiding existential threats and not AI research. I am (highly) skeptical they will produce anything of value. Let me rephrase. Would you expect any productive work in 1840 by philosophers concerning potential dooms day weapons yet to be created by physicists?
It has been suggested that humans recovered from a population bottleneck of less than 30,000 individuals (the Toba catastrophy). A nuclear war would leave billions alive.
Let's hope that the increased notice taken of AI, and fear thereof, doesn't suddenly shut down a major source of hope for the future.
Poking around a little, I also find this essay, which I think does a better job of getting to the root of the issue: http://edge.org/conversation/one-half-a-manifesto
"Bostrom introduced the philosophical concept of 'existential risk' in 2002."
EDIT: I am unable to find the source. Might have been a halluzination, but I thought I remembered them having a discussion where Eliezer more or less still thought "let the AGI win, humanity be damned". Ah, my brain and I are not in a good relationship.
Oh, come on. This has been a theme in science fiction back to at least the 1940s. Jack Williamson's "With Folded Hands" (1947) is probably the clearest early writing on the subject.
"The car turned off the shining avenue, taking him back to the quiet splendor of his prison. His futile hands clenched and relaxed again, folded on his knees. There was nothing left to do."
That's a very funny short story. A magic story, though, not AI.
https://en.wikipedia.org/wiki/R.U.R.
And there is the Golem, which is even earlier.
Also i find it funny how much attention this theoretical problem receives, when we don't have a good explanations on what intelligence is and how it works.
I think that the Robot will not have such an in built desire - if it is based on reasoning then the machine might actually be more reasonable. In other words there is enough room for both Humans and Robots.
The point is that our gadgets become more powerful and more potent as weapons; so as we have a good history of past abuses it is sort of easy to extrapolate to the future.
What is still don't understand: would intelligence be an inhibiting factor - like more reasonable Humans are supposedly less destructive; Maybe the same goes for machines.
I don't buy the argument about competition for resources - With enough effort you can always stretch it so that there is enough for everybody/everything.
Another interesting aspect: once upon a time people would become very agitated when discussing politics (that was when we still had ideological differences and when people thought that their stance does matter); In our time we have discussions about sci-fi instead of that.
"The first registered thoughts on the possibility of artificial, machine intelligence, becoming a risk to humanity stems from the late Industrial Revolution. It was Samuel Butler, in 1863, in his Darwin Among the Machines that suggested machines could eventually replace humans as the dominant agents on Earth"
https://wiki.lesswrong.com/wiki/History_of_AI_risk_thought
[1] https://books.google.com/ngrams/graph?content=existential+ri...
https://news.ycombinator.com/newsfaq.html
Btw, we're working on a better approach to duplicate detection that will reduce the number of reposts and will more often give credit to the original submitter.
Of course, just adding another person doesn't necessarily make things go faster (mythical man-month and all that). But then again, just adding more hardware isn't going to necessarily make a brain in a box more powerful, either.
The very nature of human intelligence is such that its architecture, scale and speed of computation are all bound within a relatively narrow range.
Now as I said, "adding another person doesn't necessarily make things go faster." It can, but not always. But what evidence do we have that a human level AI would be able to scale easily? That seems to be a pretty huge assumption that gets glossed over.
The fact it would probably be software, executing on a digital platform. These things typically lend themselves to optimization and scaling.
Seems to me that the chance we'll have just enough computing power to run human-level AI—without sufficient resources to go beyond that—is a small one. It is far more likely that we will either have too little, or more than ample computing power available.
What evidence do you have that human-level AI won't scale easily?
The best argument I can think of in your favor would be prohibitive computational complexity requirements, but the aforementioned dynamic still applies—it's unlikely that we'd have only just enough computing power to achieve human-level and not go beyond that.
And that's just talking about a raw increase in computing power. We don't have any idea whether a human level intelligence program would even be able to put increased processing power to good use (and it could theoretically be detrimental; look at what happens to humans who have too many neuronal connections).
Simulating the brain isn't necessarily the path to human-level AGI though. Simulating biology in digital form tends to be inefficient.
Modern machine learning scales easily, or at least relatively easy.
>(and it could theoretically be detrimental; look at what happens to humans who have too many neuronal connections)
Perhaps a constraint of our brain's cognitive architecture and not intrinsic to cognition itself. Still, interesting point.
With an AI, you just dump that money directly into the brain by slapping on more compute power and storage. It'd be a direct increase in ability to its will. I'm not sure what exactly the raw costs would be (maybe it would still cost more than hiring a human), but at least it seems like it'd be a much more efficient use of each dollar.
It seems likely that the first simulated human level intelligences, at whatever point in the future they pop up, will at first be extremely costly and extremely slow. We don't really know if, say, connecting two supercomputers two simulate twice as many virtual neurons would make it more effective or if it would make it too unstable to do anything (how would a human brain with twice as many neurons react? It might just be mentally unstable and unproductive). But even a simple doubling experiment like that would probably take a long time to plan out and realize.
Now, if that intelligence also far exceeds our own in certain aspects but is not empathetic enough, and if it were to have a keen understanding of how to control humans, and if it were much more able to quickly control and adapt software and hardware, then it would become a much bigger risk.
But, you're right- in theory, there could be a natural intelligence that could be the same level of risk as an artificial one. Most of us, including myself, are just not aware of one.
That's a thing I don't care for about most science-fiction. The AI entity can just magically take over every technology it's connected to, because magic. No concern about the computational complexity of breaking the code-signing certificates on the affected computers. It's like everything on the Internet of Things is hopelessly and completely insecure!
... okay now that I type that out, maybe that's more realistic than I gave it credit for after all ... :b
Of course there are other reasons why having everything be "connected to the internet" (light switches, refrigerators, nuclear warheads) is probably not the wisest idea... would an AI with bad intentions really be any more harmful then a human being with bad intentions who had access to every interconnected thing?
I've been reading Hannu Rajaniemi lately. He has in some ways a trippier take on this stuff than even Peter Watts does.
The universe appears lifeless because it is a Dark Forest. http://www.goodreads.com/book/show/23168817-the-dark-forest
A bomb is dumb, its explosion just raw chemistry. I would argue we're like small children playing with a sleeping parent: when it finally wakes up, it's going to have its own agenda for us.
If we're playing this game, then humanity is more like a small child trying to wake up Cthulhu in hopes that he'll play nice.
We build machines using a highly human-dependent artificial selection process. That's not going to change because the number one feature being selected for is usefulness to humans. I'm confused about how that's going to result in an even modestly-powerful, fully integrated machine that is not demonstrably useful to us.
That's a bit like saying we should reconsider having sex because our babies might evolve grenades for hands, and that would be suicidal.
I'm completely failing to understand why a hypothetical god-AI is being taken seriously as a threat to survival, but the many potential and actual horrors of the human world are being ignored here.
We're perfectly capable of destroying ourselves without needing to invent an AI to do it for us. Why is this not taken seriously as a problem?
I usually see arguments about harmful AGI based on the idea that AGI will be able to optimize itself and acquire computational resources to become powerful in a scary way.
I just sort of think that the human brain is already a tremendously powerful computational resource and augmenting it with weaker AI is going to be easier, cheaper, and ultimately, much powerful than AGI. But, of course that depends on the development of really powerful BCIs.
Why not develop AI systems that complement the already insanely sophisticated computational system of the human brain instead of trying to replace it? Or enable groups of humans to work together along with AI to solve really hard problems?
People often seem to think of AI as an arms race, but is it possible that systems that leverage human intelligence may have an insurmountable advantage?