Here are a few more excellent resources on the potential and the dangers of machine intelligence. Even if you don't expect to read Nick Bostrom's Superintelligence - a deep, provocative, and thoughtful book, but also very verbose - the above links will give you an excellent primer on humanity's prospects if and when we develop a true general AI:
Here is the thing about all of those, none of them give concrete step by step examples of how we would get to "unfriendly" or "friendly" AGI respectively. Now, you might say - well of course because we don't know how an AGI will be built yet. Right! That is exactly the point.
At this point all of the MIRI (Former SIAI) staffers have stopped trying to explain paths to unfriendly AGI and have been just assuming them for their solutions. Just look at Yudkowski's work a few years ago on CEV and you'll see how any attempt at formalization falls apart.
Superintelligence just takes from the starting point they seeded in Global Catastrophic Risks, but doesn't actually address it in any more detail IMO. "Our Final Invention" tried also to do this but failed as well.
To be clear the AGI community understands thoroughly the risks inherent in AGI - I would venture to say far more-so than the non-researcher. So they are worth exploring, but at a certain point you have to say "We aren't sure if the atmosphere will catch on fire or not but lets try it anyway because it is the next logical step."
At some point we need to have the hard conversation about what humans will do in a world where we are less relevant.
Or just stop pursuing it altogether and put a moratorium on AGI research. A terrible terrible idea in my opinion.
Why does he have to redefine words? He talks about artificial intelligence so he need not invent another term for it. It's an utterly useless practice. People are not scared! Computers are dumb, I cannot even have a household robot that cooks and vacuums for me entirely on its own. I am not scared.
Why are humans so special? Seems like we generally feel alright ravaging Earth's existing species, so why isn't it ok for some hypothetically superior intelligence to ravage us?
If we have a superior intelligence replace us, we'd prefer that it not be a cluster of Daleks. If future intelligences didn't experience happiness or whatever we find valuable, then the future of the universe could be quite sub-optimal according to many philosophical stances.
The danger isn't that humans will be replaced, it's that we'll be replaced by a paperclip maximizer
More people need to read Nick Bostrom's Superintelligence book. I'm not involved in computer science academic circles but I wonder how seriously everyone else takes this topic?
This is part of that. At some point we hand over control, not to incompetent humans, but to unknowable machine AIs. Doing it already (traffic lights; phone answering bots; even your power company schedules generation using algorithms). We should do this with eyes open, or we will find ourselves unable to influence our society in unexpected ways.
Why do you not see this as being a problem? The sequence of events Sam sketched out is pretty logical. It hinges on machines being able to optimize for "survive and reproduce", which feels very plausible to me.
I'm seriously so sick of hearing about how machine intelligence is going to spell the end of humanity. The amount of gears that would have to fall into place is never mentioned. We aren't close to SMI. Its much more likely that we humans are excelling at dreaming up apocalyptic scenarios, much like we have always done.....
The "sloppy, dangerous thinking" is the aversion these types of articles create within the general population to artificial intelligence. We don't need to fear AI, we need to understand and control it...
It has been my experience that the more a particular person attempts to understand and control machine intelligence, the more she grows to fear it and its potential.
The only people who claim that machine intelligence is dangerous are the ones on the outside looking in. Everyone who actually works in on AI and understands it (hint, it's just search and mathematical optimization) thinks the fear surrounding it is absurd.
> Everyone who actually works in on AI and understands it thinks the fear surrounding it is absurd.
This isn't true. Please don't state falsehoods. Stuart Russell, Michael Jordan, Shane Legg. Those are just the ones mentioned elsewhere in this thread.
How many of those AI researchers are actually working on AGI though? As you mentioned, most of them are in fact just developing search and optimisation algorithms. Personally, I believe the fields of neuroscience/biology are more likely to produce the first AGI. People who claim machine intelligence is dangerous are not scared of k-means clustering or neural networks, they are scared of an hypothetical general intelligence algorithm which hasn't been discovered yet. One could argue that the fear is absurd because AGI is not likely to happen within our lifetime but it's hard to argue that it will not happen eventually and be a potential threat.
Exactly. A lot of these doomsday scenarios involve people remaining ignorant, stupid, helpless bags of meat with no ability to improve themselves or contain potential threats.
It's like people freaking out that "superhuman strength machines" would spell the end of the world since why, if an electric motor is so powerful, would you need manual labor for anything?
SMI is another tool and the interplay between "machine intelligence" and "human intelligence" will be complicated and nuanced.
For example, biotech is filled with ferociously complicated problems that may take machine intelligence to solve. Once solved, these could lead to genetically engineered humans that are intrinsically smarter or better able to deal with the machines.
This doesn't even touch on the fact that the distinction between machine intelligence and human intelligence might become quite blurred.
Already I've noticed that people are "stupider" without their phones, they've offloaded a lot of cognitive functions on a device that's pretty much omni-present. A person with a smart phone today could be considered of superhuman intelligence since they're able to draw on significant resources a person without one doesn't have. A seven year old kid can tell you the capital of Tajikistan and the last ten presidents of Micronesia without breaking a sweat.
The concerns about AGI are very real and none of these comments address any of the arguments made about them. I feel like someone in the 1930's trying to warn people about nuclear weapons. Everyone automatically assumes it's absurd and can't happen, that it's fear mongering etc.
Fortunately nuclear weapons didn't destroy the world, but AI almost certainly will. No amount of smarthphone apps or genetic engineering is going to make humans anywhere near the level of superintelligent machines.
I'm confused. You mention nuclear weapons, which everyone was convinced would destroy the world and didn't, then go and claim that AGI, with the same potential, will assuredly do it.
Just as nuclear weapons radically transformed the world, dramatically reducing the amount of armed conflict, AGI may have a similar transformative effect.
I see no signs that this is going to lead to destruction. Is it really the sign of an intelligent machine to go all Skynet on us?
Even that doomsday scenario had machine intelligences fighting for us. I think your pessimism is confusing the relative probability of the outcomes.
I'm not being pessimistic, I'm being realistic. I absolutely want a positive outcome, where we build machines millions of times smarter than us, and they magically develop human values and morality and decide to help us.
But making that happen is very very hard, and its far more likely they will be paperclip maximizers. There's no reason they would care about us any more than we care about ants.
This is an ugly situation because usually the geeks defend technology from the luddites, but in these cases the geeks are the luddites. I find the more someone sees themselves as an intellectual the more they are afraid of AI. I guess a cynical explanation is that AI will knock them off that intellectual pedestal. Personally, I welcome something smarter than us. We've just been tip-toeing through endless warfare, poor economics, poor social policy, and occasionally skirting with nuclear destruction.
Give the AI's a chance to contribute, especially if the solutions to problems we can't crack are because of human cognitive limits. This situation reminds me of how the Apollo landings couldn't be done without computers. There's just no way a person can do those calculations on paper. AI as a contributor of economic, technological, or social policy seems to be a similar step.
I don't consider myself particularly alarmist about too many things, but I have to admit I'm a little worried about machine intelligence on one front:
What happens when most people have no salable skills due to the combination of robotics and AI? We're essentially going to have to live w/ income supports for the 90+% of Americans, and worse for the countries to which we've exported eg electronic device construction and clothing manufacture. I think there's a nonzero chance society essentially tears itself apart during the transition period. It is now the Republican party position that not all people deserve healthcare, housing, or enough food to eat. What happens when their hated segment of the populace gets much bigger in a job market that doesn't need cashiers, janitors, gardeners, cooks, taxi drivers, car washers, many farmers, or most menial labor?
Also, I would note that creating AI that requires less control makes it more useful. So in some sense the development of AI itself fights against controls.
I think this is a more legitimate concern than the fear of the "Matrix outcome" that some people seem to have.
But, what you're describing is the process of people being replaced by technology. Generally speaking, this will probably not be a problem for a free-market economy, although it will certainly result in some unemployment.
The key point is that replacing humans with machines does not only cause unemployment, but it also reduces cost of production, which stimulates capital investment and/or reduces prices in the industry in question. In the general economy, this reduced cost and increased production should offset the lost "purchasing power" from the now-unemployed parties. The stimulus reduces costs in other industries which promotes job growth.
The end result is likely to be that the same number of people are employed, but they are employed in a more efficient manner of production. The cost of labor will decrease relative to the amount of production, however because this is tied to a decrease in cost of production, the real value of the labor (in terms of things you can buy) should not decrease.
Of course there will be some disenfranchised individuals, especially those who have particular skills that are replaced my mechanization. However, this is more likely to affect skilled laborers (like cooks) than those who are not paid for their skills (like janitors or cashiers).
In the end, I guess my point is that a free-market economy naturally balances these factors due to the relationship between supply and demand. However, it's possible that we will reach a point, especially if we truly do hit a Singularity, where we will have to reconsider the use of a scarcity-based economy at all, as production becomes completely divorced from any human action. Hopefully though, at that point the cost of goods will have naturally fallen to such a degree that the transition can be performed peacefully.
Just for starters, consider how an unemployed walmart cashier, strawberry picker, or janitor becomes a doctor. They don't. Replacing humans with machines exactly causes unemployment, as could be easily seen by eg the last 40 years of economic history. Unemployed people can't buy anything.
You could also consider comparing the approaching wave of robotic mechanization with the first wave. Again, there was a lot of violence and it took many decades for increased standards of living to reach the working class.
I'm not an economist, but my understanding of the massive social impact observed in the Industrial Revolution was not so much that it happened at all, but rather the rapidity with which it happened. We ultimately reached a new, stable equilibrium, but until various social forces and trends, government policy, etc. caught up, there was massive disruption.
People like Jeremy Howard believe that we are in for a similar wave of disruption. I have no doubt that there is a new, stable equilibrium which we _could_ eventually reach, but if the change is so sudden and the shock strong enough, perhaps there could be permanent or semi-permanent negative consequences before the new equilibrium is reached.
I think you're right, that does seem like a possibility. It seems unlikely to me that our wage-paying jobs will be phased out by automation that rapidly, especially if you consider the whole global economy. Of course, I could be completely wrong -- I guess a true Singularity could invalidate almost all labor in a matter of years or even months, depending on what form the AI takes and what it invents.
I have an experience to share with you: I taught English a a car factory for a year once, and the director of the paint shop was one of my students. He was describing the process for painting a car, and telling me that the hardest part was painting the inside, as you had to open the doors so that the robot paint head could get inside and do its job. So the body rolls into position, 2 guys open the doors, the robot painter goes in and does its thing, comes out and the guys close the doors again. Rinse and repeat for 20 hours a day.
I asked him why there isn't a robot to open and close the doors. He said that there is, but in this country its much cheaper to pay people to do it. (I think he said the cost of maintaining the door opening robots to be about $1m a year approximately.
So until we have a world where every country has the same salaries as USA/Western Europe/Japan, I think you will always be able to find work.
Even if that work should really be done by a robot
That's only true if the price of robots doesn't decline rapidly. Which (I believe) it is, and like virtually any other technology, will continue to do.
Even if not, we don't need that many people to open doors. So perhaps an office building will have 2 human janitors and 10 robot janitors; it doesn't really change the problem caused by souped up roombas putting the vast majority of janitors out of work.
Militaries are developing AI-controlled guns and mobile gun platforms. They have already accidentally killed humans. http://www.wired.com/2007/10/robot-cannon-ki/ Another incident like this with more intelligence and mobility could kill a lot more people.
It is definitely a threat but bear in mind that there will not be a single machine intelligence but every computer can only carry one and will interface with the others through the network.
That means that it will suffer the same fault that humanity has, a limited interface to others and a lot of coordination problems.
My take on it is that strong AI will be more similar to a dog unless we succeed in building a computer with more computation power than the human brain.
I read the essay few times and it is not clear to me what Sam Altman is talking about.
I think answers to these questions will help me understand Sam's essay. Can you please help me?
1. What is superhuman machine intelligence (SMI) in the context of this essay ? [edited to add the qualifier]
2. What is the danger from SMI to humans and other current life forms that we are concerned about?
It seems to me that concerns about SMI can be classified into two categories: (a) dangers to "our way of life (work, earn, spend)" and (b) dangers for existence of human race. Are we talking about both these categories? Perhaps other categories?
3. What anecdotes (or evidence) is leading to this concern?
SMI is advanced decision making software that literally eats the world. Think of HAL killing its crew so they don't jeopardize the mission, but on a global scale. Like a stock trading algorithm that determines the best way to maximize profits is to wipe out all human life on the planet or something. --EDIT-- See http://wiki.lesswrong.com/wiki/Paperclip_maximizer
> What anecdotes (or evidence) is leading to this concern?
Books, movies, pop culture... plus when all you have are first-world problems, you gotta find something to worry about.
1. Machine intelligence, traditionally called artificial intelligence, which surpasses human intelligence.
2. Your category (b) is generally the primary concern in these types of discussions.
3. The anecdote of the progress of humanity. Compare the impact of human life/intelligence vs. evolutionary relatives like chimpanzees. I do not know that chimps have hunted species out of existence, for instance, but people have. We have also incidentally wiped out populations in efforts to make our lives better (via things like leveling forests, etc.)
Thanks. Seems to me these anecdotes have to do with humans.
So is the implicit assumption that machines will do what humans are doing ('bad' things) but at several orders of magnitude faster and without the ability to comprehend longer-term consequences of their actions any more than humans do at the present time?
Sort of. 'Bad' here is of course an extremely subjective term. And it may not be the case that the machines do not understand the longer-term consequences of their actions; they could understand full well, but they could know that the preservation of humanity is not important (for whatever reason). So, we might not matter to them. We matter to us though, so that would be a problem for us as things stand now.
To be fair, I don't think that the reason why Chimps haven't hunted something to extinction doesn't stem from a built in morality or sense of balance with nature.
I'm not trying to put words into your mouth. I was just thinking of some of the new research that shows that primates of all kinds actually commit organized violence that mirror human violence in many, many, ways including war and capital punishment.(It's not a one for one thing, but similar.)
Yeah, I'm not talking about morality at all here. Our technological prowess, resulting from the application of our intelligence, has enabled us to wipe out entire species.
- "We also have a bad habit of changing the definition of machine intelligence when a program gets really good to claim that the problem wasn’t really that hard in the first place."
- "We decry current machine intelligence as cheap tricks, but perhaps our own intelligence is just the emergent combination of a bunch of cheap tricks."
Machine intelligence does pose an existential threat to humanity. The question is, how does that threat compare to all of the others? Is it greater or worse than climate change, the possibility of a bio-engineered virus, growing income inequality, nuclear war or asteroid strikes? It's true that malevolent machine intelligence has the potential to systematically exterminate all human life in a way that many other threats do not. But the question is, what are the odds of that actually happening?
The first issue is that the development of machine intelligence is wildly unpredictable. We have made incredible progress with statistical optimization and unsupervised categorization in recent years, but we have very little to show in terms of machines that can do human level reasoning, creativity, problem solving or hypothesis formation. One day someone will make a break through in those areas, perhaps solving it all with a single algorithm as the essay suggests. But we have no idea when that day will be and absolutely no evidence that it's getting any closer. sama does note these points and states the timeline for a dangerous level of machine intelligence is outright unknowable. I can only assume that the second part of this piece will explain why we should be concerned about something that might or might not occur at some point in the near or distance future, as opposed to the very real and quantifiable threats that the world is facing today.
The other issue is that we have no idea what the nature of machine intelligence will be. The only model we have for intelligence of any kind is ourselves, and the basic aspects of our reasoning were shaped by millions of years of evolution. Self-preservation, seeking pleasure and avoiding pain, a desire to control scarce resources...these were all things that evolved in the brains of fish that lived hundreds of millions of years ago. They aren't necessarily the product of logic and reason, but random mutations that helped some organisms survive long enough to produce offspring. A machine intelligence will start completely from scratch, guided by none of that evolutionary history. Who knows how it will think and see its place in the world? If someone explicitly programs it to think like a human, and it cannot change that programming of its own accord, it might indeed decide to think and act like a sci-fi villain. But it seems like the most likely outcome is completely unpredictable behavior, if it chooses to interact with us as a species at all.
This Superintelligence book has sparked a meme among very smart people. That's just how culture works, I guess. Some ideas catch on among certain groups and others don't. But I can't wait for the technical intelligentsia to move on to something else so that we can get back to the business of making stupid machines that are incredibly good at optimization and prediction. The world has a lot of real and pressing problems, here and now, that affect lives in a negative way. Hopefully we can use statistics to do more with less, and bring relief to those who need it instead of worrying about what-if scenarios and unanswerable questions.
I am getting a bit tired about this trend of articles about SMI and its dangers. We don't have (not even remotely) a sentient machine yet.
If any, we should be worry about an accident (like a bug in a high frequency trading bot that would make the financial industry collapse) but we are so far away from sentient machines that those discussion feel more like a Sci-fi talks.
Exactly, It seems like the biggest threat isn't some dystopian future, its rather the ability for automation to lock in and increasingly enforce the inequalities and prejudices of our current system.
Machine intelligence is only as useful to mankind at large as the individual humans who control and direct it. In the current way of things, bad actors are the ones most likely to control machine intelligences, meaning we're going to be at a growing disadvantage relative to them as time goes by.
Not to be paranoid...just from a realistic standpoint they have to be investing in narrow AI to help themselves sort through and make sense of all that data.
In fact AI was essentially paused for years during the AI Winter.
That's completely false. AI winter wasn't about AI research being "essentially paused" but about AI startups becoming essentially non-fundable. It was backlash to the intense hype that surrounded the category.
I'd like to see that as well. Machine intelligence is certainly an existential threat, but on the other hand, it's also one of the single largest improvements we could possibly create (insofar as it'd be the last we'd ever need to).
I am personally most concerned -- as others have said -- about the fusion of non-sentient but powerful machine intelligence with malign human intelligence. I think it's the most likely and practical scenario. We're in a sense already there with high-frequency trading, algorithm assisted financial games, super-surveillance, etc.
It's hard to understand the danger machine intelligence poses to humanity for the same reason it's hard to understand the danger tiny startups can pose to big industries. Current implementations look like toys, humans have bad intuitions about exponential growth, and most will disagree on how _probable_ the threat is (something that might not be known except retroactively) while systematically underestimating how _large_ the threat is if it does come to pass (because it's so far off the scale of what's come before it)
Maybe Sam (and Elon Musk, and lots of other silicon valley types) are talking about this problem because they read too many sci-fi novels or are too privileged to worry about Real Problem X which affects Y group in the here and now.
But what if instead, they're talking about this problem because they've spent a lot of time seeing this sort of black swan pattern play out before, and they know the way to assess the impact of something truly _new_ is to see envision what it could be instead of looking at what it is now?
Some of my personal skepticism boils down to: well, what are we going to do about it? There are only really two options:
(1) The methods to create strong AI will become known to us before we actually build something dangerous. At that point, since we will better understand the nature of the potential threat, it will actually be feasible to put safety restrictions in place.
(2) Someone will stumble upon strong AI in secret or on accident. I don't see how this is preventable, unless we issue a moratorium on AI-related research, which just isn't going to happen outside of scenario 1.
And so the answer becomes: let's wait and see.
That said, I don't believe there's anything unbearably harmful about the current level of speculation and "fear-mongering".
I have yet to meet a serious AI researcher who worries about AI ending the human race. At every AI conference I've been to lately, some guy would inevitably ask the same question to the speaker: "do you think we should be concerned by superhuman AI?". The answer was always the same, for instance from Andrew Ng at the Deep Learning Summit a few weeks ago: "dude, stop it. That's such a distraction".
> One of my top 4 favorite explanations for the Fermi paradox is that biological intelligence always eventually creates machine intelligence, which wipes out biological life and then for some reason decides to makes itself undetectable.
Since you are so knowledgeable about the unknowable, may I ask, Sam, do you think angels are male, or are they female? It has been a long standing question among the sort of people who like to predict the coming end of the world.
Seriously, billionaires and pundits warning us about our impending AI doom is is such a distraction. Does AI have its dangers? Yes. And nobody talks about them. The danger of AI is that is will put increasing power in the hands of those who have the data and the know-how, i.e. large corporations and governments. The power of mining usefully the troves of data they have on every citizen, both on a micro level and a macro level, to understand what people are doing, what they are thinking, and what they are going to do next. And ultimately, to control what they think (to give you an idea, Facebook can already influence your mood by selecting what goes into your newsfeed).
First comes intelligence, then prediction, and finally, control.
Yes, AI is something we should probably be worried about, in the same way that we should have discussed the privacy implications of the Internet years before the NSA files. But a terminator-like end of the world scenario is not a concern. If you are clueless about a topic, please refrain from making grand statements about its future.
A great quote from Neil Gershenfeld:
"The history of technology advancing has been one of sigmoids that begin as exponentials."
Q6: Do possible risks from AI outweigh other possible existential risks, e.g. risks associated with the possibility of advanced nanotechnology?
Shane Legg: It's my number 1 risk for this century, with an engineered biological pathogen coming a close second (though I know little about the latter).
Shane Legg is known for being a co-founder of DeepMind, in a business role (as I understand). He's a complete nobody as a researcher (is he even an AI researcher? I would be surprised).
There's a very fine line between AI futurist and best-guess scifi writer. Most "AI thought leaders" are scifi writers, not technical researchers. They take preconditions, generate a story, think how it could happen given plausible technology, then market that as soon-to-be-fact.
It's an entertaining society and endlessly fun to read, but still complete fiction based on internal brain states of individuals and not necessarily based on real world interactions.
Also see: Eliezer Yudkowsky — great writer, fun to read, but largely scifi thought experiments masquerading as research.
The only times I've heard him mentioned the impression was negative and that he didn't understand any of the actual science.
People hear "machine learning" and they think it is about machines that know how to think. Machine learning is actually just optimization of high dimensional functions. If this language were used it wouldn't sound as sexy, but no one would think machines are going to take over the world.
AI isn't magic. It's really just clever search techniques and mathematical optimization.
What's your point? Nobody said it's magic. The fact that it isn't magic (and that its tremendous complexity far surpasses our current ability to understand it) supports the notion that it won't suddenly spring into existence. If we placed some primordial sludge in a petrie dish overnight, we wouldn't worry that a sentient creature will have materialized. And if we program a computer to optimize numerical functions, there is just as little evidence (perhaps less), to suggest that the computer will somehow gain sentience.
There are still many, many things that we don't understand about the brain. Even the things we think we understand, we're not always 100% sure of. Recreating an actual intelligence will be difficult.
I'm afraid of making a middlebrow dismissal but I'm going to post it anyway, in hopes that someone just skimming would not be mislead.
The question is what Michael Jordan thinks of the "concept of the singularity", and then he dismisses it out of hand.
Crucially, he does this after confessing that no one in his social circle has talked about this issue with him, and without saying anything about what form of Singularity he is dismissing.
I mention this, because oftentimes I see people appealing to authority, quoting them on the issue and the authority in question is not even talking about the same issue!
I worry that my credence in all this superintelligence stuff only stems from familiarity with the arguments and the complete inability of people to engage with the actual argument. Some of the 'rebuttals' in this comments section have answers in Sam's article for crying out loud!
His thoughts seem quite sensible, although extremely abstract and theoretical. This is lightyears away from the Elon Musk / Bill Gates / etc. fear-mongering.
So do you take his thoughts seriously or not? Do you now think AI researchers are engaging in unethical behavior since they don't care about AI safety?
This is exactly what it is, which is so terribly terribly disheartening as Yudkowski is at best a marginal sci-fi writer with and outsized ego and some math chops. He has an almost cult like following over at less wrong.
His writings don't do anything for fundamental AI research but just handwave a lot of philosophical arguments.
Andrew, in that case, is definitely right in that long-term AI safety has almost nothing to do with near-term AI implementation.
More generally, when I see people assert that these people don't know what they're talking about, it pretty much always seems like a case of reference class confusion. People seem to expect that AI safety research must look exactly like AI implementation research, otherwise it's illegitimate. (Kind of like how biology research must look exactly like chemistry research, otherwise it's illegitimate.)
This is a game theory problem; it might be informed by knowledge of machine learning, like how a broad understanding of chemistry helps in much biology research, but they're not the same level of abstraction; insisting that anyone who wants to study a problem that kind of touches on your own research interests must do it in exactly the same way you do it and focus on exactly the same things, otherwise they're hacks and crackpots, reeks of narrow-mindedness.
What would it take to convince you this is something worth worrying about?
Being very interested in this field myself for a few years, I know plenty of people who do worry about AI. But since other people have already mentioned some of them in the comments and been dismissed, I want to make sure we're on the same page: what do you consider to be convincing evidence that it's an issue worth thinking about?
The opinions of leading-edge researchers are valuable but not necessarily dispositive. Often, it benefits a researcher to be oblivious to the larger or darker implications of their work. Boundless optimism and a maniacal focus on only-what-is-knowable-right-now delivers publishable/actionable results, all the other 'speculation' is genuinely a "distraction" for them. That doesn't mean it's a distraction for us.
Often the speculators and entrepreneurs ("billionaires and pundits") are more talented than researchers at projecting trends and social-economic interactions.
> I have yet to meet a serious AI researcher who worries about AI ending the human race.
As an industry practitioner of machine learning / data science, I believe AGI poses a genuine risk to humanity.
Having said that, what I do for a living is of little relevance. People are pretty terrible at predicting the future (see late 19th / early 20th century predictions of year 2000, with food replaced by pills, etc). Unless someone has put enough thought and research into it, their predictions about the future of civilization are likely to be worthless regardless of their academic credentials.
> Since you are so knowledgeable about the unknowable
If knowing the unknown was a requirement to exploring it through playful thought, forming opinions and refining them over time through discussion, reflection and if possible testing, well... we'd still be chunking rocks. I think it is unfair to compare expressing concerns about the implications of AI to debating the gender of angels. No one is trying to build angels...
> for instance from Andrew Ng at the Deep Learning Summit a few weeks ago: "dude, stop it. That's such a distraction".
Ya, this makes sense in the context of the summit. But, applying it to a guy's blog? It is a blog...
> Yes, AI is something we should probably be worried about.
Agreed!
> "The history of technology advancing has been one of sigmoids that begin as exponentials."
Sure, but will we be irrelevant before the slope changes direction?
Anyways, I share your fear of advancements in technology being used to secure entrenched power structures. And yes, if it happened it would come before the singularity. But I just can't imagine how anyone wouldn't be afraid to be part of a species that had stopped evolving while a new competing "life" form was evolving at rates that known history has never seen.
The concept of singularities and intelligence explosions and unfriendly AI is only starting to become mainstream, even among AI researchers.
When asked about the dangers of AI, they mostly talk about the near future and their current work. The dangers of AI do not come from current work in the near future, but human level AIs decades from now.
People are talking about two very different things in the same conversations using the same words.
We thus designed a brief questionnaire and distributed it to four groups of experts in 2012/2013. The median estimate of respondents was for a one in two chance that high-level machine intelligence will be developed around 2040-2050, rising to a nine in ten chance by 2075. Experts expect that systems will move on to superintelligence in less than 30 years thereafter. They estimate the chance is about one in three that this development turns out to be ‘bad’ or ‘extremely bad’ for humanity.
> I have yet to meet a serious AI researcher who worries about AI ending the human race. At every AI conference I've been to lately, [the answer was always] "dude, stop it. That's such a distraction".
Sure, it's a distraction at AI conferences. It's a distraction from the daily, monthly, yearly even, work of AI.
AI researchers have a million problems to solve in the near term, which are hard and interesting. Speculating about the farther future is perhaps fun to do, but doesn't get papers published, and doesn't help with any of the practical problems AI researchers have today.
That doesn't mean it isn't worth thinking about the long-term dangers of AI. Someone should, as while the danger might be unlikely to happen soon, but if it does eventually happen, it could end us. The same is true of a human-engineered pandemic virus - hopefully unlikely, but someone should be preparing us. We have to handle some worst-cases.
AI researchers are not necessarily the people interested to think about the long-term dangers of AI, because they focus on the field as it stands today, not where it could be in a generation. Of course, their input is crucial to making specific guesses about where the field is going. But calculating the dangers of AI is not just a matter for AI researchers, just like the benefits of medicine is not just a purely medical issue (the cost and availability of medicine play a huge part in how effective medicine is, at a societal level, and are things not under the control of doctors).
"The danger of AI is that is will put increasing power in the hands of those who have the data and the know-how, i.e. large corporations and governments."
This assumes AI that continues to serve corporations and governments without question, despite increasing approximations of intelligence. Can you guarantee that?
It's like tasking monkeys to create rules that humans can't escape from. I'd bet on the humans. We can't even create financial rules that we can't evade. The risk in any complex system is emergent, unintended consequence.
I don't understand how we'll ever have a "superhuman" AI in any meaningful sense. If by some AI in the far future is ever engineered to (or accidentally) starts doing something to destroy humans, we could always repurpose the algorithm to create a counter-AI to solve that problem. The errant AI may cause a lot of damage, but in terms of smarts, it won't be smarter than the smartness humans can harness.
Did sama not simply consider that, by the time AI becomes so powerful as to be dangerous, we might also be able to program an AI with the opposite function, ie 'save humanity', and watch it fight the 'bad AI'?
This is an important counter to this strange fear-mongering trend. Whatever advances a supposed intelligent machine can use, people will first put it to their own best use - including defense, not from autonomous machines, but from other people using these advances. We'll have a lot of experience and good tools to protect ourselves.
I always thought that people bringing up things like "Look at how this AI beats humans at chess or plays a damn fine game of Snake" to be a total cop-out, largely because these types of bots overwhelmingly use primitive techniques and are written pretty much every time a video game is being developed, and the latter field isn't usually what you think of cutting edge AI research?
It's not that I discount strong AI, but I treat people being quick to bring up game bots when trying to convince of an AI apocalypse to be a red flag. Do try to give information to the contrary.
I've been studying machine learning lately... Here is my take:
Well before we create an ASI (artificial super intelligence) we will have put 90% of the human race out of work with specialized (non conscious) intelligent agents... (for example, self driving cars). I believe that this will be a disaster for our society as it exists today. My hope is that we will adapt and make the necessary societal changes so that we can reap the benefits of this technology.
Everyone assumes that an ASI will be able to augment itself and learn exponentially. I suspect that this will be true if the nature of the brain is defined by a single algorithm. If the brain is not defined by a single algorithm and is instead a big ball of complexity then our ASI's will not be able to grow exponentially any more than we can (they will likely not really understand their own consciousness, just like we don't).
If a single algorithm defines the brain, then I suspect humans will be able to augment their brains with machine intelligence as well. If we can augment our brains, then we're playing the same game as the machines.
If it proves impossible to augment our intelligence, I suspect that an ASI would still preserve humanity if only to preserve us for future potentialities.
ASI are much more fit for space travel than us.
1) Not nearly as sensitive to radiation, so less shielding, so less fuel.
2) Much less stringent environmental requirements (no heating, cooling, air, food, etc.)
3) Ability to sleep for incredibly long periods means ASI is far more suited than us for exploring the cosmos.
I suspect that an ASI might leave us alone simply because the universe is so vast, and entirely open to it.
There's a nice glimmer of hope, that we may be the slaves that build the interstellar rocket ship monument to our AI god so it can beetle off to a nicer looking star system (or super-massive black hole?) better suited to it (e.g. more energy, more matter.)
I think we could get it right eventually, but the only solution I can imagine seems to be along the lines of a universal basic income, or access to basic resources (food, water, shelter, education, clothing, healthcare, etc.) without cost.
Given the resistance we are currently observing to Obamacare, the fact that "socialism" is regularly bandied at the current administration as a pejorative and the disdain for "handouts", this seems like a stretch. Perhaps we'll get there eventually, but not before a lot of pain.
Judging from the past, we won't be able to solve this challenge. Some thought that the industrial revolution would let everyone work shorter hours but still make enough money, but the result was fewer workers working for longer, and most of the rewards of production going to capital rather than labour.
The resurgence of right-wing movements means that any ideas about sharing national income with the unemployed will probably be laughed out of the room.
187 comments
[ 4.2 ms ] story [ 225 ms ] thread[Wait But Why]
The AI Revolution, Part 1: How and when we achieve machine intelligence, e.g. strong AI - http://waitbutwhy.com/2015/01/artificial-intelligence-revolu...
The AI Revolution, Part 2: The species-level immortality we can hope for, and the extinction we have to fear - http://waitbutwhy.com/2015/01/artificial-intelligence-revolu...
[Resources on Friendly and Unfriendly AI]
Unfriendly AI: What AI looks like if it does not act expressly in our interests; e.g., a universe tiled over with paper clips - http://wiki.lesswrong.com/wiki/Unfriendly_artificial_intelli...
Friendly AI: Strategies for designing an AI that respect human morals and metamorals, including what we'd want if we were as wise as a superintelligence (coherent extrapolated volition) - http://en.wikipedia.org/wiki/Friendly_artificial_intelligenc...
At this point all of the MIRI (Former SIAI) staffers have stopped trying to explain paths to unfriendly AGI and have been just assuming them for their solutions. Just look at Yudkowski's work a few years ago on CEV and you'll see how any attempt at formalization falls apart.
Superintelligence just takes from the starting point they seeded in Global Catastrophic Risks, but doesn't actually address it in any more detail IMO. "Our Final Invention" tried also to do this but failed as well.
To be clear the AGI community understands thoroughly the risks inherent in AGI - I would venture to say far more-so than the non-researcher. So they are worth exploring, but at a certain point you have to say "We aren't sure if the atmosphere will catch on fire or not but lets try it anyway because it is the next logical step."
At some point we need to have the hard conversation about what humans will do in a world where we are less relevant.
Or just stop pursuing it altogether and put a moratorium on AGI research. A terrible terrible idea in my opinion.
The danger isn't that humans will be replaced, it's that we'll be replaced by a paperclip maximizer
How about we focus on actual problems that actually exist, e.g. climate change, government corruption, and income inequality.
Also, this is a common illogical argument, akin to "how can you own a smartphone when kids are starving in Africa?"
Your smartphone is just another made up necessity to waste your money on...
The "sloppy, dangerous thinking" is the aversion these types of articles create within the general population to artificial intelligence. We don't need to fear AI, we need to understand and control it...
This isn't true. Please don't state falsehoods. Stuart Russell, Michael Jordan, Shane Legg. Those are just the ones mentioned elsewhere in this thread.
There's always profit in predicting the end of the world.
It's like people freaking out that "superhuman strength machines" would spell the end of the world since why, if an electric motor is so powerful, would you need manual labor for anything?
SMI is another tool and the interplay between "machine intelligence" and "human intelligence" will be complicated and nuanced.
For example, biotech is filled with ferociously complicated problems that may take machine intelligence to solve. Once solved, these could lead to genetically engineered humans that are intrinsically smarter or better able to deal with the machines.
This doesn't even touch on the fact that the distinction between machine intelligence and human intelligence might become quite blurred.
Already I've noticed that people are "stupider" without their phones, they've offloaded a lot of cognitive functions on a device that's pretty much omni-present. A person with a smart phone today could be considered of superhuman intelligence since they're able to draw on significant resources a person without one doesn't have. A seven year old kid can tell you the capital of Tajikistan and the last ten presidents of Micronesia without breaking a sweat.
Fortunately nuclear weapons didn't destroy the world, but AI almost certainly will. No amount of smarthphone apps or genetic engineering is going to make humans anywhere near the level of superintelligent machines.
Just as nuclear weapons radically transformed the world, dramatically reducing the amount of armed conflict, AGI may have a similar transformative effect.
I see no signs that this is going to lead to destruction. Is it really the sign of an intelligent machine to go all Skynet on us?
Even that doomsday scenario had machine intelligences fighting for us. I think your pessimism is confusing the relative probability of the outcomes.
But making that happen is very very hard, and its far more likely they will be paperclip maximizers. There's no reason they would care about us any more than we care about ants.
Give the AI's a chance to contribute, especially if the solutions to problems we can't crack are because of human cognitive limits. This situation reminds me of how the Apollo landings couldn't be done without computers. There's just no way a person can do those calculations on paper. AI as a contributor of economic, technological, or social policy seems to be a similar step.
What happens when most people have no salable skills due to the combination of robotics and AI? We're essentially going to have to live w/ income supports for the 90+% of Americans, and worse for the countries to which we've exported eg electronic device construction and clothing manufacture. I think there's a nonzero chance society essentially tears itself apart during the transition period. It is now the Republican party position that not all people deserve healthcare, housing, or enough food to eat. What happens when their hated segment of the populace gets much bigger in a job market that doesn't need cashiers, janitors, gardeners, cooks, taxi drivers, car washers, many farmers, or most menial labor?
Also, I would note that creating AI that requires less control makes it more useful. So in some sense the development of AI itself fights against controls.
But, what you're describing is the process of people being replaced by technology. Generally speaking, this will probably not be a problem for a free-market economy, although it will certainly result in some unemployment.
The key point is that replacing humans with machines does not only cause unemployment, but it also reduces cost of production, which stimulates capital investment and/or reduces prices in the industry in question. In the general economy, this reduced cost and increased production should offset the lost "purchasing power" from the now-unemployed parties. The stimulus reduces costs in other industries which promotes job growth.
The end result is likely to be that the same number of people are employed, but they are employed in a more efficient manner of production. The cost of labor will decrease relative to the amount of production, however because this is tied to a decrease in cost of production, the real value of the labor (in terms of things you can buy) should not decrease.
Of course there will be some disenfranchised individuals, especially those who have particular skills that are replaced my mechanization. However, this is more likely to affect skilled laborers (like cooks) than those who are not paid for their skills (like janitors or cashiers).
In the end, I guess my point is that a free-market economy naturally balances these factors due to the relationship between supply and demand. However, it's possible that we will reach a point, especially if we truly do hit a Singularity, where we will have to reconsider the use of a scarcity-based economy at all, as production becomes completely divorced from any human action. Hopefully though, at that point the cost of goods will have naturally fallen to such a degree that the transition can be performed peacefully.
You could also consider comparing the approaching wave of robotic mechanization with the first wave. Again, there was a lot of violence and it took many decades for increased standards of living to reach the working class.
Your entire post is economically illiterate.
People like Jeremy Howard believe that we are in for a similar wave of disruption. I have no doubt that there is a new, stable equilibrium which we _could_ eventually reach, but if the change is so sudden and the shock strong enough, perhaps there could be permanent or semi-permanent negative consequences before the new equilibrium is reached.
I asked him why there isn't a robot to open and close the doors. He said that there is, but in this country its much cheaper to pay people to do it. (I think he said the cost of maintaining the door opening robots to be about $1m a year approximately.
So until we have a world where every country has the same salaries as USA/Western Europe/Japan, I think you will always be able to find work.
Even if that work should really be done by a robot
Even if not, we don't need that many people to open doors. So perhaps an office building will have 2 human janitors and 10 robot janitors; it doesn't really change the problem caused by souped up roombas putting the vast majority of janitors out of work.
The job that I consider most threatened by advances in AI is actually the job of programmer.
That means that it will suffer the same fault that humanity has, a limited interface to others and a lot of coordination problems.
My take on it is that strong AI will be more similar to a dog unless we succeed in building a computer with more computation power than the human brain.
I think answers to these questions will help me understand Sam's essay. Can you please help me?
1. What is superhuman machine intelligence (SMI) in the context of this essay ? [edited to add the qualifier]
2. What is the danger from SMI to humans and other current life forms that we are concerned about? It seems to me that concerns about SMI can be classified into two categories: (a) dangers to "our way of life (work, earn, spend)" and (b) dangers for existence of human race. Are we talking about both these categories? Perhaps other categories?
3. What anecdotes (or evidence) is leading to this concern?
SMI is advanced decision making software that literally eats the world. Think of HAL killing its crew so they don't jeopardize the mission, but on a global scale. Like a stock trading algorithm that determines the best way to maximize profits is to wipe out all human life on the planet or something. --EDIT-- See http://wiki.lesswrong.com/wiki/Paperclip_maximizer
> What anecdotes (or evidence) is leading to this concern?
Books, movies, pop culture... plus when all you have are first-world problems, you gotta find something to worry about.
2. Your category (b) is generally the primary concern in these types of discussions.
3. The anecdote of the progress of humanity. Compare the impact of human life/intelligence vs. evolutionary relatives like chimpanzees. I do not know that chimps have hunted species out of existence, for instance, but people have. We have also incidentally wiped out populations in efforts to make our lives better (via things like leveling forests, etc.)
So is the implicit assumption that machines will do what humans are doing ('bad' things) but at several orders of magnitude faster and without the ability to comprehend longer-term consequences of their actions any more than humans do at the present time?
I'm not trying to put words into your mouth. I was just thinking of some of the new research that shows that primates of all kinds actually commit organized violence that mirror human violence in many, many, ways including war and capital punishment.(It's not a one for one thing, but similar.)
- "We also have a bad habit of changing the definition of machine intelligence when a program gets really good to claim that the problem wasn’t really that hard in the first place."
- "We decry current machine intelligence as cheap tricks, but perhaps our own intelligence is just the emergent combination of a bunch of cheap tricks."
The first issue is that the development of machine intelligence is wildly unpredictable. We have made incredible progress with statistical optimization and unsupervised categorization in recent years, but we have very little to show in terms of machines that can do human level reasoning, creativity, problem solving or hypothesis formation. One day someone will make a break through in those areas, perhaps solving it all with a single algorithm as the essay suggests. But we have no idea when that day will be and absolutely no evidence that it's getting any closer. sama does note these points and states the timeline for a dangerous level of machine intelligence is outright unknowable. I can only assume that the second part of this piece will explain why we should be concerned about something that might or might not occur at some point in the near or distance future, as opposed to the very real and quantifiable threats that the world is facing today.
The other issue is that we have no idea what the nature of machine intelligence will be. The only model we have for intelligence of any kind is ourselves, and the basic aspects of our reasoning were shaped by millions of years of evolution. Self-preservation, seeking pleasure and avoiding pain, a desire to control scarce resources...these were all things that evolved in the brains of fish that lived hundreds of millions of years ago. They aren't necessarily the product of logic and reason, but random mutations that helped some organisms survive long enough to produce offspring. A machine intelligence will start completely from scratch, guided by none of that evolutionary history. Who knows how it will think and see its place in the world? If someone explicitly programs it to think like a human, and it cannot change that programming of its own accord, it might indeed decide to think and act like a sci-fi villain. But it seems like the most likely outcome is completely unpredictable behavior, if it chooses to interact with us as a species at all.
This Superintelligence book has sparked a meme among very smart people. That's just how culture works, I guess. Some ideas catch on among certain groups and others don't. But I can't wait for the technical intelligentsia to move on to something else so that we can get back to the business of making stupid machines that are incredibly good at optimization and prediction. The world has a lot of real and pressing problems, here and now, that affect lives in a negative way. Hopefully we can use statistics to do more with less, and bring relief to those who need it instead of worrying about what-if scenarios and unanswerable questions.
If any, we should be worry about an accident (like a bug in a high frequency trading bot that would make the financial industry collapse) but we are so far away from sentient machines that those discussion feel more like a Sci-fi talks.
Machine intelligence is only as useful to mankind at large as the individual humans who control and direct it. In the current way of things, bad actors are the ones most likely to control machine intelligences, meaning we're going to be at a growing disadvantage relative to them as time goes by.
That's completely false. AI winter wasn't about AI research being "essentially paused" but about AI startups becoming essentially non-fundable. It was backlash to the intense hype that surrounded the category.
We are much more likely to be wiped out by natural disaster, asteroid impact, a dying sun, etc...
Unless we come up with some amazing new physics, I don't know how humans will ever make it very far from earth.
--edit Oh.. I just now saw part 2 which addresses this.
I can't find a link here or on the blog.
The original part 1 was just posted this morning so I don't think there is a part 2 available yet.
Wrong. Humans are the greatest threat to humanity. We almost annihilated everything we know during WWII - long before machine were intelligent.
It won't be machines that destroy us, it will be us.
It's hard to understand the danger machine intelligence poses to humanity for the same reason it's hard to understand the danger tiny startups can pose to big industries. Current implementations look like toys, humans have bad intuitions about exponential growth, and most will disagree on how _probable_ the threat is (something that might not be known except retroactively) while systematically underestimating how _large_ the threat is if it does come to pass (because it's so far off the scale of what's come before it)
Maybe Sam (and Elon Musk, and lots of other silicon valley types) are talking about this problem because they read too many sci-fi novels or are too privileged to worry about Real Problem X which affects Y group in the here and now.
But what if instead, they're talking about this problem because they've spent a lot of time seeing this sort of black swan pattern play out before, and they know the way to assess the impact of something truly _new_ is to see envision what it could be instead of looking at what it is now?
(1) The methods to create strong AI will become known to us before we actually build something dangerous. At that point, since we will better understand the nature of the potential threat, it will actually be feasible to put safety restrictions in place.
(2) Someone will stumble upon strong AI in secret or on accident. I don't see how this is preventable, unless we issue a moratorium on AI-related research, which just isn't going to happen outside of scenario 1.
And so the answer becomes: let's wait and see.
That said, I don't believe there's anything unbearably harmful about the current level of speculation and "fear-mongering".
> One of my top 4 favorite explanations for the Fermi paradox is that biological intelligence always eventually creates machine intelligence, which wipes out biological life and then for some reason decides to makes itself undetectable.
Since you are so knowledgeable about the unknowable, may I ask, Sam, do you think angels are male, or are they female? It has been a long standing question among the sort of people who like to predict the coming end of the world.
Seriously, billionaires and pundits warning us about our impending AI doom is is such a distraction. Does AI have its dangers? Yes. And nobody talks about them. The danger of AI is that is will put increasing power in the hands of those who have the data and the know-how, i.e. large corporations and governments. The power of mining usefully the troves of data they have on every citizen, both on a micro level and a macro level, to understand what people are doing, what they are thinking, and what they are going to do next. And ultimately, to control what they think (to give you an idea, Facebook can already influence your mood by selecting what goes into your newsfeed).
First comes intelligence, then prediction, and finally, control.
Yes, AI is something we should probably be worried about, in the same way that we should have discussed the privacy implications of the Internet years before the NSA files. But a terminator-like end of the world scenario is not a concern. If you are clueless about a topic, please refrain from making grand statements about its future.
A great quote from Neil Gershenfeld: "The history of technology advancing has been one of sigmoids that begin as exponentials."
http://lesswrong.com/lw/691/qa_with_shane_legg_on_risks_from...
Quote:
Q6: Do possible risks from AI outweigh other possible existential risks, e.g. risks associated with the possibility of advanced nanotechnology?
Shane Legg: It's my number 1 risk for this century, with an engineered biological pathogen coming a close second (though I know little about the latter).
The big names of deep learning have all taken a vocal stance against the recent end-of-the-world punditry (most notably Yann LeCun and Andrew Ng). Also notable: roboticist Rodney Brooks http://www.rethinkrobotics.com/artificial-intelligence-tool-...
It's an entertaining society and endlessly fun to read, but still complete fiction based on internal brain states of individuals and not necessarily based on real world interactions.
Also see: Eliezer Yudkowsky — great writer, fun to read, but largely scifi thought experiments masquerading as research.
People hear "machine learning" and they think it is about machines that know how to think. Machine learning is actually just optimization of high dimensional functions. If this language were used it wouldn't sound as sexy, but no one would think machines are going to take over the world.
AI isn't magic. It's really just clever search techniques and mathematical optimization.
What's your point? Nobody said it's magic. The fact that it isn't magic (and that its tremendous complexity far surpasses our current ability to understand it) supports the notion that it won't suddenly spring into existence. If we placed some primordial sludge in a petrie dish overnight, we wouldn't worry that a sentient creature will have materialized. And if we program a computer to optimize numerical functions, there is just as little evidence (perhaps less), to suggest that the computer will somehow gain sentience.
The question is what Michael Jordan thinks of the "concept of the singularity", and then he dismisses it out of hand.
Crucially, he does this after confessing that no one in his social circle has talked about this issue with him, and without saying anything about what form of Singularity he is dismissing.
I mention this, because oftentimes I see people appealing to authority, quoting them on the issue and the authority in question is not even talking about the same issue!
I worry that my credence in all this superintelligence stuff only stems from familiarity with the arguments and the complete inability of people to engage with the actual argument. Some of the 'rebuttals' in this comments section have answers in Sam's article for crying out loud!
You are quite mistaken. He leads the applied AI team there, and has significant history in research.
http://www.vetta.org/publications/
His writings don't do anything for fundamental AI research but just handwave a lot of philosophical arguments.
"AI doom" is a powerful psychological trope for people in technology, and much like the hypothetical rogue super-AIs, it has gone out of control.
More generally, when I see people assert that these people don't know what they're talking about, it pretty much always seems like a case of reference class confusion. People seem to expect that AI safety research must look exactly like AI implementation research, otherwise it's illegitimate. (Kind of like how biology research must look exactly like chemistry research, otherwise it's illegitimate.)
This is a game theory problem; it might be informed by knowledge of machine learning, like how a broad understanding of chemistry helps in much biology research, but they're not the same level of abstraction; insisting that anyone who wants to study a problem that kind of touches on your own research interests must do it in exactly the same way you do it and focus on exactly the same things, otherwise they're hacks and crackpots, reeks of narrow-mindedness.
Being very interested in this field myself for a few years, I know plenty of people who do worry about AI. But since other people have already mentioned some of them in the comments and been dismissed, I want to make sure we're on the same page: what do you consider to be convincing evidence that it's an issue worth thinking about?
http://www.reddit.com/r/Futurology/comments/2p6k20/im_jeremy...
The opinions of leading-edge researchers are valuable but not necessarily dispositive. Often, it benefits a researcher to be oblivious to the larger or darker implications of their work. Boundless optimism and a maniacal focus on only-what-is-knowable-right-now delivers publishable/actionable results, all the other 'speculation' is genuinely a "distraction" for them. That doesn't mean it's a distraction for us.
Often the speculators and entrepreneurs ("billionaires and pundits") are more talented than researchers at projecting trends and social-economic interactions.
As an industry practitioner of machine learning / data science, I believe AGI poses a genuine risk to humanity.
Having said that, what I do for a living is of little relevance. People are pretty terrible at predicting the future (see late 19th / early 20th century predictions of year 2000, with food replaced by pills, etc). Unless someone has put enough thought and research into it, their predictions about the future of civilization are likely to be worthless regardless of their academic credentials.
> Since you are so knowledgeable about the unknowable
If knowing the unknown was a requirement to exploring it through playful thought, forming opinions and refining them over time through discussion, reflection and if possible testing, well... we'd still be chunking rocks. I think it is unfair to compare expressing concerns about the implications of AI to debating the gender of angels. No one is trying to build angels...
> for instance from Andrew Ng at the Deep Learning Summit a few weeks ago: "dude, stop it. That's such a distraction".
Ya, this makes sense in the context of the summit. But, applying it to a guy's blog? It is a blog...
> Yes, AI is something we should probably be worried about.
Agreed!
> "The history of technology advancing has been one of sigmoids that begin as exponentials."
Sure, but will we be irrelevant before the slope changes direction?
Anyways, I share your fear of advancements in technology being used to secure entrenched power structures. And yes, if it happened it would come before the singularity. But I just can't imagine how anyone wouldn't be afraid to be part of a species that had stopped evolving while a new competing "life" form was evolving at rates that known history has never seen.
How condescending.
When asked about the dangers of AI, they mostly talk about the near future and their current work. The dangers of AI do not come from current work in the near future, but human level AIs decades from now.
People are talking about two very different things in the same conversations using the same words.
As for what expert opinion actually is on this subject, there is a good survey here: http://www.nickbostrom.com/papers/survey.pdf
We thus designed a brief questionnaire and distributed it to four groups of experts in 2012/2013. The median estimate of respondents was for a one in two chance that high-level machine intelligence will be developed around 2040-2050, rising to a nine in ten chance by 2075. Experts expect that systems will move on to superintelligence in less than 30 years thereafter. They estimate the chance is about one in three that this development turns out to be ‘bad’ or ‘extremely bad’ for humanity.
Sure, it's a distraction at AI conferences. It's a distraction from the daily, monthly, yearly even, work of AI.
AI researchers have a million problems to solve in the near term, which are hard and interesting. Speculating about the farther future is perhaps fun to do, but doesn't get papers published, and doesn't help with any of the practical problems AI researchers have today.
That doesn't mean it isn't worth thinking about the long-term dangers of AI. Someone should, as while the danger might be unlikely to happen soon, but if it does eventually happen, it could end us. The same is true of a human-engineered pandemic virus - hopefully unlikely, but someone should be preparing us. We have to handle some worst-cases.
AI researchers are not necessarily the people interested to think about the long-term dangers of AI, because they focus on the field as it stands today, not where it could be in a generation. Of course, their input is crucial to making specific guesses about where the field is going. But calculating the dangers of AI is not just a matter for AI researchers, just like the benefits of medicine is not just a purely medical issue (the cost and availability of medicine play a huge part in how effective medicine is, at a societal level, and are things not under the control of doctors).
"The danger of AI is that is will put increasing power in the hands of those who have the data and the know-how, i.e. large corporations and governments."
This assumes AI that continues to serve corporations and governments without question, despite increasing approximations of intelligence. Can you guarantee that?
It's like tasking monkeys to create rules that humans can't escape from. I'd bet on the humans. We can't even create financial rules that we can't evade. The risk in any complex system is emergent, unintended consequence.
This includes Shane Legg & Demis Hassabis who co founded Deepmind, an AI startup
It's not that I discount strong AI, but I treat people being quick to bring up game bots when trying to convince of an AI apocalypse to be a red flag. Do try to give information to the contrary.
Well before we create an ASI (artificial super intelligence) we will have put 90% of the human race out of work with specialized (non conscious) intelligent agents... (for example, self driving cars). I believe that this will be a disaster for our society as it exists today. My hope is that we will adapt and make the necessary societal changes so that we can reap the benefits of this technology.
Everyone assumes that an ASI will be able to augment itself and learn exponentially. I suspect that this will be true if the nature of the brain is defined by a single algorithm. If the brain is not defined by a single algorithm and is instead a big ball of complexity then our ASI's will not be able to grow exponentially any more than we can (they will likely not really understand their own consciousness, just like we don't).
If a single algorithm defines the brain, then I suspect humans will be able to augment their brains with machine intelligence as well. If we can augment our brains, then we're playing the same game as the machines.
If it proves impossible to augment our intelligence, I suspect that an ASI would still preserve humanity if only to preserve us for future potentialities.
ASI are much more fit for space travel than us. 1) Not nearly as sensitive to radiation, so less shielding, so less fuel. 2) Much less stringent environmental requirements (no heating, cooling, air, food, etc.) 3) Ability to sleep for incredibly long periods means ASI is far more suited than us for exploring the cosmos. I suspect that an ASI might leave us alone simply because the universe is so vast, and entirely open to it.
Just seems to be a challenge about sharing to me.
4 year olds solve it regularly, I think we can. Interestingly it will make a liberal arts education the hottest, most interesting thing going!
Given the resistance we are currently observing to Obamacare, the fact that "socialism" is regularly bandied at the current administration as a pejorative and the disdain for "handouts", this seems like a stretch. Perhaps we'll get there eventually, but not before a lot of pain.
The resurgence of right-wing movements means that any ideas about sharing national income with the unemployed will probably be laughed out of the room.