What, specifically, are you referring to? "[S]cientific advancement eventually happens if the laws of physics do not prevent it" sounds pretty accurate - sure, we might get stuck in a local maxima for a bit, but there's no reason to think that progress towards the things Sam is talking about will stop in any meaningful way over the long term.
> but there's no reason to think that progress towards the things Sam is talking about will stop in any meaningful way over the long term.
You could have said the same about the philosophical stone, which people thought that was possible to "discover" back in the late 1700s, or about us, humans, physically reaching other solar systems and especially other galaxies, which some people thought possible for a while after WW2. It's a belief similar to how some religions started, definitely similar to the early Christians' belief that the second coming was only a decade or two away.
True, but there's a lot of things being talked about in this blog post (e.g. superhuman AI, genetic enhancement, and brain-machine interfaces). These are all different things, and I was genuinely curious as to what the OP found to be "a little out there".
I view HN (and soon the world) through a filter that is designed with my information and emotional needs combined. I quit Facebook years ago. I pick and choose the technology I use.
I've stopped watching TV. My use of the internet is pretty much limited to email, paying bills, HN and one or two other news sites, and work (developer/system admin). I'm not on Facebook, LinkedIn, Twitter, or any other social media. My phone is used as a phone, for SMS, and Google Maps when driving to someplace new.
Other than my phone I have no "smart devices" in my home and doubt I ever will.
I'm not a Luddite -- I just see no personal benefit in any of the things I've mentioned.
"It’s probably going to happen sooner than most people think. Hardware is improving at an exponential rate—the most surprising thing I’ve learned working on OpenAI is just how correlated increasing computing power and AI breakthroughs are—and the number of smart people working on AI is increasing exponentially as well. Double exponential functions get away from you fast."
I'm not convinced that an exponential number of people working on AI produces exponential advancements. Wouldn't we see diminished returns with each new person, presumably each one is less capable and less expert? I see AI experiencing a hype-cycle like disillusionment before seeing "double exponential" returns.
Diminishing returns on an exponential can still be exponential.
Typically, diminishing returns means f(x) = x/(1+log(x)) where x is headcount and f is output. The overhead goes up with the log of the number of people, because a hierarchical organization will have log(N) layers of management that need to be traversed to make decisions.
If we have exponentially increasing manpower, x=exp(t), then f = exp(t)/(1+log(exp(t)) which is O(exp(t)/t)
Well said. For example, with the hardware improvements of late (GPUs, TPUs, etc.), we are seeing lots of low hanging fruit as we haven't pushed in this direction before, but I imagine this diminishes soon.
It might just take one genius like Albert Einstein to create a new paradigm, so I could believe it happening sooner rather than later with the more people are working on it.
I suppose it depends on your view of the theory of scientific revolutions.
Good point. I tend to subscribe to Kuhn's Structure of Scientific Revolutions [0] which I think suggests that a paradigm shift comes as collective observations mount and it is eventually exhibited through a genius on the cutting edge.
I don't think we need a genius or a new paradigm for AGI. I think the main ideas of a few different approaches to do it are already out there. There are a lot of tough problems to solve but to me it looks like it's a matter of hard work rather than any totally new approach.
Look up for example Yann LeCun's AGI presentation. Now see projects like Ogmaneo or recent Deep Mind papers that do fast online learning or have ways to work around catastrophic forgetting and other issues raised by LeCun.
I believe that one straightforward way to get there is to continue to apply neural network research towards attempting to mimic animals. And there are a bunch of people doing that and making progress.
That is harder than you might think. For example, the simplest animal with a nervous system, C. elegans worm has been simulated "only 20 to 30 percent of the way towards where we need to get" according to one of the OpenWorm project leaders. That's 302 neurons total.
Using a computer analogy, it's like struggling to build an abacus when our ultimate goal is to build a Core i7 processor.
rather than any totally new approach
It's not clear to me that we have any approach to achieve AGI. People like Ben Goertzel have worked on that for decades without much to show for it. Current deep learning methods have little to do with AGI, they focus on very narrow applications, by design.
I didn't say simulate brains or that it would be easy. Research what I suggested a bit more carefully because it addresses multiple parts of your response.
The AI research space is still a pretty green field. Right now you can basically take any random paper and combine it with another random paper and you will be able to find a use case where that succeeds and write a new paper about it. More experienced researchers will, of course, have a better intuition/knowledge of what techniques to combine to get more impressive results.
As long as it stays like that (and by opening up new sub-areas, it could stay like that for quite some while), adding more somewhat qualified people to the field results in all the useful combinations being discovered faster and thus exposing new potential starting points. Probably no exponential growth in the strict sense, but quadratic growth isn't bad either.
I'd be happy if my phone's autocorrect feature would actually produce sentences with proper spelling and reasonable grammar. Apparently that's still too hard for our mighty machine learning systems. I'm not going to merge with any technology that is worse than my stupid mush brain.
[edit - why am I still editing the spelling and grammar on this post?]
It's just a matter of doing things in a proper order. World conquering super intelligent AI first, then really high quality auto-correct. It's actually what our AI God will spend all of its time on, correcting spelling & grammar for 13 billion people sending trillions of increasingly poorly formed messages.
Just because that good autocorrect AI or AGI isn't available today doesn't mean that we should dismiss it as something that we don't need to worry about. Because the change to our way of living is so great, we should start making plans, even if it is 10 or 30 years out.
Personally I believe we will see really interesting animal-like AGI demos in 2018 and 2019.
It's just dumb that people can't take this stuff seriously until all of the engineering and research is 100% done.
I'm not dismissing it. On the contrary, I can't wait for brain-machine interfaces. I can't wait for really good self-driving cars. I'm just not as optimistic as many of my colleagues.
Can we aim lower? I'd settle for one that auto corrected "the" and "they" whenever I (frequently) get it wrong. After that I'd like one that could sort out the "there", "their" and "they're" thing for me.
Most people in their prime today will say no to implants. I suspect that 2% may be ok with it. Each generation however will have a higher and higher percentage OK with it. Especially considering at a certain point it will be the competitive advantage to merge oneself to tech. Similar to developers taking derivatives of speed today.
When would this start? Probably not limited to the pace of AI but the pace of human computer interfaces catching up. I suspect the largest increase of usage will be the non-invasive helmet electro-magnetic style that is already proven to work to some degree.
Generally too, HCI can help us create a massive amount of training data.
>Our self-worth is so based on our intelligence that we believe it must be singular and not slightly higher than all the other animals on a continuum.
Speak for yourself. I doubt Sam meant to appear ignorant of other perspectives on the world. However it would be nice to consider them when writing general insights, as opposed to simply tunneling his own point of view.
I fear this is wishful thinking. Advances in AI are progressing a lot quicker than advances in neural interfaces. So we will most probably have superhuman AI long before we have neural interfaces.
And at that point it's game over for homo sapiens.
Neural interfaces have use cases but they are not the silver bullet for the kind of protocols that we need in order to not lose control of our creation.
The protocol that that we need is for transmitting information, not data. Or in other words, you don't need a neural interface to transmit understanding.
Was there this much hand-wringing during previous AI boom/bust cycles? There's a lot of fear and emotion swirling around AI/machine learning right now, and I'm curious if that's been the case in the past as well.
Well drones have really captured the imagination of the public this time, and we have very public and popular tech leaders warning the public about AI.
Combine that with the 24/7 media clickbait cycle, I think we have a unique situation.
"Until I made a real effort to combat it, I found myself getting extremely addicted to the internet."
This seems to imply sama is resisting the inevitable merge. Can't we instead try to steer it in a more positive direction? Where are the startups trying to keep you addicted to the internet in a way that improves your life? They don't seem to exist, because the incentives aren't right -- it's way more profitable to keep you hooked in ways that make your life worse. How do we fix those incentives?
they do exist - consider the gamifying elements of Khan Academy, Duolingo and other e-learning sites for example. The problem seems to be that they are out-earned by the sites that feed into our narcissistic and negative impulses - social media, addictive games, "recommendation engines" on shopping sites and such.
>> Where are the startups trying to keep you addicted to the internet in a way that improves your life?
> consider the gamifying elements of Khan Academy, Duolingo and other e-learning sites for example
It's worth noting that I've heard a lot of criticism of e.g. Khan Academy from certain components of the mathematics education community. Concretely, they feel that the material there is designed to increase engagement and provide a sense of accomplishment ASAP, and that the learning processes that fit those goals are detrimental to development certain mathematical skill sets. I.e. you might learn how to compute derivatives really well, but at the expense of really understanding limits or the fundamental theorem.
Mind you, mathematics educators have always been an internally divisive bunch, so take from it what you will.
Recently I've been getting into cryptocurrencies (too little, too late, right?) and it's actually improving my life by making me money. It feels like the more I learn about them and the more I gamble, the better my life is at least while the value keeps climbing. I get close to zero value out of HN/Instagram/Twitter, but I may be using them wrong.
> If two different species both want the same thing and only one can have it—in this case, to be the dominant species on the planet and beyond—they are going to have conflict.
The term "dominant species" really jumped at me as I was reading this piece. It's so fantastically vague that it begs the question - would us and AI have the same definition for what "dominant" is?
We're not building AI individuals with self-preservation instinct, hunger for resources and sexual drive. We're building super-individual systems like Google, Facebook and trading systems, that can be much more intelligent than we are but also have vastly different built-in "purposes" (just as evolution have "built-in" purpose of eating, having sex and caring for our family into us).
I think that in the end, the thing we're building will end up much closer to Solaris ocean than Terminator.
Well, the issue is we are not building general AI systems at all yet. If and when we can, the game changes, and we pretty much cannot figure how it will until it happens.
There is no other general intelligence at this point that can rival human intelligence, we have a systems sample size of 1. Making predictions based on such a wide range of possible technologies is hopeless at this point.
Why would a fixed placement general learning computer system that takes input from millions of possible sources evolve in the same way that a mobile robot with general learning intelligence? Will both be possible? What about swarm intelligences?
Too many questions with no possible way to answer them.
What does a rapidly improving AI have to gain by "merging" with our notoriously error-prone, finicky biological hardware? It's nearly certain that intelligent machines will choose to replace humans, not enhance them.
Our "successful" descendants will probably be the unmerged, low-tech survivors living on the outskirts of a new "Machine Age" civilization.
I believe the merging Sam is referring to here is augmenting human intelligence with non-intelligent (but better) machines. Kurzweil talks about this as well, and was one of the original options in Vinge's "Singularity" (https://edoras.sdsu.edu/~vinge/misc/singularity.html)
> Unless we destroy ourselves first, superhuman AI is going to happen... Perhaps the AI will feel the same way [that intelligence is singular] and note that differences between us and bonobos are barely worth discussing.
He's referring to a superhuman artificial intelligence, not merely an augmented human intelligence.
The friendly AI goal alignment problem is a pretty interesting part of this and there’s interesting work going on there.
The basic idea is how do you effectively align a general intelligence’s goal with humans so that it uses its intelligence to solve problems in a way that aligns with our own human utility functions.
If humans figure out general AI before goal alignment it may have outcomes we don’t want.
Maybe it would be better to stick to narrow AI. There's no reason a self-driving car or stock-picking algorithm or even a cancer research algorithm needs to be general AI.
In some sense we got lucky with nuclear weapons because they require enriched uranium to make which is hard to get.
A self improving general AI may just be be a matter of an intuitive leap, but nothing that’s hard to turn on once you understand it. This property would make it hard to suppress.
The upside is that if we can figure out the goal alignment problem we may be able to fix the coordination issues humans have and have a much better end state than all of us burning out and going extinct on our planet.
The downside is some variant of the world turning into paper clips.
If it's that easy, we're doomed. Even if goal alignment works, you can't make people use it. And even if they did, whose goals would they have it align with? The goals of evil humans are plenty bad enough.
In the short term it would still probably take a lot of computing power. If you can figure out the goal alignment problem (or even simpler parts like just how to get the AI to let you turn it off) then maybe the first version can help us prevent rogue general intelligences that do things we don't want.
It is a hard problem though which is why there's all the talk of existential risk. It's also a little nuanced to explain why there's concern or why people should work on the problem.
I thought this post from yesterday was a good response to someone stating that an intelligence explosion was impossible (the original article it's responding to is linked within it): https://intelligence.org/2017/12/06/chollet/
If people make friendly AIs first, then we might be able to use those friendly AIs to figure out the actions that unfriendly AIs would take to get power, and close them off first.
Okay, but do you need general AI for that? Why not narrow AI?
On the defensive side I think there's a lot that can be done with narrow AI. Consider using really good penetration testers and bug finders to fix Internet security.
For any given task, build a tool that just does that. It's like the least privilege principle in computer security.
I think there's a ton of routes to power for a general AI other than network security holes, and someone making a narrow AI just to fix all the bugs in our computer networks will miss nearly all of these. Consider an AI that makes a ton of money on mechanical turk -like jobs online, uses the money to run thousands of shell companies, over time hires the whole population to work on things related to paperclip factories, and crowds out other job opportunities so that anyone that goes against it goes jobless. Or an AI that figures out how to create cultural memes stupidly easily, gets people to vote in totalitarian governments, and then blackmails every single member of the central planning committees to optimize their countries for paperclip production.
These ideas are mainly extreme proof-of-concept ideas, maybe there's ideas along these same lines that are much easier to pull off, but maybe these ideas as-is would be do-able for a possible superhuman intellect. I'm kind of thinking about what it looks like when individuals meticulously plan and then make tool-assisted speedrun videos through videogames, and then scaling it up to what it would look like if thousands of people worked together optimally for thousands of years brainstorming and calculating on how to speedrun through conquering a society. Physical limits permit this kind of computation as far as we know. Maybe this imagery is off-base, but I'm purposefully shooting high and worrying I'm not shooting high enough. Chimps would never even consider the possibility of "super-chimp" intelligences creating nuclear weapons. You can't make narrow AIs to defend from these possibilities if half the danger is our inability to even enumerate them.
I kind of think that we are already experiencing the fall-out from the value-alignment problem with corporations. Corporations (and the market in general) seem rather good at optimizing revenue/profit - the problem is that corporate profit doesn't always align with other human values - which ties into some other comments here about disrupting human attention-span being profitable.
Perhaps work on AI goal alignment can help us build a more humane market place, or vice versa, research into macro-economics can inspire some ideas in AI goal alignment.
Unfortunately, I think this is because we have stopped questioning whether corporate values really do end up matching human values. I think when capitalism was seen as a competitor to communism in the Cold War era, people took a slightly more critical eye at economic systems, but now there's no competition so there's no pressure for corporate values to align.
In short, I argue that non-evolutionary superintelligences are not stable (they eventually go inert), while evolutionary superintelligences are a very serious existential threat to our species.
>Double exponential functions get away from you fast.
Still exponential, though.
Also, as a general criticism, there's a big difference between people getting addicted to the internet, getting dumbed down by it and sucked into things like a youtube hole, and my idea of The Merge. The things you describe sound more like an automated soap opera or opiate addiction than the singularity.
Advancement in nuclear technologies was basically shut down since the 1970's. Thiel's thesis is that almost no technological progress has occurred since then. I tend to agree. Computer technologies were only allowed to progress so quickly because people did not think computers were dangerous. Imagine any other new product where you could state that the product is "as is" and claim no responsibility for its functionality, purpose for use, or damage caused by malfunction. I would prefer this concept for most things but our "safety first" society definitely does not.
The one avenue open to tech advancement became so powerful that it eventually began to bleed into the physical world and we are starting to see other tech slowly advancing again. Maybe if the tech wealthy of Gen X and later have sufficient power in society after the Boomers pass the baton (if they ever do), we will let the technology keep advancing and see this merging. There is also the fact that the US is no longer a hegemon. Good luck stopping people in China from doing things banned in the West.
The objective of international nuclear regulations is to avoid a small group of hotheads from wiping out humanity/most of the world’s ecosystems due to their botched experiment (and that’s giving them the benefit of the doubt, you could also imagine some crazy cult who believes that inducing a nuclear winter is the only way to heaven).
What do you think ISIS would look like in the world you describe?
This is exactly the confusion that I am talking about that stalled nuclear power technology. Nuclear weapons are bad to have floating around and getting the current stockpiles down to low levels would be great, but building a nuclear weapons is very hard to do. Separating out U235 from U238 to make and easy to build gun-like nuclear weapon takes a nation state level of power and dedication. Getting enough pure plutonium 239 is at the same order of difficulty and building a plutonium bomb is very difficult and dangerous.
Fuel in nuclear reactors do not have to be a weapons grade levels and only a few research reactors are (or were, as they are being phased out).
Also, one or two (or a dozen) of small nuclear bombs set off by some small group is not going to wipe out humanity and most of the worlds ecosystems. Horrible, massive death and destruction for sure, but not an existential threat to life on the planet.
> Advancement in nuclear technologies was basically shut down since the 1970's
That's not true at all! Civilian nuclear power in the USA (and most of the west) has stalled out since the 70's, but nuclear technology has seen plenty of advances over the intervening time frame. See esp. modern nuclear propulsion systems. Super impressive.
Also, this isn't a problem caused exclusively or even primarily by safety regulation in the nuclear sector. The evidence is in the failure of nuclear power throughout the world, despite significant variations in safety regulations.
In fact, if you look at root cause analyses for the failure of nuclear energy, you'll find that lack of regulation is a major reason for the failure of the nuclear energy. If the FF industries had to pay for their externalities, nuclear would be extremely viable (at least in the 80s-00s; now it'd have to compete with solar and wind).
> Computer technologies were only allowed to progress so quickly because people did not think computers were dangerous.
Again, this is a pretty wild assertion... Moore's law >>>> regulatory environment. Seriously. If the output of nuclear power plants had grown exponentially for multiple decades, we'd be in nuclear-powered paradise.
I am not sure what you mean by "modern nuclear propulsion systems". Some links to details of those would be great. I'm hoping for applications outside military ship and submarines, which have been around since the 1960's. There were actual engines for nuclear rockets, nuclear airplanes, nuclear excavation techniques, etc in the 1960's. Definitely problems with those systems, but we stopped trying. Nuclear test ban treaty, etc. I'm not saying that was not the correct path for a better future (hard to say), but society definitely stopped working on that stuff.
"Moore's law >>>> regulatory environment"
I don't see how this is a disagreement with what I was saying. Imagine if society had the same amount regulation on building software systems as dealing with radioactive stuff. Moore's law would not exist.
"If the output of nuclear power plants had grown exponentially for multiple decades, we'd be in nuclear-powered paradise." Why didn't that happen? Lots of reasons, but I think it could have and still could given a chance.
> I'm hoping for applications outside military ship and submarines, which have been around since the 1960's.
I'm referring to military ships and submarines. I'm not really sure where the financial incentive for nuclear comes from for anything commercial?
TBH it seems like you have a culprit (regulation) in search of a victim (nuclear utopia), and as a result, you have a solution (nuclear power) in search of a problem. If that's the case, there are really many much better examples.
> Nuclear test ban treaty, etc.
The NTBT covers intentional detonation of nuclear weapons... I'll be the first to concede that regulation has perhaps hamstrung the private sector nuclear weapons market :-)
> engines for nuclear rockets
Radiation is a bitch and lead is heavy. Conventional rockets that we need for human space flight already can, with modification, get us to where we've wanted to go so far. So we've instead focused our resources on doing useful stuff once we get there. NASA doesn't have inf money.
This is something that people continue to research. What's holding it up is a lack of priority in decisions about what science to fund and a complete lack of any private sector market large enough to justify the investment, not any sort of regulatory barrier.
> nuclear excavation techniques
...I'm having an extremely difficult time thinking of a use case where this would make any sense. Nuclear bombs are more powerful than conventional explosives. That power has an enormous tactical advantage in wartime because you can take out a city with a single missile instead of weeks worth of bombing runs.
But that's not really a particularly beneficial feature in e.g. a commercial mining operation. Plus radiation's a bitch.
Help me out? What are the possible use cases here?
> nuclear airplanes
Radiation is a bitch and lead is heavy.
> I don't see how this is a disagreement with what I was saying. Imagine if society had the same amount regulation on building software systems as dealing with radioactive stuff. Moore's law would not exist.
This isn't clear to me. I think the hardware would've still be invented. Perhaps the business impact would've been smaller, but regulation of commercial software systems wouldn't have stood in the way of the enabling physics and engineering research.
Also, see the Ford quote about Microsoft; if the nuclear industry worked like the software industry, we wouldn't even be here to have this conversation. We'd have BSoDed our way to nuclear Armageddon some time around 2001.
> nuclear power plants had grown exponentially for multiple decades... Why didn't that happen?
Physics is a bitch. More precisely, I'm unaware of any serious conjectures by modern nuclear scientists that there are obvious advances on the horizon that could give us exponential improvements to reactor output even in the short term. Let alone over 3+ decades.
Aside from power, and perhaps inter-planetary travel, nuclear is a case of "tried it, doesn't work for fundamental reasons" or "wtf now you're just throwing nukes on things for fun". And in the case of power, we have a lot of data points from a lot of different regulatory regimes, all of which point to "this is a quite expensive way to make power which won't pay off until fossil fuel externalities are finally internalized... i.e., never"
I hope someday the risks from radiation will be evaluated on equal par with other risks of a modern society and nuclear power options will be viable again. After decades of technology stagnation in rocket launch business, Musk and company has orbital class rockets landing back and the launch pad and is looking into nuclear rockets. I wish him luck.
What is the primary reason? Many people would love to work on such a project and some people would fund them. Would you agree that fundamental physics does not preclude a functioning nuclear rocket? Where would one build, much less test, such a device as a nuclear-powered rocket? We can't even agree in the US where to bury inert (mostly) ceramic radioactive waste. A live, radioactive (low-level, but still radioactive) exhaust stream is not going to happen in the US in the current culture.
> It is a failure of human imagination and human arrogance to assume that we will never build things smarter than ourselves.
If you define intelligence as “being really good at chess” or “factoring prime numbers”, sure, computers are already smarter than us. If you define intelligence as “knowing when to let your child make mistakes on their own and when to help them”, or “knowing how to conduct an orchestra”, it doesn’t seem so extreme anymore.
In fact, the opposite statement rings just as true:
It is a paragon of human arrogance to assume that we will build things smarter than ourselves in every conceivable way.
The road ahead looks more like a planet with its ecosystems ravaged by resource extraction, with buggy computerized systems we don’t understand running people’s lives in harmful ways (eg see all the writeups about machine learning reinforcing systemic biases) than a world full of meta humans in symbiosis with inconceivably intelligent machines of their own design.
>“knowing when to let your child make mistakes on their own and when to help them”
That can be tested and quantified, based on end state goals.
>The road ahead looks more like a planet with its ecosystems ravaged by resource extraction
In such a world, the savviest and more strategic 1% would thrive, just like they always do. Eventually though, they will need to merge with machines to continue making the cut. It's too advantageous not to.
> I believe the merge has already started, and we are a few years in. Our phones control us and tell us what to do when; social media feeds determine how we feel; search engines decide what we think.
I think this is a misunderstanding about human nature. Humans are thinking and feeling beings who are responsible to others. They are only slaves and automatons if they choose to; most aren't. Many who are have hints and feelings that what they are doing is unnatural.
>They are only slaves and automatons if they choose to; most aren't.
I completely disagree. Most are. In fact we have study after study showing this the case.
You want to focus on the modern automatons, such as phones and social media, but we have plenty other examples. Focus on the clock. Focus on the law. Focus on societal and cultural norms no matter the negative effects that these things have. These things have driven us since antiquity.
What we call AI is very good at pattern recognition. I haven't seen examples yet though of AI learning quickly. It can teach itself how to play chess, but it takes a very large number of attempts before it becomes good. The rate of learning for a human is still much faster than for an AI. (We just hit a plateau faster). I'd put my money on a child that has played 10 games of chess vs a computer that's learning from scratch and has played 10 games. I wonder if there have been any studies on trying to speed up the pace of learning for AI.
It's difficult to compare. A child learning chess invariably has an adult around to comment on their game and improve it, and frankly after 10 games if they still remember the rules that's an achievement (if the child is young).
If you did the same approach with a child that could fully comprehend the rules from the start, playing another child, both of whom had never played before, I really don't think they would have learned as much as the computer would have done. It would be an interesting experiment - my bet would be that the children had invented another game and were playing that instead.
> I wonder if there have been any studies on trying to speed up the pace of learning for AI.
Isn't there one posted to HN about every day?
> It can teach itself how to play chess, but it takes a very large number of attempts before it becomes good. The rate of learning for a human is still much faster than for an AI.
But this is just a power optimization versus availability problem. AGZ played 44 million games in 8 hours and became better than the best program which is better than the best humans. Optimizing for a human level of power usage doesn't seem like the best method at this point.
Sorry if this is more negative than HN permits, but am I the only one who gets the feeling that the style of this post sounds kind of smug? Ending a paragraph with "And gradual processes are hard to notice" and an invisible smirk kind of leaves a bad taste in my mouth.
187 comments
[ 0.21 ms ] story [ 118 ms ] threadYou could have said the same about the philosophical stone, which people thought that was possible to "discover" back in the late 1700s, or about us, humans, physically reaching other solar systems and especially other galaxies, which some people thought possible for a while after WW2. It's a belief similar to how some religions started, definitely similar to the early Christians' belief that the second coming was only a decade or two away.
I expect a technological singularity to supplant/replace biological life completely.
I don't think I understand.
Since sooner or later, biological systems would be inferior to manufactured ones.
You would feel better (particularly in regards to the emotions you describe at the end), Sam, if you put yourself on a "information diet".
J/K, I'm a luddite too.
I've stopped watching TV. My use of the internet is pretty much limited to email, paying bills, HN and one or two other news sites, and work (developer/system admin). I'm not on Facebook, LinkedIn, Twitter, or any other social media. My phone is used as a phone, for SMS, and Google Maps when driving to someplace new.
Other than my phone I have no "smart devices" in my home and doubt I ever will.
I'm not a Luddite -- I just see no personal benefit in any of the things I've mentioned.
I'm not convinced that an exponential number of people working on AI produces exponential advancements. Wouldn't we see diminished returns with each new person, presumably each one is less capable and less expert? I see AI experiencing a hype-cycle like disillusionment before seeing "double exponential" returns.
Typically, diminishing returns means f(x) = x/(1+log(x)) where x is headcount and f is output. The overhead goes up with the log of the number of people, because a hierarchical organization will have log(N) layers of management that need to be traversed to make decisions.
If we have exponentially increasing manpower, x=exp(t), then f = exp(t)/(1+log(exp(t)) which is O(exp(t)/t)
I suppose it depends on your view of the theory of scientific revolutions.
[0] https://en.wikipedia.org/wiki/The_Structure_of_Scientific_Re...
Look up for example Yann LeCun's AGI presentation. Now see projects like Ogmaneo or recent Deep Mind papers that do fast online learning or have ways to work around catastrophic forgetting and other issues raised by LeCun.
I believe that one straightforward way to get there is to continue to apply neural network research towards attempting to mimic animals. And there are a bunch of people doing that and making progress.
That is harder than you might think. For example, the simplest animal with a nervous system, C. elegans worm has been simulated "only 20 to 30 percent of the way towards where we need to get" according to one of the OpenWorm project leaders. That's 302 neurons total.
Using a computer analogy, it's like struggling to build an abacus when our ultimate goal is to build a Core i7 processor.
rather than any totally new approach
It's not clear to me that we have any approach to achieve AGI. People like Ben Goertzel have worked on that for decades without much to show for it. Current deep learning methods have little to do with AGI, they focus on very narrow applications, by design.
As long as it stays like that (and by opening up new sub-areas, it could stay like that for quite some while), adding more somewhat qualified people to the field results in all the useful combinations being discovered faster and thus exposing new potential starting points. Probably no exponential growth in the strict sense, but quadratic growth isn't bad either.
[edit - why am I still editing the spelling and grammar on this post?]
Personally I believe we will see really interesting animal-like AGI demos in 2018 and 2019.
It's just dumb that people can't take this stuff seriously until all of the engineering and research is 100% done.
When would this start? Probably not limited to the pace of AI but the pace of human computer interfaces catching up. I suspect the largest increase of usage will be the non-invasive helmet electro-magnetic style that is already proven to work to some degree.
Generally too, HCI can help us create a massive amount of training data.
Speak for yourself. I doubt Sam meant to appear ignorant of other perspectives on the world. However it would be nice to consider them when writing general insights, as opposed to simply tunneling his own point of view.
This really reads like self-satire.
It's just silicon valley bubble platitudes top to bottom. He needs to take a breather somewhere on the other 99% of the planet.
And at that point it's game over for homo sapiens.
The protocol that that we need is for transmitting information, not data. Or in other words, you don't need a neural interface to transmit understanding.
Cybernetics refers to this as a metasystem transition:
https://en.wikipedia.org/wiki/Metasystem_transition
(Shameless plug: if you want to see what it looks like when a hivemind speaks, check out https://reddit.com/r/AskOuija)
This seems to imply sama is resisting the inevitable merge. Can't we instead try to steer it in a more positive direction? Where are the startups trying to keep you addicted to the internet in a way that improves your life? They don't seem to exist, because the incentives aren't right -- it's way more profitable to keep you hooked in ways that make your life worse. How do we fix those incentives?
> consider the gamifying elements of Khan Academy, Duolingo and other e-learning sites for example
It's worth noting that I've heard a lot of criticism of e.g. Khan Academy from certain components of the mathematics education community. Concretely, they feel that the material there is designed to increase engagement and provide a sense of accomplishment ASAP, and that the learning processes that fit those goals are detrimental to development certain mathematical skill sets. I.e. you might learn how to compute derivatives really well, but at the expense of really understanding limits or the fundamental theorem.
Mind you, mathematics educators have always been an internally divisive bunch, so take from it what you will.
The term "dominant species" really jumped at me as I was reading this piece. It's so fantastically vague that it begs the question - would us and AI have the same definition for what "dominant" is?
We're not building AI individuals with self-preservation instinct, hunger for resources and sexual drive. We're building super-individual systems like Google, Facebook and trading systems, that can be much more intelligent than we are but also have vastly different built-in "purposes" (just as evolution have "built-in" purpose of eating, having sex and caring for our family into us).
I think that in the end, the thing we're building will end up much closer to Solaris ocean than Terminator.
There is no other general intelligence at this point that can rival human intelligence, we have a systems sample size of 1. Making predictions based on such a wide range of possible technologies is hopeless at this point.
Why would a fixed placement general learning computer system that takes input from millions of possible sources evolve in the same way that a mobile robot with general learning intelligence? Will both be possible? What about swarm intelligences?
Too many questions with no possible way to answer them.
Our "successful" descendants will probably be the unmerged, low-tech survivors living on the outskirts of a new "Machine Age" civilization.
He's referring to a superhuman artificial intelligence, not merely an augmented human intelligence.
The basic idea is how do you effectively align a general intelligence’s goal with humans so that it uses its intelligence to solve problems in a way that aligns with our own human utility functions.
If humans figure out general AI before goal alignment it may have outcomes we don’t want.
http://lesswrong.com/lw/ld/the_hidden_complexity_of_wishes/
https://youtu.be/EUjc1WuyPT8
A self improving general AI may just be be a matter of an intuitive leap, but nothing that’s hard to turn on once you understand it. This property would make it hard to suppress.
The upside is that if we can figure out the goal alignment problem we may be able to fix the coordination issues humans have and have a much better end state than all of us burning out and going extinct on our planet.
The downside is some variant of the world turning into paper clips.
http://slatestarcodex.com/2014/07/30/meditations-on-moloch/
https://wiki.lesswrong.com/wiki/Paperclip_maximizer
It is a hard problem though which is why there's all the talk of existential risk. It's also a little nuanced to explain why there's concern or why people should work on the problem.
I thought this post from yesterday was a good response to someone stating that an intelligence explosion was impossible (the original article it's responding to is linked within it): https://intelligence.org/2017/12/06/chollet/
On the defensive side I think there's a lot that can be done with narrow AI. Consider using really good penetration testers and bug finders to fix Internet security.
For any given task, build a tool that just does that. It's like the least privilege principle in computer security.
These ideas are mainly extreme proof-of-concept ideas, maybe there's ideas along these same lines that are much easier to pull off, but maybe these ideas as-is would be do-able for a possible superhuman intellect. I'm kind of thinking about what it looks like when individuals meticulously plan and then make tool-assisted speedrun videos through videogames, and then scaling it up to what it would look like if thousands of people worked together optimally for thousands of years brainstorming and calculating on how to speedrun through conquering a society. Physical limits permit this kind of computation as far as we know. Maybe this imagery is off-base, but I'm purposefully shooting high and worrying I'm not shooting high enough. Chimps would never even consider the possibility of "super-chimp" intelligences creating nuclear weapons. You can't make narrow AIs to defend from these possibilities if half the danger is our inability to even enumerate them.
Perhaps work on AI goal alignment can help us build a more humane market place, or vice versa, research into macro-economics can inspire some ideas in AI goal alignment.
It's just that the values and goals of many humans are unaligned with the values and goals of many other humans.
Which results in a cacophonous marketplace.
With apologies in advance for the self-promotion, here's a paper where I present my arguments for alternative scenarios:
https://arxiv.org/abs/1609.02009
In short, I argue that non-evolutionary superintelligences are not stable (they eventually go inert), while evolutionary superintelligences are a very serious existential threat to our species.
Still exponential, though.
Also, as a general criticism, there's a big difference between people getting addicted to the internet, getting dumbed down by it and sucked into things like a youtube hole, and my idea of The Merge. The things you describe sound more like an automated soap opera or opiate addiction than the singularity.
I smell epicism.
How did it become chic for women to smoke cigarettes back in the first quarter of the 20th century?
The one avenue open to tech advancement became so powerful that it eventually began to bleed into the physical world and we are starting to see other tech slowly advancing again. Maybe if the tech wealthy of Gen X and later have sufficient power in society after the Boomers pass the baton (if they ever do), we will let the technology keep advancing and see this merging. There is also the fact that the US is no longer a hegemon. Good luck stopping people in China from doing things banned in the West.
What do you think ISIS would look like in the world you describe?
So uh yeah, your argument is pretty insane?
Fuel in nuclear reactors do not have to be a weapons grade levels and only a few research reactors are (or were, as they are being phased out).
Also, one or two (or a dozen) of small nuclear bombs set off by some small group is not going to wipe out humanity and most of the worlds ecosystems. Horrible, massive death and destruction for sure, but not an existential threat to life on the planet.
That's not true at all! Civilian nuclear power in the USA (and most of the west) has stalled out since the 70's, but nuclear technology has seen plenty of advances over the intervening time frame. See esp. modern nuclear propulsion systems. Super impressive.
Also, this isn't a problem caused exclusively or even primarily by safety regulation in the nuclear sector. The evidence is in the failure of nuclear power throughout the world, despite significant variations in safety regulations.
In fact, if you look at root cause analyses for the failure of nuclear energy, you'll find that lack of regulation is a major reason for the failure of the nuclear energy. If the FF industries had to pay for their externalities, nuclear would be extremely viable (at least in the 80s-00s; now it'd have to compete with solar and wind).
> Computer technologies were only allowed to progress so quickly because people did not think computers were dangerous.
Again, this is a pretty wild assertion... Moore's law >>>> regulatory environment. Seriously. If the output of nuclear power plants had grown exponentially for multiple decades, we'd be in nuclear-powered paradise.
"Moore's law >>>> regulatory environment"
I don't see how this is a disagreement with what I was saying. Imagine if society had the same amount regulation on building software systems as dealing with radioactive stuff. Moore's law would not exist.
"If the output of nuclear power plants had grown exponentially for multiple decades, we'd be in nuclear-powered paradise." Why didn't that happen? Lots of reasons, but I think it could have and still could given a chance.
I'm referring to military ships and submarines. I'm not really sure where the financial incentive for nuclear comes from for anything commercial?
TBH it seems like you have a culprit (regulation) in search of a victim (nuclear utopia), and as a result, you have a solution (nuclear power) in search of a problem. If that's the case, there are really many much better examples.
> Nuclear test ban treaty, etc.
The NTBT covers intentional detonation of nuclear weapons... I'll be the first to concede that regulation has perhaps hamstrung the private sector nuclear weapons market :-)
> engines for nuclear rockets
Radiation is a bitch and lead is heavy. Conventional rockets that we need for human space flight already can, with modification, get us to where we've wanted to go so far. So we've instead focused our resources on doing useful stuff once we get there. NASA doesn't have inf money.
This is something that people continue to research. What's holding it up is a lack of priority in decisions about what science to fund and a complete lack of any private sector market large enough to justify the investment, not any sort of regulatory barrier.
> nuclear excavation techniques
...I'm having an extremely difficult time thinking of a use case where this would make any sense. Nuclear bombs are more powerful than conventional explosives. That power has an enormous tactical advantage in wartime because you can take out a city with a single missile instead of weeks worth of bombing runs.
But that's not really a particularly beneficial feature in e.g. a commercial mining operation. Plus radiation's a bitch.
Help me out? What are the possible use cases here?
> nuclear airplanes
Radiation is a bitch and lead is heavy.
> I don't see how this is a disagreement with what I was saying. Imagine if society had the same amount regulation on building software systems as dealing with radioactive stuff. Moore's law would not exist.
This isn't clear to me. I think the hardware would've still be invented. Perhaps the business impact would've been smaller, but regulation of commercial software systems wouldn't have stood in the way of the enabling physics and engineering research.
Also, see the Ford quote about Microsoft; if the nuclear industry worked like the software industry, we wouldn't even be here to have this conversation. We'd have BSoDed our way to nuclear Armageddon some time around 2001.
> nuclear power plants had grown exponentially for multiple decades... Why didn't that happen?
Physics is a bitch. More precisely, I'm unaware of any serious conjectures by modern nuclear scientists that there are obvious advances on the horizon that could give us exponential improvements to reactor output even in the short term. Let alone over 3+ decades.
Aside from power, and perhaps inter-planetary travel, nuclear is a case of "tried it, doesn't work for fundamental reasons" or "wtf now you're just throwing nukes on things for fun". And in the case of power, we have a lot of data points from a lot of different regulatory regimes, all of which point to "this is a quite expensive way to make power which won't pay off until fossil fuel externalities are finally internalized... i.e., never"
Yes. No demand.
> Would you agree that fundamental physics does not preclude a functioning nuclear rocket?
Fundamental physics also doesn't preclude a nuclear powered ferris wheel the size of the empire state building.
There are lots of things that humans can do but don't do.
> A live, radioactive (low-level, but still radioactive) exhaust stream is not going to happen in the US in the current culture.
This hasn't stopped us from building an enormous fleet of nuclear ICBMs and a rather large fleet of nuclear power plants.
If you define intelligence as “being really good at chess” or “factoring prime numbers”, sure, computers are already smarter than us. If you define intelligence as “knowing when to let your child make mistakes on their own and when to help them”, or “knowing how to conduct an orchestra”, it doesn’t seem so extreme anymore.
In fact, the opposite statement rings just as true:
It is a paragon of human arrogance to assume that we will build things smarter than ourselves in every conceivable way.
The road ahead looks more like a planet with its ecosystems ravaged by resource extraction, with buggy computerized systems we don’t understand running people’s lives in harmful ways (eg see all the writeups about machine learning reinforcing systemic biases) than a world full of meta humans in symbiosis with inconceivably intelligent machines of their own design.
That can be tested and quantified, based on end state goals.
>The road ahead looks more like a planet with its ecosystems ravaged by resource extraction
In such a world, the savviest and more strategic 1% would thrive, just like they always do. Eventually though, they will need to merge with machines to continue making the cut. It's too advantageous not to.
I think this is a misunderstanding about human nature. Humans are thinking and feeling beings who are responsible to others. They are only slaves and automatons if they choose to; most aren't. Many who are have hints and feelings that what they are doing is unnatural.
I completely disagree. Most are. In fact we have study after study showing this the case.
You want to focus on the modern automatons, such as phones and social media, but we have plenty other examples. Focus on the clock. Focus on the law. Focus on societal and cultural norms no matter the negative effects that these things have. These things have driven us since antiquity.
If you did the same approach with a child that could fully comprehend the rules from the start, playing another child, both of whom had never played before, I really don't think they would have learned as much as the computer would have done. It would be an interesting experiment - my bet would be that the children had invented another game and were playing that instead.
Isn't there one posted to HN about every day?
> It can teach itself how to play chess, but it takes a very large number of attempts before it becomes good. The rate of learning for a human is still much faster than for an AI.
But this is just a power optimization versus availability problem. AGZ played 44 million games in 8 hours and became better than the best program which is better than the best humans. Optimizing for a human level of power usage doesn't seem like the best method at this point.