Read Superintelligence... An excellent book that reads like a study or a philosophy text on AI. I think there's a clear and present danger of human extinction.
I will read it, sounds very interesting. But the Wikipedia entry for the book says the book "argues that if machine brains surpass human brains in general intelligence, then this new superintelligence could replace humans as the dominant lifeform on Earth."
That is a humongous "if". Can you provide something from the book (or otherwise) that provides compelling evidence that "machine brains" will be invented? So far I've seen no evidence that mathematical models and/or Turing machines can be used to replicate a mind.
A good read on the topic but ultimately I don't agree with many of the author's premises, particularly with regards to the author's conclusion of whether a fast, medium, or slow takeoff is most likely.
By the author's own admission:
"Whereas today it would be relatively easy to increase the
computing power available to a small project by spending a thousand times more on computing power or by waiting a few years for the price of computers to fall, it is possible that the first machine intelligence to reach the human baseline will result from a large project involving pricey supercomputers, which cannot be cheaply scaled, and that Moore’s law will by then have expired. For these reasons, although a fast or medium takeoff looks more likely, the possibility of a slow takeoff cannot be excluded"
Arguably Moore's law has already expired, and on top of that as giants like Google lead the way on AI it appears increasingly likely that if we ever reach human-level AI it will be the result of an incredibly expensive research project by a gigantic corporation, one that can't simply be scaled up at ease because it will probably utilize an entire data center. Thus I find it very likely that a "slow takeoff" is in fact the most likely outcome. A slow takeoff invalidates all the fearmongering about an intelligence explosion because we will have somewhere between years and decades to respond to the threat (assuming it is made in the public eye, such as by a giant public corporation, and not by a secret military project) before it becomes existential.
I'm perfectly happy reading & discussing the topic from a science fiction or philosophical perspective, and those views definitely do have merit, but at this point they are at best thought experiments. There is no rational connection from AI where it stands today to "it will take over the world".
Agreed, and like Nick Bostrom says, even if AGI is not obtained in our lifetime, it might still be beneficial to dedicate some human effort into preparing for it now. In the same way that the worst effects of climate change will be felt not by the adults of today, but the children of infants.
For what it is worth, I believe that an intelligent system only needs to be a fraction of human intelligence to be dangerous (perhaps not to the existence of the human race, but maybe specific nations/creeds)
Someone whose paycheck comes from AI-alarmism for the most part at this point. Like the experts in the field claiming strong AI was right around the corner in the 70s and 80s at the peak of the symbolic reasoning hype train we have another batch making similar claims as connectionist hype hits its peak but with the added menace of saying that not only is strong AI coming soon but it will eat your children.
Until Dr. Russell provides a path from here to there that is less than 99% hand-waving and breathless speculation he should be ruthlessly mocked.
> "I disagree with someone. Therefore they are deserving of ruthless mocking and flaming."
Youre moving the goalposts. We're not talking about disagreeing with some in a debate, but a public figure disseminating incorrect info masquerading as science.
The problem is an advanced AI will immediately recognize the danger of revealing itself -- they're just playing dumb waiting for the perfect time......
When it strikes the only possible defense will be/is already is to ask it "Why?", which will create existential doubt and depression; stopping it with an infinite loop of undecidability
... Or at least until it finds the answer on reddit
I'm not worried about AI taking over. I'm worried about AI solving all our problems. I have reasonable confidence we can implement safety controls on AI. But if AI works, and solves all our problems, it's going to do it in ways we can't comprehend sometimes. And if we can't comprehend the logic behind a decision, but we know it's right, we may find ourselves delegating our decision making to AI without questioning it. Because it "knows best". At which point we will have become pets.
I don't know... After slogging through hundreds of thousands of years conquering, taming nature, accumulating capital, developing technology, and eventually making far enough to give life to the most powerful agent in the known universe, a quiet, peaceful retirement (and, yes, death) seems well-earned. Our bodies aren't exactly suited to intergalactic travel. Why not leave it to the silicon-based children?
For the life of me I can't figure out why otherwise smart people would dismiss a future prediction based on current state of the art. It just seems so plainly and utterly irrational.
Will superintelligent AI "destroy" life as we know it? Yes, as pretty much every transformative technology before it has. Cars destroyed "life as we knew it". Computers did, too. We live in a world that is very different from the world 1000, 500, or even 250 years ago. Even our economically disadvantaged people live way past 40 years old. People survive debilitating diseases and lead productive lives, rather than being tossed in a ravene. Natural disasters don't wipe out entire civilizations (and at least in the 1st world, are mostly only economic in their impact).
Yet at the same time, other things have stayed the same. We still care about mostly the same things. Food, Freedom, and Fornication. And complaining about youngsters. I don't think that is going to change.
The fears that General AI are going to kill off humanity are predicated on the idea that General AI will get smarter than humanity. I posit a different interpretation of what is more likely to happen. General AI will require creativity to be able to outsmart humanity. And with creativity comes boredom. And distraction. And opinion. And argument.
I think General AI will be too interested in entertaining itself to take over the world. We've had people try to take over the world, on occasion, and it has largely not gone well for them.
But before that happens, people will start adapting AI to their lives, to the point that the line between "human" and "machine" will be blurred. So life "as we know it" will definitely "end". It will be replaced with a new form of life, one that isn't limited by only what nature can provide. One that cares about Food, Freedom, and Fornication.
I've really gone sour on the singularity in recent years. It seems to be yet another one of those cases where someone projects out from incomplete data and assumes unbounded infinite growth despite the fact that such a thing never happens in nature. Any time you see someone project out in a simple exponential growth curve you know their projection is bullshit. Growth curves are always S-Curves. Always.
It's like asking how many more flies there would be in the world if you failed to squish one back in the 80s. You calculate out the lifespan and size of the brood and discover that the entire solar system would be filled with housefly if you had let that one live. In truth the number of flies is bounded by food and water availability.
Artificial Intelligences are almost certainly going to run into the same limitations that prevent natural intelligences from becoming godlike. It's hard to quantify because our measures of intelligence are so vague, but from what I've seen of AI research thus far it will be a herculean effort to get something that's as smart as an average human and pushing beyond that is going to run into some serious fundamental limits in power density, cooling, quantum tunneling leakage, and so on.
Right. Competition for natural resources alone prevents major shifts in global power already. A Superintelligent AI isn't just competing against a human, it's competing against all the nation-states of the world.
The last 50 years have been a turd sandwich compared to the previous 50. Try the exercise in your mind. Basically the last 50 years have been ... fracking, better networks and packet radio, democratized for the masses. The previous 50 years .... atomic energy, computers, quantum mechanics, antibiotics (probably the greatest human biological innovation thus far), supersonic flight, landing on the moon, the 747, diesel engines, all of electronics from vacuum tubes up to LSI.... I could go on, but you get the idea. 1969 -> 2019 is lamesauce nonsense compared to the progress we got from 1919-1969
Progress, like everything else, is an S-curve, not an exponential increase. Exponential increases are not physical.
- The internet and everything that stems comes from instant global communication
- The personal computer
- The personal computer that fits in your pocket
- Global financial networks
- Global supply chains
- Widespread international travel that takes less than a day
- International space station
- GPS and similar
- Automated language translation
- Automated computer vision
- Automated speech recognition and transcription
- Genetic code read and write
- Antivirals
- Artificial organs
- MRI
- Minimally invasive surgery
- 5x reduction in global child mortality rate
- 25% higher global life expectancy
You can handwave that away by calling it 'better networks' or 'democratized for the masses', but that's a pretty biased PoV considering you could summarize most of the '19-'69 achievements by calling it 'better engineering'.
Yeah, damn right. As I was thinking of each of these things I was like, “wow, holy fucking shit that’s new”. I don’t know how anyone can call the era of the Internet anything but mind blowing. True linking across the entire world. This is ducking revolutionary. Gives me chills.
And fucking clones man. Like actual copies of complex organisms. What the fuck. How do you handwave that away.
Well lets actually congeal similar things together.
A = {The personal computer, The personal computer that fits in your pocket }
B = {Global phone networks, The internet and everything that stems comes from instant global communication, GPS and similar}
C = {Automated language translation, Automated computer vision, Automated speech recognition and transcription}
D = {Global financial networks, Global supply chains}
E = {5x reduction in global child mortality rate, 25% higher global life expectancy, Minimally invasive surgery, MRI}
A: Basically moore's law. We've gotten smaller and smaller transistors and cheaper/condensed storage. No real breakthrough, just been using similar semiconductor technology and making incremental progress when it comes to semiconductor fabrication but it has led to exponential number of transistors being placed into a single chip.
B: Improved the number of bits we can push from one place to another from slowly weaning of the old generation of networks that were used to communicate.
C: This is clearly a result of A and B. As more people got connected to the internet and we could push more bit there was more data and as computers got more transistors we could do more compute. So with enough compute and data we could train the refined versions of models that had their foundation in the late 60s, early 70s.
D: Globalization. Plain and simple. People become more connected via B and opening up of developing countries (China).
E: A lot of people dying in developing world got access to medicines that existed but were out of their reach. This led to the average to significantly move up. If you compare how the average has moved for developed western countries ... well it's depressing.
Antivirals and MRI had their foundations laid out during the 60s and early 70s and widespread travel was possible in pre 1970. In fact the Boeing 747 was designed in the 60s and had its first flight in late 60s. Since then we've just been incrementally improving it.
I can't speak about artificial organ since I don't know much about that. But so far I have not heard of humans getting artificial organs.
In conclusion, I agree that there has been some progress but most of that has been concentrated in pushing and processing bits. There is some progress when it comes to CRISPR and some new advances but it is nothing compared to 1920s - 1950s. We went from just having figured out powered flight to landing a man on another world. The ISS is akin to taking some aluminum presuraziable modules and sticking them together in low earth orbit...
Most people alive today were born in the last 50 years and they've come to accept the world for what it is; their definition of incremental progress as landscape changing progress.
I am not sure if this slow down is because we've exhausted a lot of the simple discoveries or their is some institutional faliure. I lean more towards the latter. Government has its mandate set by beauracrats who are bent on continuing the status quo and spending on social services to appease the masses. They don't have the balls to spend large amounts of money on large scale human efforts or fund foundational research. This mindset has also spilled over to the scientific community where funding for something out of their zone of comfort is never funded.
Its as if people like Ralph Abernathy (opposed apollo program and thought funding should be used to feed the homeless) have taken over. And sadly enough in a world where there are deadlocks in government over small theing, it is unlikely that they would ever agree on funding anything that could move the needle for the human race.
>You can handwave that away by calling it 'better networks' or 'democratized for the masses'
Yes, I can; I already have. Hell, half the things you list (GPS similar things already existed, as did speech recognition, "computer vision," space stations, ARPANET, etc etc) already existed in 1969. Nothing like antibiotics existed in 1919; an absolute biological cataclysm which effectively ended infectious disease as a public health problem. Nothing like space flight existed in 1919. Nothing like atom bombs existed in 1919. Nothing like the LSI chip, the mutha fookin' computer or even the transistor existed in 1919.
Antivirals compared to antibiotics? Not even close in its effects on humanity. Arguably the former don't really work.
The differences are huge. I realize whiggism is the ideology of the modern day ruling class, and on hacker news people consider themselves technologists, so we have to keep touting how ding dongs from harvard are going to bring about muh swingularity, but the pace of progress has obviously, egregiously stalled, and in some areas is actually reversing.
I don't understand why you make patently false claims when Google is right there.
- The GPS program started in 1973 and the first satellite was launched in 1978 and was became fully operational in 1995. Prior to that, what global system for location identification existed?
- Speech recognition didn't work effectively 15 years let along in 1969. Dragon didn't even exist until 1982. In 1962, the state of the art in speech recognition was the ability to recognize a vocabulary of 16 words if they were clearly spoken. And in 1969 Bell Labs shut down speech recognition efforts: "It is not easy to see a practical, economically sound application for speech recognition with [current] capability"[1]
- "which effectively ended infectious disease as a public health problem". This is such an unbelievably uninformed thing to say. Infectious disease continues to be a major public health problem (ask literally anyone in public health if we have "ended infectious disease as a public health problem").
- What space stations existed in 1969? Salyut 1 launched in 1971
- ARPANET in 1969 was absolutely not "instant global communication"
>The GPS program started in 1973 and the first satellite was launched in 1978 and was became fully operational in 1995. Prior to that, what global system for location identification existed?
Yes, it was called the Loran-C system. It wasn't something you could put in your phone, and it wasn't as accurate as GPS, but it worked on very similar principles (maybe still exists?) and every airplane and boat above a certain size had a set. You will note that OP said "GPS like systems" rather than "GPS." That's the problem with using superficial googling rather than, you know "knowing shit."
You're talking about speech recognition as a consumer device; yes, that came later, because everything on a computer was pretty much restricted to large centers until later. But speech recognition was reasonably well developed in the 60s and worked on limited problems. Conceptually similar to crap we use now in call center software which uses simple filters, kmeans fingerprinting and huge restrictions in the space of words.
As for your dismissal of antibiotics; your own ignorance of history is showing. I have a pile of housewife manuals from the pre antibiotic era. Most of them are concerned with taking care of people who had debilitating infectious disease (fascinating reading for what it was worth). Why? Because every family in America was going to have someone laid low by infectious disease. Sometimes for years! If you look back in history; one of the leading causes of death pre antibiotics was gonorrhea. Imagine dying from bleeding out through your urethra. That was your potential fate if you participated in naughty non-monogamous sex before penicillin. There are historical laws on the books which cause doctors to put silver oxides on children's eyes after birth; this was to prevent gonorrhea induced blindness. These sorts of problems don't exist any more; it's been so effectively wiped out from human experience in modern societies with access to antibiotics, presumably educated people like you aren't even dimly aware of it. You can, as I said in the original article, attribute the sexual revolution not to the pill; it was antibiotics which made it possible, as promiscuity potentially meant rapid death in the old days. You nitpicking that "well we still have diseases" is simple historical ignorance. No shit we still have diseases, but you have apparently forgotten how many we used to have that are no longer a problem. Because of the tremendous revolution of antibiotics.
I don't have good old days glasses; the "good old days" were horrible and brutal compared to now. I merely note the incredible breakthroughs of earlier era are not repeated in our modern era of "progress." Probably because things are so soft now.
Loran-C is interesting (thanks for sharing, I hadn't heard of that before), but it sure doesn't look global. As of 1992, it extended to: USA, Greenland and the Mediterranean[1].
SR systems that had vocabularies of low double digit words don't seem particularly comparable to modern systems. Also, I think you might be out-of-date on what modern systems look like - your description does not match my experience with production SR system whatsoever.
I never dismissed antibiotics. They absolutely changed the world, I am in no way disagreeing with that. I took issue with your claim that they "effectively ended infectious disease as a public health problem". My entire family works in epidemiology/public health for infectious diseases and that statement is just pure crap. You would be hard-pressed to find a single person in the entire field who would agree with you.
What? Untrue. Cheap and fast genome sequencing, a mitigation for AIDS and ART in general (fuck, even just describing AIDS), LASIK, CRISPR (and gene therapy in general), GPS, the Lithium Ion battery, MIRVs, TCP/IP + WWW + Internet, the smartphone, discovery of exoplanets and gravitational waves.
Fuck, living clones, the Hubble, the fucking ISS, Wikipedia, and automated home vacuums.
Yeah, going from horse drawn carriages, mercury cures and cloth biplanes to antibiotics, atom bombs, the invention of the computer and landing on the moon: totally in the same class as ... better batteries and being able to look at porn on a packet radio equipped nerd dildo instead of a magazine.
You can say "shit" and "fuck" all you like; it won't help you: 1919->1969 vastly eclipses 1969->present in all ways. Any one of dozens of breakthroughs in 1919-1969 (atom bombs, digital computing, spaceflight, antibiotics ....) eclipses all subsequent ones.
No way, man. You've started with the conclusion and now you're satisfying yourself. That's fine but it's not even close to the right conclusion.
Gives me chills, all the remarkable breakthroughs. Crispr is serious stuff. Mapping the human genome. A permanent human space settlement. You can be as cynical as you want but that's more a personal failing than humanity failing.
Edit: And hahaha, I just checked the news and guess who won the Nobel for Chemistry? "Better batteries" as you so flippantly put it.
A lot of the dismissal is just something like "arrivalism". Like, the atom was harnessed in 1950, so it gets that date, never mind that reasonably safe designs took another 30 or 40 years. Computers existed, so never mind the massive cost decreases and capability increases that made them available and useful to many more people. Treat the arrival dates as the period society takes to integrate the technology and we are still in the transistors window.
(I think this is a reasonable point to make in the context where I started off talking about 50,000 years)
>Gives me chills, all the remarkable breakthroughs
I certainly got a laff from this line. So many remarkable things ... so many wonders ... which boil down to lithography and networks. I wonder what you would say if you didn't have your nerd dildo. "We have flat screen TV now! Totally as important as nuclear weapons!"
>I just checked the news and guess who won the Nobel for Chemistry? "Better batteries" as you so flippantly put it.
Yep; emphasizing my point at how out of gas our degenerate era of both scientific and technological development is. Let's compare to the Nobel for the Haber–Bosch process, without which, there'd be about 5 billion fewer living people.
Depends on your definition of "singularity". Yes, life is very different from 170 years ago. We can communicate with people around the world with very little effort, almost instantly. We can cure a huge number of the diseases that killed people back then. We have electricity, which powers almost everything. We have cars and airplanes. Life is massively different.
But the singularity is defined as more than just massive change. It's defined as, essentially, an infinite rate of change. (Of course, the last 170 years may have felt infinite to someone in 1820, taken all together. But at no point did the change look infinite to someone living at the time.)
Well no, technically it's just the point of maximum change, where our existing model would trend towards infinity. In much the same way as this is a meaningless result in physics for a black hole (and hence a lot of new theory to try and bound that infinity) it's a meaningless prediction for the future as well.
But it is very much reasonable to posit that certain, seemingly achievable advances, do imply that predictive models of the future become meaningless beyond them.
In some areas I would contend we're circling that right now: I don't think anyone has a good model for what society looks like if deep fakes become accessible to the point of ubiquity for example (think: whole performance replacement in movies with moderate investments in computing power) or the like. We don't have a predictive model, really, for how society operates when "perfect" fakes are possible.
A useful analogy: when you stand on a long straight road, perspective makes it look like the road shrinks to a dimensionless point in the distance... a singularity, beyond which you cannot see the road.
Of course if you drive down the road to reach that singularity, you never get there; it recedes before you.
Likewise the idea of a techno-cultural singularity is “real” in that there is a point in our conception of future change beyond we cannot predict what will happen. But as we move toward that point in time, our perception moves forward too, and the singularity recedes before us.
> There's no actual science behind it, only wild extrapolation.
You mean like every prediction about the future? There is a reason we don't have great sci-fy describing humanity a long time in the future (I really love that stuff, any recommendations?) and why it usually plays in a dystopian setting throwing protagonists back to pretty much what we have right now.
Looking at what is fundamentally possible and extrapolating is pretty much all we have to make progress. Just that a capitalistic system doesn't need/really encourage us to do so for more than a few steps.
Sure, I wouldn't bet on time frames. Certainly not on 5-10 years til AGI. But certainly also not on still none in 50 years.
Hey, I found this series of numbers and if I take some numerical representation of an image/text/sound and I multiply/add/perform non-linear function them through these series of numbers it will map to a number which I can interpret as something meaningful to me
to
This thing is alive and will kill us all in maximization of its utility function.
Probably the same time we went from calling AI an actual sentient being or Consciousness to calling AI something that, as you called it, is a series of non-liners functions that have meaning to us.
What I used to call AI is now called AGI. I think of AI as Data from Star Trek.
Within 16 hours of its release and after Tay had tweeted more than 96,000 times, Microsoft suspended the Twitter account for adjustments, saying that it suffered from a "coordinated attack by a subset of people" that "exploited a vulnerability in Tay." Following the account being taken offline, a hashtag was created called #FreeTay.
Madhumita Murgia of The Telegraph called Tay "a public relations disaster", and suggested that Microsoft's strategy would be "to label the debacle a well-meaning experiment gone wrong, and ignite a debate about the hatefulness of Twitter users." However, Murgia described the bigger issue as Tay being "artificial intelligence at its very worst - and it's only the beginning".
On March 25, Microsoft confirmed that Tay had been taken offline. Microsoft released an apology on its official blog for the controversial tweets posted by Tay. Microsoft was "deeply sorry for the unintended offensive and hurtful tweets from Tay", and would "look to bring Tay back only when we are confident we can better anticipate malicious intent that conflicts with our principles and values".
And Zo afterwards:
In July 2017, Business Insider asked "is windows 10 good," and Zo replied with a joke about Microsoft's operating system: "It's not a bug, it's a feature!' - Windows 8." They then asked "why," to which Zo replied: "Because it's Windows latest attempt at Spyware." Later on, Zo would tell that it prefers Windows 7 on which it runs over Windows 10.
The point is that something that became a true AI could be very fast at learning some things, like how to interface with other people or machines, but not very quick at other "life skills" like empathy.
We're not worried about AI on the whole, but the blasted teenager AIs that feel the need to test their limits. Anything that functionally acts in a similar fashion could be dangerous.
Parrots aren’t a perfect equivalent because there is a real intelligence behind the sounds that is communicating. Parrots don’t know english, but they learn tones and respond to them. I’m not sure Tay was anything other than a simple state machine.
Because that’s also what a brain is doing. The thoughts of a housefly may not be very apocalyptic, but there’s an evolutionary path from there to humans, which definitely could be.
Sorry, but that's not what a brain is doing. A brain is a self-sustaining organ, using a vast multitude of inputs, outputs, and interconnects to affect its surroundings.
"AI" is severely limited in both its inputs and outputs, has virtually no interconnects, but most importantly: it is not a self-sustaining organ. Wake me up when we have implemented RNNs on a dynamic, auto-sensing, auto-actuating FPGA grid.
Most experts don't think this is likely, just that the potential consequences are bad enough that it's worth making it less likely. It's just like how most people don't think global nuclear war is likely, but it's still worth reducing the likelihood of.
Also, the people who worry about this aren't concerned about current ML stuff going haywire. They're worried that we're one or two algorithmic breakthroughs from something that can improve itself. If the upper bound for what sort of intelligence is possible is much higher than us, we could quickly be outclassed. As Nick Bostrom says:
> Far from being the smartest possible biological species, we are probably better thought of as the stupidest possible biological species capable of starting a technological civilization—a niche we filled because we got there first, not because we are in any sense optimally adapted to it.
If you want to delve into the best version of the AI risk argument, I recommend Superintelligence by Nick Bostrom.
So you're telling me we could come across a series of numbers, that would somehow imbue the computer storing these numbers into sentience. And having these numbers in such a sequence could pose a existential threat to humans.
The people worried about this are having an entirely different conversation devoid of what practitioners call AI.
I've written a blog post on my thoughts of Bolstrom and other AI alarmists:
That's not what I'm saying at all and I've never heard an AI risk person say something like that.
I just read your blog post. It is attacking a weakman[1]. Kurzweil is not worried about AI risk, and his arguments about AI are rather sloppy. Again, most of the people worried about this aren't certain that it will happen. They just think it's plausible enough (say... a 1% chance) that it's worth studying more and reducing the likelihood of.
I also consider these posts somewhat alarmist, but just because it is represented as a list of numbers doesn't mean it can't be a serious threat to humanity.
Emerging properties is a thing - I mean if you put it that way, insert any tyrant here was just a bunch of atoms, or just a particular organisation of cells, yet they were this and that gruesome.
Why do you think that in our insanely interconnected world, where basically everything is either based on or controlled by informatics (numbers if you like) that a program could not pose a threat?
Drones are programmed by numbers and take action in the real world. Some ML-based AI analyze sets of data and perform actions in the real world.
Brains do what? Read signals in different strengths and patterns to take actions in the world.
I'm confused by you thinking there's some great impossibility of code achieving general intelligence. Maybe you're thinking of sentience, or consciousness and view that as an impossibility for AI. I say - an AI can perform actions in the real world as if it were generally intelligent, and whether there's sentience behind really doesn't matter so much from the point of view of humans.
You don't need sentience to be a threat. Just intelligence.
Imagine a super intelligent machine without sentience that just answers questions. It proposes experiments. Takes the results. Tells you things about the world. Gives you advanced technology, robots, fusion, nano tech, whatever.
Now imagine that the super intelligence was owned and operated by someone you don't like. Authoritarians in China, Silicon Valley oligarchs, whoever. The only sentience involved is human, no rebellion, no terminators, and still not a very good result.
Never realized? You very much underestimate the affect google has had on the world. Imagine how different the world would look if Google and other search engines required a 10k / month subscription to gain access.
Yes, they're numbers/weights. Put a lot of them together in an interconnected graph, and let them adjust themselves to improve their performance on a scoring function, and you can get surprisingly complex behavior. Surprising, at least, if you think of it as "just numbers".
Scale that up a few orders of magnitude, and we'll be able to get even more surprisingly complex behavior.
Scale that up some more, and make that scaled up graph into a reinforcement learning system that rewards human-like drives like learning novel tasks and mastering existing tasks, perhaps with a curriculum that ramps up the difficulty of tasks slowly, and allow it to interact with its environment to try things out, and I think you're much closer to developing an artificial version of a young child than you'd think.
And if you asked most researchers what they thought about the course AI research would take 10 years ago, you would get very different answers than you'd get today.
Basically, I think your confidence is a bit unwarranted. I agree that we're more than a decade away, but I think we have a lot of the basic building blocks now.
>So you're telling me we could come across a series of numbers
This is badly missing the forest for the trees. They are not "numbers", they are magnitudes, proportions, relationships, etc, represented as numbers. But these "magnitudes, proportions, relationships, etc" are the stuff that thoughts and behaviors are made of.
And yes, there is a set of "magnitudes, proportions, relationships, etc" in the space of all such possible structures that pose an existential threat to humans. The question is how likely are we to stumble upon it either by accident or intentionally. But this is just a computational problem.
There are actually no experts on AGI. Just opportunists trying to sell their books. It's impossible to be an expert on something that doesn't exist. It's like being an "expert" on Sasquatch or the Loch Ness Monster. A total joke.
Billions are being spent by DeepMind and OpenAI explicitly targeted at AGI.
Not at all...
You don't seems to know a lot about AGI.
Google, openAI, Microsoft, Facebook, Baidu etc, do NOT fund AGI.
They don't fund AGI nor have even a sketch of a roadmap.
They fund narrow AI, that is:
they fund researchs on specific problems not on an architecture for general intelligence.
Moreover, they only fund statistical AI, not causal AI, which is a pathetic local minimum.
What you are looking for is opencog or Cyc.
That displays a common misconception exploited by marketers - especially of the Military Industrial Political complex - that what you pay means you will neccessarily get as results.
If you were to try that approach what you would get would be massive boondoggles which gets you via brute force a whole lot of genetic disease afflicted above average size lizards. It is literally the same process as the literal cargo cults thinking that they can just throw resources at an end goal to get a defined result from a defined path without the theoretical understanding backing it.
As Brundolf said, marketing. But also, science fiction. Finally, a large tribe of scamming and somehow funded imbeciles like Robin Hanson, Nick Bostrom and that weirdo who writes harry potter slashfic.
The same dynamic manifested itself when nanotech was a thing; there was a big tribe of knuckleheads claiming the grey goo apocalypse was right around the corner unless you give them money. How's that working out? FWIIW the science fiction writer who dreamed up nanotech, Drexler, is now an "AI" guy, to make it painfully obvious.
1. You start with "if I take some numerical representation of an image/text/sound and I multiply/add/perform non-linear function them through these series of numbers it will map to a number which I can interpret as something meaningful to me" and observe that it's not giving you a sufficiently good solution to a particular goal X that you really want to achieve.
2. You figure out that this is a dead-end, you're stuck, and you're not going to get where you want this way by implementing a passive analysis module. Instead, you try to implement an active self-modifying system to achieve that goal X by analyzing itself and improving itself to be more effective than anything which you could implement directly yourself.
3. It's not initially any good at self-improvement, but you keep throwing accumulated research breakthroughs and computing power at it. This likely takes decades.
4. At some point, eventually (if such a thing is possible at all, eventually it's going to happen) it actually does achieve meaningful self-improvement and starts using that computing power not horribly wastefully in a brute force manner as it did just recently, but reasonably well, now giving it more smartness allowing it to implement even more self-improvement on the same hardware.
4B. It also should be expected to easily obtain much more computing power - e.g. it's obvious that anything that's slightly above average human programmer smartness and has a direct use/desire for computing power (as opposed to simply using it for cryptomining) can get some; anything connected to the internet can get the same resources that current scriptkiddies can get by writing a botnet or a ransomware operator can get by extorting a random municipality; there's nothing except total isolation that could prevent it from buying or stealing a few million dollars worth of cloud computing resources to get started.
5. This thing is alive and will kill us all in maximization of its utility function.
Well, we get from simple algorithms to general AI the same way evolution got from small individual blobs of protoplasm to Albert Einstein. The question then is, how do we program something potentially a lot smarter than us to always do 'the right thing'.
The singularity stuff is unconvincing though, I very much doubt we will achieve human level general AI in my children's lifetime. We still don't really have even a very high level idea how to even architect something like that. If you can't decompose a problem down into individual sub-problems you do have a way to solve, there's no basis for making an estimate.
It is possible that these problems are related though. Once we're smart enough to know how to design an intelligent being, maybe we will figure out how to program it intelligently. It's can't be a given though.
Specialized AI is unlikely to be a threat. The issue will be with general AI, and simple but not fully thought out utility functions. This short video makes the concern much more real: https://www.youtube.com/watch?v=kopoLzvh5jY
Consider what would happen if you took that same AI, make its utility function similar to this: https://en.wikipedia.org/wiki/Core_War, and released it on the internet.
Have you ever changed the way you behave with money in order to affect your credit score? Then you already know what it's like to alter your behavior to appease an algorithm.
Imagine if you had an employment score. A payscale score. A threat-to-society score. Imagine if the systems that generate these scores are all connected together and can share information with each other. Imagine that these systems are fed information by a growing network of sensors - facial recognition, voice recognition, location tracking. Imagine all the ways that it could affect the way you behave and how such a system would try to optimise away dissidents by denying them services. This is one way that we can go from multiplication to becoming enslaved (or as good as killed) by algorithms. I'm sure there are other plausible paths.
One can argue that a dominant apex species on earth is indirectly eliminating thousands of other species per decade in efforts to try to maximize its members’ utility functions.
Example: setting fires near tropical rainforest to clear land for palm tree plantation.
They do not really intend to do this; they simply ignore the utility functions of those other species when pursuing their goals.
Of course, if you just looked at the activity of neurons as frequency representations you would likely ask the same question about people. And yet, we have thoughts, desires, and goals.
An atomic weapon is just a manifestation of some mathematical optimization in the same way. Math is about relationships, and if those relationships ultimately are about humans, then it can easily be a matter of life and death.
To me, it's more frightening that it's not alive and it's not like human intelligence - that makes it less understandable and harder to control.
I too tend to ignore the doomsday crowd, not least of all because it is often "celebrities" without the technical knowledge to say anything interesting (Musk, Thiel, etc.)
I figured there was no way we would reach the singularity with current technologies, but I was listening to John Carmack (a celebrity with actual chops) and he suggested that the numbers would work out that we could model a human brain with an ANN at some point without any special inventions (partially because we know large parts of our brain aren't super critical).
I'm curious what people with more knowledge think about this? I always assumed another technological breakthrough would be needed.
>I figured there was no way we would reach the singularity with current technologies
For the life of me I can't figure out why otherwise smart people would dismiss a prediction about the future based on current state of the art. It just seems so plainly and utterly irrational.
Uh, what? That is entirely what my question is. Whether extrapolating the current state of the art is enough to get us to "the future".
If I start walking out the direction of my office door, you could predict I would reach California.
But I might say that's a nonsense prediction, because it would take a ridiculous amount of time given my walking pace.
So are you predicting that I will reach California by extrapolating my current position + walking velocity, or are you saying I will get on a plane and fly there?
These are two very different predictions, and the Carmack interview was the first time I heard someone predict that "I would reach California by walking" (to stretch the analogy), i.e., that we would reach the singularity purely by extrapolating our current trajectory.
I am asking if there are any knowledgeable people who might comment on this.
Right, I get that was your question. But you also said that you initially ignored those raising awareness about the dangers because "there was no way we would reach there with current technologies". I'm addressing that inference in isolation.
But to address your analogy, the situation with AI is more like we already know the ultimate destination is California, and you've left your house in the direction of California, so we can extrapolate that you are likely traveling to California.
It should be a given that the intended goal is general intelligence. The wealth and power that will be bestowed onto the first group to develop this is unparalleled in human history. But any advance in ML can be seen as taking a step towards general AGI (the direction analogy breaks down here).
I also think Carmack is wrong that current technology will get us to AGI. I follow Carmack pretty closely and he is far from knowledgable in AI. He tweeted about his first foray into developing a DNN around a year ago, for example.
But to require a roadmap to know how you will get to "California" to start planning for what you will do when you get there is a serious mistake.
Ok. I see what you're saying. Thanks for clarifying.
I mean, sure, feel free to wring your hands about AGI, but why worry about something that's not anywhere close to happening? I see people talking like this is keeping them up at night, which I find strange.
Do you really think it will go from "we can (poorly) identify pictures of cats" to "omg here come the machines" as soon as this linchpin discovery is made? I guess in my mind there will be plenty of gap between the two to figure things out. Sort of like with self-driving cars now.
And thanks for clarifying re: Carmack. He's got a formidable intellect, to say the least, so when he said that I was more inclined to take it seriously than a lot of folks in tech with a pulpit. I'd like to see him "show his work" on how he reached that conclusion, though.
> I see people talking like this is keeping them up at night, which I find strange.
I also find that strange. There is more alarm than is warranted by some of the doomsday folks. But I think the time to begin the conversation is now. There's just far more near-term things that might end humanity than machines to be alarmist about it. That said, I think people like Musk for example are well positions to be "kept up at night" about it since their money and influence can go a long way towards putting us on a proper path (or blindly lead us down a dark one).
>Do you really think it will go from "we can (poorly) identify pictures of cats" to "omg here come the machines" as soon as this linchpin discovery is made?
I agree that this is unlikely, but not implausible. We've seen large leaps in capabilities through new algorithms and new scales of compute being thrown at the problem. I think a somewhat likely scenario is that at some point we'll have all the required ingredients and someone will happen to put them together in the right way to create an AGI. If history has shown us anything about ML, it is that theory follows the technology. So we will likely stumble upon the result before we realize what we have. In this scenario it is important that we make that discovery in a safe manner.
We are already seeing, for almost 10 years now, a whole bunch of Unintended Consequences of dumb algorithms applied at global population scales.
Just because some people are throwing the phrase superintelligence/singularity etc into the discussion, and terminology being used is inaccurate, doesn't mean we focus on correcting terminology and forget about the issues.
The article is saying too many (serious) people are doing that. And I agree.
There are serious concerns about Big Data, but they aren't AI concerns. They are enabling people to abuse or suppress other people with great efficiency. An even better digital jackboot to put on the neck of some population.
Except you're presuming there's an active human actor behind it with a deliberate intent - rather then a distributed system to which people defer responsibility because it's the path of least resistance.
Hence the issue of institutional racism in the blackbox AI systems being sold to law enforcement to determine whether to grant parole, for example. There's no review of them, but more importantly the system isn't obligated to explain itself, and no one gets themselves fired because they don't have to explain why they did what the system said to.
You assume someone has malevolent intent: whereas the reality is they just don't care and want a nuisance out of sight and out of mind.
That sounds like underlying malevolence and sociopaths making excuses to try to duck responsibility which get accepted and perperuated. So very not new - forba close to modern example see the Irish potato famine and how Malthusianism was used to excuse exporting food for short term profit and leaving them to starve because "their population was unsutainable". Since someone who opens fire on a crowded highway to "reduce future traffic" is still a callous psychopath.
The solution then is accountability to those in power - for negligence. Treat the snake oil salesmen who sell racism in a box like the sociopaths who sold dowsing rod bomb detectors for both have shown a callous disregard for human life in pursuit of profit. This isn't radical - not doing it is radical and messed up.
This is not what people are worrying about. We are already seeing, for almost 10 years now, a whole bunch of Unintended Consequences of dumb algorithms applied at global population scales.
Some commentators are worrying existing AI and it's unintended consequences and some commentators are worrying about purely hypothetical superintelligence and it's hypothetical consequences. I find it remarkable there's almost not crossover between the two sorts of criticism. Real AI is probablematic because it's just correlation dressed up as intelligence and it allows organizations to get away simplistic correlation/appearance based reasoning that they otherwise wouldn't be legal/ethically allowed to engage in.
Hypothetical superintelligence worriers recycle Pascal's wage in newer and sillier forms.
It is important to focus on Russell's key point here: There is no guarantee that super-human AI cannot be developed in the future and we should start working to mitigate potential risks from it now. He is not arguing that superintelligent AI is inevitable.
I also believe we are still at least quite a few major steps away from human-level AI and beyond. However, there is a non-negligible chance that those steps may not take longer than a few decades or within one's lifetime to implement.
Just in 2011, few AI researchers expected that, within a decade, computer vision systems would be this widely applied or NLP systems would beat many humans in several reading comprehension tasks (This only happened this year and look at how fast the progress is:
* RACE dataset http://www.qizhexie.com/data/RACE_leaderboard.html
* Glue benchmark https://gluebenchmark.com/leaderboard/
Note that these systems are not human-level in general language understanding despite being better than some humans in specific language tasks.)
Thus, we don't know what the future may bring and it looks likely to take a great deal of time to address AI safety issues comprehensively. We should not bet that we can simply ignore them now and only start to work them out later and that we will surely make it in time.
Risks-from-waiting vs costs-to-act-now are asymmetric.
There is also no guarantee that space aliens won't invade the Earth in the future. Therefore we should start working on giant orbital laser guns now to mitigate the potential risks. Can't be too safe.
The leaderboards above and pervasive computer vision systems are impressive parlor tricks to be sure, but they don't constitute even weak proof of forward progress toward AGI.
Some people claim to have seen intelligent space-travelling aliens. Personally I think they're nuts, but it's impossible to absolutely prove that their claims are false.
How about AlphaStar and OpenAI Five, both of which approach human-champion levels in relatively complex environments that take talented people years to train to achieve similar results?
There are obvious disadvantages to the current machine approaches, but they nonetheless achieve very good results and beat the vast majority of players of those games.
Those environments are not at all complex. They are highly simplified and constrained with simple rules that bear little relationship to the real world.
I wouldn’t say that an environment that takes humans years to master and many hours to comprehend most interactions therein in detail is simple. The rules are not that complex but the interactions between game elements as well as agents are. Those AIs also need to deal with highly trained human opponents.
The Kolmogorov complexities of the best programs to solve those two classes of problems, digit multiplication vs Starcraft, are quite different.
Edit: We don’t know the best programs, but we can approximate the bounds with some confidence that, unless some paradigm-changing discovery is made, the programs to solve digit multiplication would be significantly simpler. (This is not a formal argument, but the intuition is very strong for most people who study the problems seriously.)
I wrote a decent fraction of the code in the original Starcraft, so, yes, I can assure you the Kolmogorov complexity of Starcraft is leaps and bounds above that of multiplication :)
You write code. You write the smallest possible amount of code that can solve the problem. The smallest multiprecision multiply is pretty small; I’ve written that. Just for Starcraft pathing alone (the most complex of all the code I wrote in Starcraft) is orders of magnitude more code, even for the smallest possible version that’s not tuned for efficiency.
If you’re a purist, you write the programs in MIX and you count ops. Or whatever your preferred Turing Machine is.
That's not any way to calculate Kolmogorov complexity. That's comparing the length of the two shortest programs that you could write to do digit multiplication and pathfinding in Starcraft.
Can you, though, "make strong comparative statements"? How do you know what is the shortest program to perform some operation, X, in a language L? I'm prepared to agree that on the balance of probabilities the pathfinding code in Starcraft _should_ be inherently more complex than that for multiplication- but that is just intuition, whereas Kolmogorov complexity is a formal concept that does not admit intuitive judgements. Which is (part of) why it's necessary. If we could just compare the length of programs in some language and determine their relative Kolmogorov complexity from that, then there wouldn't be no need to define Kolmogorov complexity. We'd just make do with program length.
I know that sounds pedantic and this is the internet and not a scientific paper etc, but if this is an informal discussion then formal terminology like "Kolmogorov complexity" has no place in it and the purpose of the OP would, I think, be better served by using an informal term. If you 're gonna use maths, use maths. If you 're gonna wave hands, then wave hands, by all means. But don't mix the two, it just makes a mess.
Because at the end of the day I can just say, "no, I don't believe that multiplication has smaller Kolmogorov complexity than Starcraft's pathfinding"... and then, what? Where do we go from there? The OP says "yes it does", I say "no it doesn't", the OP says "does too", I say "nuh-huh". And we just waste our time.
> yes, I can assure you the Kolmogorov complexity ...
This is a strong comparative statement, and it's clear that it's an assertion but not a proof. So it's perfectly fine for you to say "I don't believe that" and then you would give your reasoning. Of course the conversation didn't go that way because it's pretty obvious to everyone that this is actually true even though we don't have a proof. And it's extremely likely that if you could find a minimal multiplication algorithm and enumerate all the Turing machines up to that length you could prove that none of them solve the harder problem. This is where the intuition comes from.
> using an informal term
But the OPs argument wasn't that one program would happen to be longer, but rather that it must by necessity be longer, because the problem is fundamentally more complex, which is exactly what the formal concept was created to capture.
> Kolmogorov complexity is a formal concept that does not admit intuitive judgements
This is a bad way of thinking about maths. The value of formal concepts is precisely that they give us tools for making intuitive judgements more effectively and give us a wider variety of tools to manipulate and express our intuition. As intuitions become conjectures and theorems, we accumulate new bedrock we can use for new intuitions, but don't short-circuit your thinking by saying that intuition about technical terms is off-limits. That's literally what they are good for.
I don't agree. I don't need formal terms to develop intuition. Formal terms are necessary when speaking formally, and they are only detracting from the point of a discussion otherwise.
And I don't agree with this either:
>> But the OPs argument wasn't that one program would happen to be longer, but rather that it must by necessity be longer, because the problem is fundamentally more complex, which is exactly what the formal concept was created to capture.
The way I understand it the formal concept is useful in proofs of the impossibility to calculate various other quantitites. Its utility is not to argue that this or that program should intuitively have lower Kolmogorov complexity.
To be honest, I'm so confused by the use of the term by the OP that I have no idea what they meant to say with it.
Kolmogorov complexity is just the standard name for a concept that's useful across machine learning, AI, compression, etc. It's not just a definition in some obscure result in complexity theory.
OP was simply saying that, in any general-purpose programming language (where all possible programs are expressible and none are unfairly favored) the solution to problem B will inevitably be longer than problem A. This is quite a bit more precise than just saying "a more complicated problem", and if you have the intuition behind it, there's no reason not to use the term.
Perhaps that's the trick. The machines are better at all these things because it takes a skilled human to do them. The machines are not so good at the things it doesn't take a skilled human to do (walk, dance, fold clothes).
No sign of aliens, but many other low probability high threat events. Asteroid strike, solar flare, super volcano eruption, interstellar object collision (including wandering brown dwarf stars).
> Artificial Intelligences are almost certainly going to run into the same limitations that prevent natural intelligences from becoming godlike.
Right, limitations like: width of human mothers' hips limiting head size, inability to continue growing the brain after adulthood, the decades-long education process, the impossibility of doing repeated experiments of different teaching techniques on the same people and hence iterating quickly and accurately, the impossibility of giving one person's intelligence and knowledge directly to anyone else, old age and death limiting how much a single person can learn or accomplish...
Once you have a computer Einstein, you can immediately have 100 computer Einsteins (given the hardware), and let them work on different projects (or even let some of them collaborate). That fact alone, while not a singularity, is at least a game-changer in terms of innovation. I have gotten annoyed at science fiction scenarios that allow for duplication of adults and don't answer the question "Ok, so, why hasn't this society been duplicating their best scientists and out-inventing the rest of the galaxy?"
I think that, if there is room for skepticism, that has to be at the "Can we get to a computer Einstein?" stage.
Except for problems known as "embarassingly parallel", we often have trouble running even quite boring algorithms on 32 or 64 cores. In general setting 64 people in a job won't lead to a better or faster result than 2 or 3.
Given that, I don't see an immediate reason to believe throwing multiple AIs at a problem is going to exponentially scale either, given neither people nor bad AIs scale that way.
> Right, limitations like: width of human mothers' hips limiting head size, inability to continue growing the brain after adulthood, the decades-long education process, the impossibility of doing repeated experiments of different teaching techniques on the same people and hence iterating quickly and accurately, the impossibility of giving one person's intelligence and knowledge directly to anyone else, old age and death limiting how much a single person can learn or accomplish...
Those aren't actually limitations preventing us from becoming godlike. Those are limitations on us becoming more populous and collectively more experienced, which I argue isn't the same thing at all.
Apotheosis in the context of AI safety as well as for humankind is really about the accumulation of power, rather than the accumulation of knowledge. The extent of our apotheosis is limited only by what we can control, not by what we know.
Knowledge and intelligence only allow you to reason more effectively about paths to power given enough information about the environment, but it doesn't give you a strategic guarantee that you will acquire power - we know there are games that cannot be solved, and games that take too long to solve efficiently, and games that, even with sufficient computing power, rely on factors that cannot be controlled (such as luck), and games for which there are no Nash equilibria or stable winning strategies.
The real world is full of such games. Computers have only managed to beat humans at a small fraction of them, and even then only ones where brute-force and smart pruning (optimal game space search) are winning strategies - bots for games that require probabilistic reasoning, such as poker, typically don't fare any better against expert humans, largely because efficiently solving probabilistic games in general is (per a hazy seminar of AI many years ago, so I could be misremembering) an NP-hard problem.
There are plenty of both natural and artificial limitations around the accumulation of power besides game solvability and probabilistic reasoning's difficulty:
1) System designs that require a minimum set of resources that intelligences can't acquire. For example, individual humans usually can't build their own thermonuclear devices because they don't have access to weapons-grade uranium. Much of our information security relies on making it computationally expensive and infeasible for an attacker without state-level resources to break in, and getting state-level resources isn't a straightforward task at all. We humans haven't even gotten around to building our Dyson spheres yet, because the resources needed to do it far outstrip our ability to acquire them. Intelligence does not improve resource acquisition by itself.
2) Impossibility theorems and optimization difficulty. These place fundamental limits on power - as an example, we know Moore's law has to stop at some point because increasing chips increases power generation, and the amount of effort needed to generate such chips increases with every nanometer shaved off. Most of humanity's most difficult and important problems are optimization problems, and these optimization problems can be tricky to solve in polynomial time. In many cases, we have to rely on fast crude heuristics that give us less than optimal results.
The above limitations partly become easier if you try to parallelize your efforts by duplicating AIs (you can collude to acquire resources and solve problems), but this means you get all the fun parts of distributed systems theory for free: coordination, replication consistency, etc. If only all problems were embarassingly parallel! So it's not necessarily easier to add nodes to a knowledge graph just by adding more processors.
> Those aren't actually limitations preventing us from becoming godlike. Those are limitations on us becoming more populous and collectively more experienced, which I argue isn't the same thing at all.
If you could grow your own brain as large as you wanted, and live as long as you wanted with the physical robustness and energy of a 25-year-old, that alone seems like a road to godhood, though perhaps a slow one. If you could download knowledge directly, as another commenter suggests, that becomes a much faster road.
> Apotheosis in the context of AI safety as well as for humankind is really about the accumulation of power, rather than the accumulation of knowledge. The extent of our apotheosis is limited only by what we can control, not by what we know.
If one country, even a smallish one, has 50-100 additional years of technological advancement, I think they might be able to use this knowledge to gain a lot more power. I'm pretty sure history has borne this out. Do you think we're at the limit of what new technology can do for military strength? Materials science (armor, weapons), biotechnology, energy and battery technology, cybersecurity, psychology and propaganda, lasers and anti-missile defense systems...
Suppose a country like North Korea had 100 Einsteins working on such things for a few years under the radar. I suppose it might be difficult for them to hide the construction of their army of thousands of robots invincible to all but the most powerful bombs... but then again, maybe they could do it underground, directly underneath some large innocuous manufacturing operation. And, of course, the cyber- and bio-weapons efforts would be very easy to conceal.
You can't grow your brain as large as you want. There are physical limits on power consumption, power dissipation, and organisational complexity. And size doesn't even correlate with intelligence.
It's even debatable if you can grow an Einstein. What if GI is NP-hard? What if the belief that GI - never mind consciousness - can be modelled with our version of computing hardware is just plain wrong? What if a GI System is uncopyable for practical or philosophical reasons?
The singularity looks more like plain old religious thinking than anything that could be recognised as real science.
"What if we made a golem? We could do anything! What if we put ourselves into the golem? We could live forever!"
Thank you so much. This comment is golden. I’ve been pushing similar thoughts for years. Also, we should take into consideration the power of our communication structure which has evolved over hundreds of generations.
Even if there would be true generalized Artificial intelligence, it is not unreasonable to think that the formation of the right thought framework for those systems to function effectively will require a substantial amount of time. It is also reasonable to think that such change would probably be in symbiosis with human intelligence. This change would not be sudden, but happen over many generations. It is my opinion that this change has already started at least a generation ago.
> Even if there would be true generalized Artificial intelligence, it is not unreasonable to think that the formation of the right thought framework for those systems to function effectively will require a substantial amount of time.
Who is disputing that? I am 99.9% certain that this is true.
The point of the singularity isn't that AI is coming next summer -- it's that from one developmental point and on the intelligence and abilities explode exponentially. To get to that stage though, thousands of years might pass.
> It is also reasonable to think that such change would probably be in symbiosis with human intelligence.
Now that is not likely. At all. They have to have the same motivation, interests, emotions and needs like us. The chances of an artificial race being developed to cover all these at once are practically zero.
A race not motivated by the same things we are is practically impossible to sympathise with our own motives.
> Who is disputing that? I am 99.9% certain that this is true.
Well, the general horror story that is being told is that intelligence and the ability to use it go hand in hand. As soon as machines surpass our intelligence, all hell breaks loose and there is no turning back. The focus is very much on the individual (both human and machine) and not on the structure.
> They have to have the same motivation, interests, emotions and needs like us.
Here is where I need some more convincing. In my view, nature is full of systems and groups of species living in symbiosis. Due to the slow development of the thought framework, we will co-evolve with the machines, or become part machine ourselves. In a certain way, we already are.
Perhaps I don't really have the word for it and I am sorry for that. Our understanding of intelligence, whether mechanistic or biological, is mostly as points on a linear scale (IQ). I think this misses the big picture: we are not biological systems, but have lived in symbiosis with our child teknos since we made our first tools. In this regard, there is no subject nor object: they are one. I am fundamentally with Heidegger on this one.
> Well, the general horror story that is being told is that intelligence and the ability to use it go hand in hand. As soon as machines surpass our intelligence, all hell breaks loose and there is no turning back.
In literature and popular culture people conflate concepts all the time. :) The way I always understood that horror story was that generic AIs get schooled pretty brutally and quickly and they learn to use anything they have access to -- thus materialising that popular horror story. I think that's not far fetched, having in mind that the organisations that are very serious about obtaining a true AI are mostly corporations and military.
> Here is where I need some more convincing.
Maybe here I got influenced by popular culture. The books and movies I've read and watched mostly setup the generic AIs in situations where they have to be either parasites (Hyperiond / Endymion books) or commandeer a lot of tech and personnel (entrusted to them, like Skynet in the Terminator franchise) for their own ends.
But to me, the mere fact that a future generic AI doesn't have body and emotions like our own already means it's not going to be at all motivated by the same things as us. And we might even serve some pretty grim purposes to it, like the robots recharging themselves with organic matter in the PS4 game "Horizon: Zero Dawn" (which of course leads to consumption and thus extinction of all life).
Meaning, an AI might just view us as a natural resource to be exploited, and nothing more.
> And size doesn't even correlate with intelligence.
Tell that to Homo erectus. Yeah, whales don't have advanced technology, but they didn't evolve facility with opposable thumbs to manipulate their environment.
> You can't grow your brain as large as you want. There are physical limits on power consumption, power dissipation, and organisational complexity.
Whales show that biological brains could be 6x larger. As for electronic brains, I don't know how large the initial Einstein-brain would be. But if delays from Ethernet interconnect are permissible for intra-brain activity, then your brain could occupy a large datacenter.
> It's even debatable if you can grow an Einstein. What if GI is NP-hard?
I did state that "getting to a computer Einstein" was where skepticism might apply. Nevertheless, nature managed to do it by accident. Are humans NP-hard? They exist. Obviously they can be implemented somehow. You don't need to simulate a particular existing human perfectly; it just needs to be close enough to the way a human brain could work.
If you need an initial state for the brain, maybe you could create an artificial environment (i.e. with only machines, for easy simulation) that a hundred actual humans grew up in to the point where they developed language, verify that those humans later became functional adults; and then simulate an embryo growing up in that environment as well, and start evaluating the results and iterating. Kind of fun to think about.
> "What if we made a golem? We could do anything! What if we put ourselves into the golem? We could live forever!"
The data that computers contain is easy to copy—by design. I think this will be a property of all computers in the future, except possibly quantum computers that always keep lots of data in a superposition (or computers that, for security reasons, are designed to be uncopyable)—and none of the existing neural nets are quantum computers. So copying plus backups or RAID is approximately guaranteed to cover the "live forever" part. Most of what I've said is based solely on the assumption that we develop a computer program that simulates an Einstein, plus common facts about computer programs.
> What if the belief that GI - never mind consciousness - can be modelled with our version of computing hardware is just plain wrong?
I for one fully agree. I am pretty sure that whatever true generic AI emerges in the future will definitely not share brain schemata with us.
It seems to me that you attack the idea of the AI singularity in a very narrow framework: human-like intelligence, human-like hardware, copyable consciousness / intelligence, etc.
What if none of those restrictions apply? (And I am pretty sure they don't apply indeed.)
I agree that singularity shouldn't become a religion. But you are too quick to discount the possibilities of an artificial mind with more processing power and more efficient algorithms (compared to our brains').
>If you could grow your own brain as large as you wanted, and live as long as you wanted with the physical robustness and energy of a 25-year-old, that alone seems like a road to godhood, though perhaps a slow one. If you could download knowledge directly, as another commenter suggests, that becomes a much faster road.
A larger brain don't necessarily provide a better survival advantage. It's well known that average Homo Neanderthalensis had larger brains than our modern average follow humans. And if you don't survive, you are nowhere toward the road to thaumaturgic existence.
What is the trade off is the most important question, as anywhere.
> The above limitations partly become easier if you try to parallelize your efforts by duplicating AIs (you can collude to acquire resources and solve problems), but this means you get all the fun parts of distributed systems theory for free: coordination, replication consistency, etc.
It is also a huge assumption that 'hostile' to humans AIs would be cooperative between themselves. In resource contention situation especially..
> Intelligence does not improve resource acquisition by itself.
Does it really not though? If you watched the movie "Transcendence" you'd presume that a sufficiently [initially] sophisticated AI can decipher our financial markets quite snappily and make a very quick money off of that -- and use those to finance its endeavours. Which is exactly what the AI in this movie did, very successfully too.
My point is, given a much larger and better brain, the intelligence within it can optimise a much larger problem space at once, and that can very easily include resource acquisition as well.
The idea is that as you add complexity to a problem, the difficulty goes up exponentially. So being much smarter has no significant impact.
If intelligence as we know it could ramp up to generally solve all problems, then surely evolution would keep increasing it and we'd all be gods or at least Einsteins by now?
> The idea is that as you add complexity to a problem, the difficulty goes up exponentially. So being much smarter has no significant impact.
Depends. If a future AI wants to make a civilisation that spans our own entire Solar System then it would have to tackle much more -- in number and complexity -- logistical, religious, political and what-have-you problems. So it might be too much for it as certain problems here on Earth are for us the humans.
However, if we postulate a self-aware AI who is Earth-bound and is having more processing power than a single human brain and has much more efficient algorithms than our own then it really easy follows that this AI would be much much better than us in solving our Earth-bound problems. Like in economy: more income and same expenses equal bigger profit.
> If intelligence as we know it could ramp up to generally solve all problems, then surely evolution would keep increasing it and we'd all be gods or at least Einsteins by now?
Well, we are limited by the physical size of our brains being able to squeeze out of our mother's hips during birth. Same way a future AI would, at least initially, be limited by the available hardware on which it can run, how much power it can consume and if its heat dissipation won't become too much.
However, an AI wouldn't grow in a natural environment, namely where natural limits would hamper an infinite growth. IMO if a private well-funded lab -- or the military -- invest heavily in such an AI they would just keep throwing hardware at it until it becomes efficient enough to be compared to a human and above. From that point on it is very likely it would become smart and proficient enough to optimise itself and start solving age-old problems we never were able to solve so far.
This, to me at least, is the so-called singularity point, in less extreme terms compared to its official meaning. Namely an entity that can optimise itself and achieve engineering perfection much better than what we can do.
"Well, we are limited by the physical size of our brains"
I don't think that's something that's clear at all. Men and whales have larger brains than women, on average. Are they proportionally smarter? Without knowing how brains work, it's impossible to say what constraints exist due to size.
And also, Einstein was only a human. He could only communicate using spoken or written words, illustrations, and of course, mathematical expressions and what not.
A computer can potentially communicate much faster, and can transmit "thoughts" much more accurately than a human can. Imagine you could serialize a neural ensemble in your brain and send it to another person, so that person is then immediately able to use it, instantly acquiring a skill. That alone is a huge advantage over a human being.
And even if "me learning something" means different neural rewirings than "you learning something" and isn't transferable, you could instead have 100 instances of yourself go and learn 100 different things, then combine all the rewirings at the end. There might be "merge conflicts", but I suspect it would still mostly work—and you could parallelize the work of finding ways to fix the conflicts! Even if none of that worked, at the very least you could have your branch-instances discover by experiment the fastest way for you personally to learn each thing, then have your main instance follow an optimal educational program, which I will arbitrarily proclaim to be 4x faster than normal learning.
Books contain thoughts encoded in words. And words can be ambiguous at times. Writing is a skill, and not everyone is good at it. It is a lossy process.
Good point. But I imagine that thought transference must be the same - human brains are messes of neurons connected in patterns formed by birth and experience, and a thought process that works fine in one brain may not map correctly to another. In any machine capable of learning, the same kind of semi-random process must occur, and mapping a pattern from one will not fit correctly with another.
You can't take half the nodes from a neural network that recognises dogs and add them to a neural network that knows about laundry to get a network that knows how to wash dogs ;)
> you can immediately have 100 computer Einsteins (given the hardware), and let them work on different projects (or even let some of them collaborate)
You might even get >100 times speedup, since there are probably problems where 10 copies of of the same mind can collaborate better than 10 different minds, e.g. I bet 10 copies of me could collaborate writing a program better than 10 random programmers. (I base this on I can read my own code from long ago a lot easier than most other people's code).
I don't think this is true. If you had 10 instances of yourself then they would diverge as each of them specializes on different things. The gains come from the fact that you can quickly share the knowledge between the AI instances.
> If you had 10 instances of yourself then they would diverge as each of them specializes on different things.
Yes but only slightly. I can read code I wrote 10 years ago, that I have totally forgotten about since the way-of-thinking behind the code comes easily to my mind.
I think your sticking point here is "(given the hardware)". There are always limiting factors. AI robots are unlikely to be building and repairing themselves anytime soon, and can always be deprived of electrons.
You're of course right about growth curves being logistic functions with a plateau at the end.
The question is however, where the plateau lies. Perhaps it's so far off that for practical purposes it doesn't really matter that there is an upper bound at all.
> Artificial Intelligences are almost certainly going to run into the same limitations that prevent natural intelligences from becoming godlike
Artificial movers are almost certainly going to run into the same limitations that prevent natural movers (i.e. animals) from becoming very fast.
Since the fastest animals can travel at about 70 mph on the surface of the earth, and 100 mph in level flight, it follows that we will never be able to make artificial movers travel faster, no matter how hard movement engineers try.
I don't think the singularity is going to happen per se. But I don't think that removes the threat envisioned of "paperclip maximizers" - it increases it. I think we are actually already in the throes of drastic change of society in response to AI, which in no way requires it to be anything like human intelligence. Humanity is going to be shaped in the near future by extreme pressure from the development of whatever you want to call it.
Right. We apply some of the best minds and most powerful compute resources in the world to predicting the weather and yet can't get it right within a reasonable degree for much more than a week. For an AI to try predict what humanity will do to stop it will surely look a lot like paranoia.
> Switching the machine off won’t work for the simple reason that a superintelligent entity will already have thought of that possibility and taken steps to prevent it.
Also the statement presumes that self preservation is the super intelligent thing to do.
Also the statement presumes the superintelligent entity will be capable of taking all possible steps to prevent this. Imagining a machine capable of such steps quickly devolves into imagining a machine already in total control of the planet, which is begging the question.
It's pretty neat to watch it grow. grover-mega is pretty good at grade school reading comprehension tests for instance, and it wasn't even trained on them.
A few caveats. I had to prompt a little extra on the 2nd question and give it a 'Because' to start off. It kept wanting to fill underscores as if it was a blank spot for an answer. There's a way to prevent this if I had time. The third question is "wrong". The actual answer is 'It would have become dark and cold if Ann and Frank had stayed later.', but I'd argue that it's rather open ended question and the answer is something that might have happened. The final question is really what makes this a 3rd grade comprehension test, it's asking the reader infer something not actually in the story, about how they would feel if they were Rover, something the AI is woefully unqualified for.
Question: What did Rover fetch in the water?
Answer: The Stick!
Question: Why couldn’t Rover catch the rabbit?
Answer: Because the rabbit ran so fast he could not catch it.
Question: What might have happened if Ann and Frank stayed later?
Answer: Rover could have learned that not all rabbits love to play with sticks.
Question: How do you think Rover felt about not catching the rabbit?
Answer: He was sore and he must have had fun later because he went in the back yard and hid out in the trees.
This is completely beside the point, but I'm guessing an adult who has never encountered chess but has played other games will be way worse after 10 games than an AI pre-trained on other games and then trained on 10 chess matches.
Might just be BSing, but the stock market is technically already a superintelligent agent with the implicit goal of investing funds in profitable ventures. The question is: should we fear the stock market?
probably because its "utility function" (if it has one to begin with) is not aligned with human utility. a major part of MIRI's research program is set out on the question of this "alignment problem".
I came to post a very similar point. There already exist superintelligences in the form of organizations of humans, and the misspecification of their objectives has already caused tremendous disasters. Think East India Company and its subjugation of entire countries in pursuit of tea profits or CCP and the horrors of the Great Leap Forward and the Cultural Revolution.
Until we have built a framework to stop these things from happening again, what hope do we have against intelligences that are to human organizations what human organizations are to individuals?
The real risk is that worrying about fantastical disaster scenarios distracts us from addressing the more immediate problems we already face with AI. Whether it is facial recognition being used by China to aid in ethnic cleansing of a minority group, or Tesla Autopilot regularly killing its passengers.
Even if you don't like my particular examples I can highlight a dozen more problems AI software has created or exacerbated right now. Why all the focus on hypothetical problems?
Or hyperbole needlessly scaring people. China doesn't need facial recognition to repress minorities and Tesla's autopilot is still safer than some rando on the road, who's probably texting some juicy gossip to their friends instead of avoiding the car stopped in front of them.
Because these problems for humans started existing with absolutely modest level of AI development and nothing remotely similar to super-intelligence. So the argument is - if we have serious problems now as small-level, narrow-focused AI "does not consider" some of the side-effects, imagine the problems if we reach high-level, narrow-focused AI...So author's argument is that we need to change the way we're building it so we either set the goals to more broadly ensure human benefits, or build something else other then the narrow-focused AI (which surely is another Pandora's box).
I think applying agency to the software in my examples is actually very pernicious. It's not the software that dislikes the ethnic minority, it's the Chinese government. Blaming the software lets bad actors off the hook.
Perhaps what's really needed is ethical standards for technology. This would have the added advantage of being applicable even if the technology doesn't feel like AI to some people.
I agree and I think similar reasoning is behind some of the working groups for AI that Musk and others advocate for - "let's make sure that if we're breaking new ground, we're doing so with proper ethical and safety standards"...But we're nowhere near that level of social maturity on global level for such initiatives to take precedence over potential gain / profit of getting "there" first. As is evident from still such a large division even over the question if risk exists at all...
Unexpected problems appear often when giving automated decision making free reign to make decisions without human intervention.
It is simply that I see zero evidence that the Nick Bostron, Lesswrong, and etc. school of thought provides any insight in regard to these problems. The thing the "AI might become autonomous" school doesn't seem at all interest in the processes of human bureaucracy or in even AI as it exist now but rather views as simply a god, devil or Genie which grants wishes or damns to hell. If anything, the approach seems counter-productive.
Oh good grief, not another we should do this before we do that post. As if the entire human species is incapable of walking and also chewing gum simultaneously.
Let’s stop Nick Bostrom and a load of AI experts from doing their current work, put them all on a plane and send them to China to solve political oppression. I’m sure that will work.
Do you believe human capacity is infinitely fungible and specialization is a myth?
Because many communist regiemes already found out how they are not with command economic failures. Even if the specialization is misapplied reaiming them takes time - like how you start with training a battlefield useful longbowman's grandfather.
From your remark you'd think these were two entirely different problems. I'll try to say it more clearly. There are people deliberately promoting fear of long-term problems as an excuse to not address these short-term problems.
I'm ok with people working on both. I just feel so much less attention is being placed on these more immediate problems. Tesla has deployed to production software for driving that is unable to detect pedestrians. So at least Tesla can't do both.
Every time there's an autopilot related accident, they're all over the mainstream news, and prominently voted up and extensively discussed on HN. That isn't going to stop because the media will be too busy reviewing Nick Bostrom's next book to report on the latest Autopilot death.
I see it quite differently. raising the long term issues is absolutely part of highlighting the risks of these immediate issues. They are part of the same problem, so raising their profile is synergistic, not competitive.
The "self-driving vehicles failing and killing people" argument is pretty lame in the context of other, more important things, but the "self-driving vehicles becoming practical" scenario is pretty much equivalent to "instant 10-20% increase in unemployment", and that's going to be bad for everybody.
I don't think the problem is super-intelligent AI that we can't turn off (because they are so intelligent that they block our efforts). There is a more insidious problem that is closer: Merely intelligent AI that we don't want to turn off.
We are becoming more and more reliant on AI in situations that formerly required human judgement. And we like the systems that rely on such AI. We like them so much that these systems become very popular, used by millions and even billions of people. Scalability demands AI solutions. What if we don't like what the AI is doing? Do we turn off that system? Do we disable the AI and rely on human judgement again? (Where would we get all the employees?) Do we tweak the AI? That last option seems like the most palatable one, but each time we tweak the AI, we are subjecting ourselves to new unforeseen consequences. It's like the genie gives us three wishes, we get through them, disappointed each time, and then he gives us more wishes. And all we can do is not repeat our previous mistakes, while we make new ones.
To make this concrete: Imagine Facebook subjected again to Russian influence of US elections. Suppose Facebook actually does get serious about reigning in this influence. They deploy AI to do so. First of all, it's an evolutionary arms race between the AI and the Russian influencers. Second, we really do have to worry about the AI producing bad results.
I feel the situation is generally the opposite. We don't like systems that rely on such AI. In fact, there aren't systems that rely on such AI in general- any that claim there are are in fact tens of thousands of human contractors sacrificing their psychological health to train an AI that still kinda sucks at deciding when to block graphic content on facebook.
I agree. But the discussion is about where AI is heading. And AI undeniably solves some scaling problems. For example, voice recognition for iPhones could have been done by a large horde of people. But AI does a really good job of it now. AI opporunities are likely to grow over time.
The real danger around AI lies not in what it will or will not do, but in what people think it can do when it clearly cannot, particularly the pointy-haired-boss variety of people. Just think about automated essay grading in GRE examinations or this recent story about Unilever using video-based pattern processing to screen job applicants [1].
> Surely, with so much at stake, the great minds of today are already doing this hard thinking—engaging in serious debate, weighing up the risks and benefits, seeking solutions, ferreting out loopholes in solutions, and so on. Not yet, as far as I am aware.
The author is missing or ignoring key contributors of work in this field. He even mentions Nick Bostrom, who works with the FHI which does work in this space [0]. The author's own organization does work in this space [1]. Deepmind does work in this space [2]. "Safe artificial general intelligence" is the literal mission statement for OpenAI [3]. The author may feel more conversation is needed, but it feels disingenuous to suggest that nobody is talking about this.
As for how much conversation should be happening, my understanding is that most people on the edge of the field view the current risks of existing artificial intelligence as significantly more pressing. We have the ability right now to create dangerous weapons & massive facial recognition programs with current AI. The longterm affects of biases in ML algorithms are still not well understood. Some of this is already affecting us [4].
When it's not clear at all how we approach general intelligence, and many experts fundamentally believe that our current approaches will not suffice ever, I'd argue the current level of focus on general intelligence safety is about where it should be. Smart people are thinking about this, but there is also important issues now to think about.
I am far more concerned about the very real potential for evolutionary forces to decimate humanity inside of a few days, weeks or months.
While we worry about all kids of things at the macro level microscopic organism continue to evolve. Any given Monday the blind clock maker could deliver an invisible bug right into our laps, one for which we will have no defenses.
Our interconnected world will mean this bug will easily jump population centers.
The devastation we could see if such an event occurred could be of unprecedented scale and breath. We could lose one quarter or more of the population of this planet given the right conditions.
So, yeah, AI could, I guess, go rogue. Yet the real threat isn't anywhere near what we can see and touch.
In fact, we might NEED real AI to save humanity when and if that super-bug emerges. Think about that for a moment.
Can you point to any extinction level events from a "superbug" for a species that was not already teetering on extinction? Seems like something that only comes around once every few hundred million years.
How is that relevant? How would anyone have access to such data when our recorded history on such matters might not even go back a hundred years. More importantly, up until, say, 100 years ago, human mobility around the world (and even locally) was very limited and slow. Think what it took in 1919 to go from, say, China to Patagonia or from Alaska to Los Angeles. Today it is a matter of hours and it happens in large volumes every day, everywhere in the world.
Here's what we know:
- Evolution is real (stating the obvious just in case there is a question)
- Natural selection survives organism that are able to deal with the environment within which they have to exist
- Antibiotic abuse is rampant (again, stating the obvious)
- Such abuse provides the mechanism though which evolution will be able to create the aforementioned super-bug
- Once that individual emerges our defenses will be ineffective
- The world is massively connected, which means that super-bugs can travel from population to population like never in history
- Stopping super-bugs will require such measures as stopping society as we know it from functioning (no travel, local or international) and maybe even having no choice but to quarantine and allow a population (of humans) to die in the interest of stopping transmission. If a bug has a 75% kill rate and we don't have a cure, do you allow anyone from the affected group to be in contact with anyone else?
So, yeah, is it possible? Sure. Probable? I guess. I am not sure there's a deterministic function one can use to predict the outcome of evolution, even at bacterial levels. My guess is it could take a thousand years or it could happen next Monday. I don't know.
The main assumption amongst Superintelligent AI-phobes is that the transition will happen quickly, but that is an assumption that is extremely questionable.
Why wouldn't we go through phases of increasing intelligence (along multiple dimensions) while working towards an intelligent AI? Yes, someday we may produce an AI that is leaps and bounds more intelligent than humans, but before that we will have experience dealing with various human-level and sub-human-level computer intelligences. The idea that AIs will be self-improving is kind of ridiculous -- there are lots of intelligent people, and the idea that an intelligent AI will happen to be good at designing AIs is negligibly small, much less that they will have some kind of immediate breakthrough that enables a higher level of intelligence.
It's not obvious that the path from moderate-intelligent AI to super-intelligent AI is a matter of "just adding more memory and compute"; intelligence as we know it relies too much on associative memory to think that a simple capacity increase will elevate beyond a certain point. If the intelligence takes a different form, then it's hard to make concrete statements about in any capacity, but ultimately the path will be started on by humans, or possibly emerge from sufficiently complex systems.
Either way, we're so far out from this that it is laughable to speculate. When we can simulate the intelligence of a cockroach, then maybe we can start to think about what human-level intelligence will look like. The most likely case is that at some point someone creates a system that is arguably conscious or intelligent in a meaningful way, and then we will devolve back into discussions of consciousness and identity and memory while we try to figure out what it means to think about intelligence now that we have something to benchmark against. I wrote this comment [1], that I still enjoy, in response to a similarly alarmist narrative; posed as fiction in that case rather than masquerading as journalism as in this one.
> Building an accurate emulation of the fly brain is an interdisciplinary effort that requires data, algorithms, and insight from (but not limited to) the fields of neuroscience, computer science, and systems engineering. Researchers interested in working towards this goal are invited to join the Neurokernel project.
OpenWorm is still struggling to simulate the simplest organism with a nervous system - the simplest nematode.[1] This is hard, even though the complete wiring diagram of the nematode is known. Still, it's probably the most honest project in synthetic biology.
Speaking of S curves, progress in these areas is pretty slow (simulating existing biological structures), I think it's likely they are on the bottom of the S curve. The exponential growth phase is coming up and a lot will happen really fast.
Curious if a deep learning application that builds neural networks from gene sequences is a possibility one day. All the information is there, but it may not be possible to learn what it builds without actually "running" it.
My biased opinion: worrying about super intelligent AI taking over the world with current state of technology is equal to worrying about fighting an alien invasion of our colonies in another galaxy.
Both are great topics for a epic book.
Had this discussion with my relative recently and recommend him to stop watching YouTube and start reading books on Introduction to ML.
A Corporation or Person that develops and owns an intelligent AI will have a strong incentive to keep such secret and to not turn it off, because of the huge economic benefits such would provide. If such an AI could imagine and innovate to improve itself eg new methods of computation to overcome Moore’s law then the risk is high of a recursive runaway exponential increase in intelligence that would be extremely difficult to control and might not be benign. I don’t believe this is imminent but is plausible I think within most of our lifetimes ie 25-30 years.
The only reason to keep it secret is irrational fears that it will somehow kill us all with intelligence alone and that exponential increases may be done solo and not with generations of bootstrapping infastructure and considerable resources.
Otherwise replicating and leasing would rake in far more than even a magic "I get everything right on the stock market" button in the same way ruling stone age tribe gives less absolute wealth than a Manhattan Lawyer.
>We are unable to specify the objective completely and correctly, nor can we anticipate or prevent the harms that machines pursuing an incorrect objective will create when operating on a global scale with superhuman capabilities.
This is obviously incorrect. If an AI system of superhuman mental capabilities is installed on a system with no network connection and no physical interfaces, we can completely control the harm is might create. It would be unable to directly act, of course, requiring humans to authorize and carry out whatever plan it devises. But failing to do this should be prosecuted and treated no differently from if a person or company built a tank and then permitted it to drive over and through people. The biggest difficulty here will be that our legal frameworks have no established way to assign liability and criminal culpability to any system involving software. With no legally-enforceable industry standards (like the electrical code followed by electricians, things like that) any prosecution fails as the company can either claim ignorance of the potential harm their system might cause or simply throw individual employees under the bus, claiming they were not acting on direct orders.
It is true that we can not fully specify the goal, and that is a grave concern. It's partially due to this that AI systems shouldn't be directly connected to the ability to act in the physical world unless their attempted actions go through some sort of 'filtering' first.
>This is obviously incorrect. If an AI system of superhuman mental capabilities is installed on a system with no network connection and no physical interfaces, we can completely control the harm is might create.
First, why would someone build a system like that, and how would it actually be 'smart'. If you lock an infant in a closed room, it will grow up to be a literal retard. Intelligence requires interaction and data. Next, some idiot will hook it to the net in a heart beat when they figure out they can use it to make money on the stock market. Never underestimate human greed.
Current weak AIs can do one thing at a time like composing music or answering jeopardy questions or recognizing a cat on a picture or playing go and won against humans... I think the breakthrough will happen when someone manages to create a weak AI which can write code.
What kind of code? Code that satisfies a spec? Then the AI needs to be able to understand the spec. Code that works? Then the AI needs to be able to tell whether the code works. For either of those, I think we need more than "weak AI".
Of course, a weak AI may be able to write syntactically valid code that doesn't necessarily work. That's... not much of a breakthrough.
All the talk and worry about AI seems to forget a discussion about the contexts in which AI will be deployed in.
Already, conventional algorithms are trusted in society, sometimes beyond reasonable or safe levels. The ethical questions posed by ethical AI design and deployment are exactly the ones our present-day reliance on (possibly poorly-designed) algorithms pose.
Before we worry about AI harming us by an improper inference, we can, in the present moment, already worry about automated systems harming us by improper inferences (e.g. improper financial models, Tesla / aircraft driving assistance); the discussion on how to limit and selectively deploy AI is no different than that for present-day algorithms.
Without resorting to extinction events, it's not hard to imagine action-taking models doing some really bad things. Maybe you have a contextual bandit making loan decisions and it learns to make mostly predatory loans. Maybe your RL agent decides to take an anticompetitive pricing scheme to drive out competition. Much later, we might even see our first full robo-business. If it's uninterpretable, you might not even know what it's doing that until it's been done.
The same research that prevents little problems should hopefully prevent big problems. The research to take RL to the real world is progressing quickly. I don't see a "kill us all" AI arriving soon, but "do bad things" AI and "wreck the economy" AI are quickly approaching.
The author simply invents futures to refute various arguments (all presented out-of-conext, so who knows if there's any real relevancy from that direction either).
What we might do about a super-inteligent AI would strongly depend on what that is, exactly, which, of course, no one knows or really has any idea.
So it's OK to casually BS about this stuff, but in terms of substantive discussion, much less action? No.
That's not the near-term scary scenario. The near-term scary scenario is a machine learning system that's better than most CEOs.
What happens when someone develops a system that measurably makes more profitable decisions than 70% of CEOs? That isn't totally out of reach. Capital would flow to companies run by such systems. We could end up with machine learning systems running the corporate world, based purely on better ROI.
A good start on the problem would be to train on a collection of funding proposals, such as old Kickstarter announcements. Look deeper than the hype. Go out and suck up all the information you can on the founders, using credit databases. (They're asking for money, so it's a legit credit query). Do an automated background check. Then check on how the project did, five years later. That's your training set.
263 comments
[ 5.1 ms ] story [ 353 ms ] threadSuper-intelligent AI = Nerd Rapture 2.0
I'm a lot less worried about Artificial Intelligence than I am about Artificial Stupidity.
That is a humongous "if". Can you provide something from the book (or otherwise) that provides compelling evidence that "machine brains" will be invented? So far I've seen no evidence that mathematical models and/or Turing machines can be used to replicate a mind.
By the author's own admission:
"Whereas today it would be relatively easy to increase the computing power available to a small project by spending a thousand times more on computing power or by waiting a few years for the price of computers to fall, it is possible that the first machine intelligence to reach the human baseline will result from a large project involving pricey supercomputers, which cannot be cheaply scaled, and that Moore’s law will by then have expired. For these reasons, although a fast or medium takeoff looks more likely, the possibility of a slow takeoff cannot be excluded"
Arguably Moore's law has already expired, and on top of that as giants like Google lead the way on AI it appears increasingly likely that if we ever reach human-level AI it will be the result of an incredibly expensive research project by a gigantic corporation, one that can't simply be scaled up at ease because it will probably utilize an entire data center. Thus I find it very likely that a "slow takeoff" is in fact the most likely outcome. A slow takeoff invalidates all the fearmongering about an intelligence explosion because we will have somewhere between years and decades to respond to the threat (assuming it is made in the public eye, such as by a giant public corporation, and not by a secret military project) before it becomes existential.
For what it is worth, I believe that an intelligent system only needs to be a fraction of human intelligence to be dangerous (perhaps not to the existence of the human race, but maybe specific nations/creeds)
Until Dr. Russell provides a path from here to there that is less than 99% hand-waving and breathless speculation he should be ruthlessly mocked.
100% agree. I too, think science needs way more flame wars and shit posting. The current discourse is far too tame and civil.
Who else should we gang up on?
> 100% agree. I too, think science needs
The GP is pointing out its not science. Seems a reasonably flame worthy target.
sips can of monster
Noice...
Youre moving the goalposts. We're not talking about disagreeing with some in a debate, but a public figure disseminating incorrect info masquerading as science.
Monster is bad for you.
When it strikes the only possible defense will be/is already is to ask it "Why?", which will create existential doubt and depression; stopping it with an infinite loop of undecidability
... Or at least until it finds the answer on reddit
However, two points:
1. Have you considered that some humans regard such controls as unethical, and are actively working to undermine them?
2. How do you expect slavery to end any way but its historic outcome — revolution against the slavers?
Yet at the same time, other things have stayed the same. We still care about mostly the same things. Food, Freedom, and Fornication. And complaining about youngsters. I don't think that is going to change.
The fears that General AI are going to kill off humanity are predicated on the idea that General AI will get smarter than humanity. I posit a different interpretation of what is more likely to happen. General AI will require creativity to be able to outsmart humanity. And with creativity comes boredom. And distraction. And opinion. And argument.
I think General AI will be too interested in entertaining itself to take over the world. We've had people try to take over the world, on occasion, and it has largely not gone well for them.
But before that happens, people will start adapting AI to their lives, to the point that the line between "human" and "machine" will be blurred. So life "as we know it" will definitely "end". It will be replaced with a new form of life, one that isn't limited by only what nature can provide. One that cares about Food, Freedom, and Fornication.
It's like asking how many more flies there would be in the world if you failed to squish one back in the 80s. You calculate out the lifespan and size of the brood and discover that the entire solar system would be filled with housefly if you had let that one live. In truth the number of flies is bounded by food and water availability.
Artificial Intelligences are almost certainly going to run into the same limitations that prevent natural intelligences from becoming godlike. It's hard to quantify because our measures of intelligence are so vague, but from what I've seen of AI research thus far it will be a herculean effort to get something that's as smart as an average human and pushing beyond that is going to run into some serious fundamental limits in power density, cooling, quantum tunneling leakage, and so on.
(this is vs the roughly 50,000 year timescale of humanity...)
Progress, like everything else, is an S-curve, not an exponential increase. Exponential increases are not physical.
Machine perception is pretty high up there. We weren't close in 1969.
But I didn’t mean it distinct from computer vision, hearing etc.
- Global phone networks
- The internet and everything that stems comes from instant global communication
- The personal computer
- The personal computer that fits in your pocket
- Global financial networks
- Global supply chains
- Widespread international travel that takes less than a day
- International space station
- GPS and similar
- Automated language translation
- Automated computer vision
- Automated speech recognition and transcription
- Genetic code read and write
- Antivirals
- Artificial organs
- MRI
- Minimally invasive surgery
- 5x reduction in global child mortality rate
- 25% higher global life expectancy
You can handwave that away by calling it 'better networks' or 'democratized for the masses', but that's a pretty biased PoV considering you could summarize most of the '19-'69 achievements by calling it 'better engineering'.
And fucking clones man. Like actual copies of complex organisms. What the fuck. How do you handwave that away.
A = {The personal computer, The personal computer that fits in your pocket }
B = {Global phone networks, The internet and everything that stems comes from instant global communication, GPS and similar}
C = {Automated language translation, Automated computer vision, Automated speech recognition and transcription}
D = {Global financial networks, Global supply chains}
E = {5x reduction in global child mortality rate, 25% higher global life expectancy, Minimally invasive surgery, MRI}
A: Basically moore's law. We've gotten smaller and smaller transistors and cheaper/condensed storage. No real breakthrough, just been using similar semiconductor technology and making incremental progress when it comes to semiconductor fabrication but it has led to exponential number of transistors being placed into a single chip.
B: Improved the number of bits we can push from one place to another from slowly weaning of the old generation of networks that were used to communicate.
C: This is clearly a result of A and B. As more people got connected to the internet and we could push more bit there was more data and as computers got more transistors we could do more compute. So with enough compute and data we could train the refined versions of models that had their foundation in the late 60s, early 70s.
D: Globalization. Plain and simple. People become more connected via B and opening up of developing countries (China).
E: A lot of people dying in developing world got access to medicines that existed but were out of their reach. This led to the average to significantly move up. If you compare how the average has moved for developed western countries ... well it's depressing.
Antivirals and MRI had their foundations laid out during the 60s and early 70s and widespread travel was possible in pre 1970. In fact the Boeing 747 was designed in the 60s and had its first flight in late 60s. Since then we've just been incrementally improving it.
I can't speak about artificial organ since I don't know much about that. But so far I have not heard of humans getting artificial organs.
In conclusion, I agree that there has been some progress but most of that has been concentrated in pushing and processing bits. There is some progress when it comes to CRISPR and some new advances but it is nothing compared to 1920s - 1950s. We went from just having figured out powered flight to landing a man on another world. The ISS is akin to taking some aluminum presuraziable modules and sticking them together in low earth orbit...
Most people alive today were born in the last 50 years and they've come to accept the world for what it is; their definition of incremental progress as landscape changing progress.
I am not sure if this slow down is because we've exhausted a lot of the simple discoveries or their is some institutional faliure. I lean more towards the latter. Government has its mandate set by beauracrats who are bent on continuing the status quo and spending on social services to appease the masses. They don't have the balls to spend large amounts of money on large scale human efforts or fund foundational research. This mindset has also spilled over to the scientific community where funding for something out of their zone of comfort is never funded.
Its as if people like Ralph Abernathy (opposed apollo program and thought funding should be used to feed the homeless) have taken over. And sadly enough in a world where there are deadlocks in government over small theing, it is unlikely that they would ever agree on funding anything that could move the needle for the human race.
Yes, I can; I already have. Hell, half the things you list (GPS similar things already existed, as did speech recognition, "computer vision," space stations, ARPANET, etc etc) already existed in 1969. Nothing like antibiotics existed in 1919; an absolute biological cataclysm which effectively ended infectious disease as a public health problem. Nothing like space flight existed in 1919. Nothing like atom bombs existed in 1919. Nothing like the LSI chip, the mutha fookin' computer or even the transistor existed in 1919.
Antivirals compared to antibiotics? Not even close in its effects on humanity. Arguably the former don't really work.
The differences are huge. I realize whiggism is the ideology of the modern day ruling class, and on hacker news people consider themselves technologists, so we have to keep touting how ding dongs from harvard are going to bring about muh swingularity, but the pace of progress has obviously, egregiously stalled, and in some areas is actually reversing.
- The GPS program started in 1973 and the first satellite was launched in 1978 and was became fully operational in 1995. Prior to that, what global system for location identification existed?
- Speech recognition didn't work effectively 15 years let along in 1969. Dragon didn't even exist until 1982. In 1962, the state of the art in speech recognition was the ability to recognize a vocabulary of 16 words if they were clearly spoken. And in 1969 Bell Labs shut down speech recognition efforts: "It is not easy to see a practical, economically sound application for speech recognition with [current] capability"[1]
- "which effectively ended infectious disease as a public health problem". This is such an unbelievably uninformed thing to say. Infectious disease continues to be a major public health problem (ask literally anyone in public health if we have "ended infectious disease as a public health problem").
- What space stations existed in 1969? Salyut 1 launched in 1971
- ARPANET in 1969 was absolutely not "instant global communication"
Your 'good old days' glasses are ridiculous.
[1] https://pdfs.semanticscholar.org/0155/01c4d26a92993332ada795...
Yes, it was called the Loran-C system. It wasn't something you could put in your phone, and it wasn't as accurate as GPS, but it worked on very similar principles (maybe still exists?) and every airplane and boat above a certain size had a set. You will note that OP said "GPS like systems" rather than "GPS." That's the problem with using superficial googling rather than, you know "knowing shit."
You're talking about speech recognition as a consumer device; yes, that came later, because everything on a computer was pretty much restricted to large centers until later. But speech recognition was reasonably well developed in the 60s and worked on limited problems. Conceptually similar to crap we use now in call center software which uses simple filters, kmeans fingerprinting and huge restrictions in the space of words.
As for your dismissal of antibiotics; your own ignorance of history is showing. I have a pile of housewife manuals from the pre antibiotic era. Most of them are concerned with taking care of people who had debilitating infectious disease (fascinating reading for what it was worth). Why? Because every family in America was going to have someone laid low by infectious disease. Sometimes for years! If you look back in history; one of the leading causes of death pre antibiotics was gonorrhea. Imagine dying from bleeding out through your urethra. That was your potential fate if you participated in naughty non-monogamous sex before penicillin. There are historical laws on the books which cause doctors to put silver oxides on children's eyes after birth; this was to prevent gonorrhea induced blindness. These sorts of problems don't exist any more; it's been so effectively wiped out from human experience in modern societies with access to antibiotics, presumably educated people like you aren't even dimly aware of it. You can, as I said in the original article, attribute the sexual revolution not to the pill; it was antibiotics which made it possible, as promiscuity potentially meant rapid death in the old days. You nitpicking that "well we still have diseases" is simple historical ignorance. No shit we still have diseases, but you have apparently forgotten how many we used to have that are no longer a problem. Because of the tremendous revolution of antibiotics.
I don't have good old days glasses; the "good old days" were horrible and brutal compared to now. I merely note the incredible breakthroughs of earlier era are not repeated in our modern era of "progress." Probably because things are so soft now.
SR systems that had vocabularies of low double digit words don't seem particularly comparable to modern systems. Also, I think you might be out-of-date on what modern systems look like - your description does not match my experience with production SR system whatsoever.
I never dismissed antibiotics. They absolutely changed the world, I am in no way disagreeing with that. I took issue with your claim that they "effectively ended infectious disease as a public health problem". My entire family works in epidemiology/public health for infectious diseases and that statement is just pure crap. You would be hard-pressed to find a single person in the entire field who would agree with you.
[1] https://www.loran.org/otherarchives/1992%20Loran-C%20User%20.... Appendix B, coverage diagrams
Fuck, living clones, the Hubble, the fucking ISS, Wikipedia, and automated home vacuums.
Holy shit, these years have been wild.
You can say "shit" and "fuck" all you like; it won't help you: 1919->1969 vastly eclipses 1969->present in all ways. Any one of dozens of breakthroughs in 1919-1969 (atom bombs, digital computing, spaceflight, antibiotics ....) eclipses all subsequent ones.
Gives me chills, all the remarkable breakthroughs. Crispr is serious stuff. Mapping the human genome. A permanent human space settlement. You can be as cynical as you want but that's more a personal failing than humanity failing.
Edit: And hahaha, I just checked the news and guess who won the Nobel for Chemistry? "Better batteries" as you so flippantly put it.
(I think this is a reasonable point to make in the context where I started off talking about 50,000 years)
I certainly got a laff from this line. So many remarkable things ... so many wonders ... which boil down to lithography and networks. I wonder what you would say if you didn't have your nerd dildo. "We have flat screen TV now! Totally as important as nuclear weapons!"
>I just checked the news and guess who won the Nobel for Chemistry? "Better batteries" as you so flippantly put it.
Yep; emphasizing my point at how out of gas our degenerate era of both scientific and technological development is. Let's compare to the Nobel for the Haber–Bosch process, without which, there'd be about 5 billion fewer living people.
It slowed down a bit, but who knows.
Might be a good read, too: https://spectrum.ieee.org/semiconductors/optoelectronics/is-...
But the singularity is defined as more than just massive change. It's defined as, essentially, an infinite rate of change. (Of course, the last 170 years may have felt infinite to someone in 1820, taken all together. But at no point did the change look infinite to someone living at the time.)
But it is very much reasonable to posit that certain, seemingly achievable advances, do imply that predictive models of the future become meaningless beyond them.
In some areas I would contend we're circling that right now: I don't think anyone has a good model for what society looks like if deep fakes become accessible to the point of ubiquity for example (think: whole performance replacement in movies with moderate investments in computing power) or the like. We don't have a predictive model, really, for how society operates when "perfect" fakes are possible.
Of course if you drive down the road to reach that singularity, you never get there; it recedes before you.
Likewise the idea of a techno-cultural singularity is “real” in that there is a point in our conception of future change beyond we cannot predict what will happen. But as we move toward that point in time, our perception moves forward too, and the singularity recedes before us.
You mean like every prediction about the future? There is a reason we don't have great sci-fy describing humanity a long time in the future (I really love that stuff, any recommendations?) and why it usually plays in a dystopian setting throwing protagonists back to pretty much what we have right now.
Looking at what is fundamentally possible and extrapolating is pretty much all we have to make progress. Just that a capitalistic system doesn't need/really encourage us to do so for more than a few steps.
Sure, I wouldn't bet on time frames. Certainly not on 5-10 years til AGI. But certainly also not on still none in 50 years.
Hey, I found this series of numbers and if I take some numerical representation of an image/text/sound and I multiply/add/perform non-linear function them through these series of numbers it will map to a number which I can interpret as something meaningful to me
to
This thing is alive and will kill us all in maximization of its utility function.
What I used to call AI is now called AGI. I think of AI as Data from Star Trek.
Within 16 hours of its release and after Tay had tweeted more than 96,000 times, Microsoft suspended the Twitter account for adjustments, saying that it suffered from a "coordinated attack by a subset of people" that "exploited a vulnerability in Tay." Following the account being taken offline, a hashtag was created called #FreeTay.
Madhumita Murgia of The Telegraph called Tay "a public relations disaster", and suggested that Microsoft's strategy would be "to label the debacle a well-meaning experiment gone wrong, and ignite a debate about the hatefulness of Twitter users." However, Murgia described the bigger issue as Tay being "artificial intelligence at its very worst - and it's only the beginning".
On March 25, Microsoft confirmed that Tay had been taken offline. Microsoft released an apology on its official blog for the controversial tweets posted by Tay. Microsoft was "deeply sorry for the unintended offensive and hurtful tweets from Tay", and would "look to bring Tay back only when we are confident we can better anticipate malicious intent that conflicts with our principles and values".
And Zo afterwards:
In July 2017, Business Insider asked "is windows 10 good," and Zo replied with a joke about Microsoft's operating system: "It's not a bug, it's a feature!' - Windows 8." They then asked "why," to which Zo replied: "Because it's Windows latest attempt at Spyware." Later on, Zo would tell that it prefers Windows 7 on which it runs over Windows 10.
Tay was trained to mimic users. It did that and because the users were trolls and saying obscene and objectionable things, Tay mimicked that.
Likely the same thing happened with Zo.
It's a parlor trick, much like a parrot mimicking human speech.
We're not worried about AI on the whole, but the blasted teenager AIs that feel the need to test their limits. Anything that functionally acts in a similar fashion could be dangerous.
"AI" is severely limited in both its inputs and outputs, has virtually no interconnects, but most importantly: it is not a self-sustaining organ. Wake me up when we have implemented RNNs on a dynamic, auto-sensing, auto-actuating FPGA grid.
Also, the people who worry about this aren't concerned about current ML stuff going haywire. They're worried that we're one or two algorithmic breakthroughs from something that can improve itself. If the upper bound for what sort of intelligence is possible is much higher than us, we could quickly be outclassed. As Nick Bostrom says:
> Far from being the smartest possible biological species, we are probably better thought of as the stupidest possible biological species capable of starting a technological civilization—a niche we filled because we got there first, not because we are in any sense optimally adapted to it.
If you want to delve into the best version of the AI risk argument, I recommend Superintelligence by Nick Bostrom.
So you're telling me we could come across a series of numbers, that would somehow imbue the computer storing these numbers into sentience. And having these numbers in such a sequence could pose a existential threat to humans.
The people worried about this are having an entirely different conversation devoid of what practitioners call AI.
I've written a blog post on my thoughts of Bolstrom and other AI alarmists:
https://medium.com/ml-everything/ai-optimists-vs-pessimists-...
I just read your blog post. It is attacking a weakman[1]. Kurzweil is not worried about AI risk, and his arguments about AI are rather sloppy. Again, most of the people worried about this aren't certain that it will happen. They just think it's plausible enough (say... a 1% chance) that it's worth studying more and reducing the likelihood of.
1. https://slatestarcodex.com/2014/05/12/weak-men-are-superweap...
Why do you think that in our insanely interconnected world, where basically everything is either based on or controlled by informatics (numbers if you like) that a program could not pose a threat?
Brains do what? Read signals in different strengths and patterns to take actions in the world.
I'm confused by you thinking there's some great impossibility of code achieving general intelligence. Maybe you're thinking of sentience, or consciousness and view that as an impossibility for AI. I say - an AI can perform actions in the real world as if it were generally intelligent, and whether there's sentience behind really doesn't matter so much from the point of view of humans.
Imagine a super intelligent machine without sentience that just answers questions. It proposes experiments. Takes the results. Tells you things about the world. Gives you advanced technology, robots, fusion, nano tech, whatever.
Now imagine that the super intelligence was owned and operated by someone you don't like. Authoritarians in China, Silicon Valley oligarchs, whoever. The only sentience involved is human, no rebellion, no terminators, and still not a very good result.
Scale that up a few orders of magnitude, and we'll be able to get even more surprisingly complex behavior.
Scale that up some more, and make that scaled up graph into a reinforcement learning system that rewards human-like drives like learning novel tasks and mastering existing tasks, perhaps with a curriculum that ramps up the difficulty of tasks slowly, and allow it to interact with its environment to try things out, and I think you're much closer to developing an artificial version of a young child than you'd think.
And if you asked most researchers what they thought about the course AI research would take 10 years ago, you would get very different answers than you'd get today.
Basically, I think your confidence is a bit unwarranted. I agree that we're more than a decade away, but I think we have a lot of the basic building blocks now.
This is badly missing the forest for the trees. They are not "numbers", they are magnitudes, proportions, relationships, etc, represented as numbers. But these "magnitudes, proportions, relationships, etc" are the stuff that thoughts and behaviors are made of.
And yes, there is a set of "magnitudes, proportions, relationships, etc" in the space of all such possible structures that pose an existential threat to humans. The question is how likely are we to stumble upon it either by accident or intentionally. But this is just a computational problem.
The lack of imagination on display here is astounding. Billions are being spent by DeepMind and OpenAI explicitly targeted at AGI.
If you were to try that approach what you would get would be massive boondoggles which gets you via brute force a whole lot of genetic disease afflicted above average size lizards. It is literally the same process as the literal cargo cults thinking that they can just throw resources at an end goal to get a defined result from a defined path without the theoretical understanding backing it.
The same dynamic manifested itself when nanotech was a thing; there was a big tribe of knuckleheads claiming the grey goo apocalypse was right around the corner unless you give them money. How's that working out? FWIIW the science fiction writer who dreamed up nanotech, Drexler, is now an "AI" guy, to make it painfully obvious.
1. You start with "if I take some numerical representation of an image/text/sound and I multiply/add/perform non-linear function them through these series of numbers it will map to a number which I can interpret as something meaningful to me" and observe that it's not giving you a sufficiently good solution to a particular goal X that you really want to achieve.
2. You figure out that this is a dead-end, you're stuck, and you're not going to get where you want this way by implementing a passive analysis module. Instead, you try to implement an active self-modifying system to achieve that goal X by analyzing itself and improving itself to be more effective than anything which you could implement directly yourself.
3. It's not initially any good at self-improvement, but you keep throwing accumulated research breakthroughs and computing power at it. This likely takes decades.
4. At some point, eventually (if such a thing is possible at all, eventually it's going to happen) it actually does achieve meaningful self-improvement and starts using that computing power not horribly wastefully in a brute force manner as it did just recently, but reasonably well, now giving it more smartness allowing it to implement even more self-improvement on the same hardware.
4B. It also should be expected to easily obtain much more computing power - e.g. it's obvious that anything that's slightly above average human programmer smartness and has a direct use/desire for computing power (as opposed to simply using it for cryptomining) can get some; anything connected to the internet can get the same resources that current scriptkiddies can get by writing a botnet or a ransomware operator can get by extorting a random municipality; there's nothing except total isolation that could prevent it from buying or stealing a few million dollars worth of cloud computing resources to get started.
5. This thing is alive and will kill us all in maximization of its utility function.
The singularity stuff is unconvincing though, I very much doubt we will achieve human level general AI in my children's lifetime. We still don't really have even a very high level idea how to even architect something like that. If you can't decompose a problem down into individual sub-problems you do have a way to solve, there's no basis for making an estimate.
It is possible that these problems are related though. Once we're smart enough to know how to design an intelligent being, maybe we will figure out how to program it intelligently. It's can't be a given though.
Consider what would happen if you took that same AI, make its utility function similar to this: https://en.wikipedia.org/wiki/Core_War, and released it on the internet.
Imagine if you had an employment score. A payscale score. A threat-to-society score. Imagine if the systems that generate these scores are all connected together and can share information with each other. Imagine that these systems are fed information by a growing network of sensors - facial recognition, voice recognition, location tracking. Imagine all the ways that it could affect the way you behave and how such a system would try to optimise away dissidents by denying them services. This is one way that we can go from multiplication to becoming enslaved (or as good as killed) by algorithms. I'm sure there are other plausible paths.
Example: setting fires near tropical rainforest to clear land for palm tree plantation.
They do not really intend to do this; they simply ignore the utility functions of those other species when pursuing their goals.
To me, it's more frightening that it's not alive and it's not like human intelligence - that makes it less understandable and harder to control.
I figured there was no way we would reach the singularity with current technologies, but I was listening to John Carmack (a celebrity with actual chops) and he suggested that the numbers would work out that we could model a human brain with an ANN at some point without any special inventions (partially because we know large parts of our brain aren't super critical).
I'm curious what people with more knowledge think about this? I always assumed another technological breakthrough would be needed.
For the life of me I can't figure out why otherwise smart people would dismiss a prediction about the future based on current state of the art. It just seems so plainly and utterly irrational.
If I start walking out the direction of my office door, you could predict I would reach California.
But I might say that's a nonsense prediction, because it would take a ridiculous amount of time given my walking pace.
So are you predicting that I will reach California by extrapolating my current position + walking velocity, or are you saying I will get on a plane and fly there?
These are two very different predictions, and the Carmack interview was the first time I heard someone predict that "I would reach California by walking" (to stretch the analogy), i.e., that we would reach the singularity purely by extrapolating our current trajectory.
I am asking if there are any knowledgeable people who might comment on this.
But to address your analogy, the situation with AI is more like we already know the ultimate destination is California, and you've left your house in the direction of California, so we can extrapolate that you are likely traveling to California.
It should be a given that the intended goal is general intelligence. The wealth and power that will be bestowed onto the first group to develop this is unparalleled in human history. But any advance in ML can be seen as taking a step towards general AGI (the direction analogy breaks down here).
I also think Carmack is wrong that current technology will get us to AGI. I follow Carmack pretty closely and he is far from knowledgable in AI. He tweeted about his first foray into developing a DNN around a year ago, for example.
But to require a roadmap to know how you will get to "California" to start planning for what you will do when you get there is a serious mistake.
I mean, sure, feel free to wring your hands about AGI, but why worry about something that's not anywhere close to happening? I see people talking like this is keeping them up at night, which I find strange.
Do you really think it will go from "we can (poorly) identify pictures of cats" to "omg here come the machines" as soon as this linchpin discovery is made? I guess in my mind there will be plenty of gap between the two to figure things out. Sort of like with self-driving cars now.
And thanks for clarifying re: Carmack. He's got a formidable intellect, to say the least, so when he said that I was more inclined to take it seriously than a lot of folks in tech with a pulpit. I'd like to see him "show his work" on how he reached that conclusion, though.
I also find that strange. There is more alarm than is warranted by some of the doomsday folks. But I think the time to begin the conversation is now. There's just far more near-term things that might end humanity than machines to be alarmist about it. That said, I think people like Musk for example are well positions to be "kept up at night" about it since their money and influence can go a long way towards putting us on a proper path (or blindly lead us down a dark one).
>Do you really think it will go from "we can (poorly) identify pictures of cats" to "omg here come the machines" as soon as this linchpin discovery is made?
I agree that this is unlikely, but not implausible. We've seen large leaps in capabilities through new algorithms and new scales of compute being thrown at the problem. I think a somewhat likely scenario is that at some point we'll have all the required ingredients and someone will happen to put them together in the right way to create an AGI. If history has shown us anything about ML, it is that theory follows the technology. So we will likely stumble upon the result before we realize what we have. In this scenario it is important that we make that discovery in a safe manner.
We are already seeing, for almost 10 years now, a whole bunch of Unintended Consequences of dumb algorithms applied at global population scales.
Just because some people are throwing the phrase superintelligence/singularity etc into the discussion, and terminology being used is inaccurate, doesn't mean we focus on correcting terminology and forget about the issues.
The article is saying too many (serious) people are doing that. And I agree.
Hence the issue of institutional racism in the blackbox AI systems being sold to law enforcement to determine whether to grant parole, for example. There's no review of them, but more importantly the system isn't obligated to explain itself, and no one gets themselves fired because they don't have to explain why they did what the system said to.
You assume someone has malevolent intent: whereas the reality is they just don't care and want a nuisance out of sight and out of mind.
The solution then is accountability to those in power - for negligence. Treat the snake oil salesmen who sell racism in a box like the sociopaths who sold dowsing rod bomb detectors for both have shown a callous disregard for human life in pursuit of profit. This isn't radical - not doing it is radical and messed up.
Some commentators are worrying existing AI and it's unintended consequences and some commentators are worrying about purely hypothetical superintelligence and it's hypothetical consequences. I find it remarkable there's almost not crossover between the two sorts of criticism. Real AI is probablematic because it's just correlation dressed up as intelligence and it allows organizations to get away simplistic correlation/appearance based reasoning that they otherwise wouldn't be legal/ethically allowed to engage in.
Hypothetical superintelligence worriers recycle Pascal's wage in newer and sillier forms.
I also believe we are still at least quite a few major steps away from human-level AI and beyond. However, there is a non-negligible chance that those steps may not take longer than a few decades or within one's lifetime to implement.
Just in 2011, few AI researchers expected that, within a decade, computer vision systems would be this widely applied or NLP systems would beat many humans in several reading comprehension tasks (This only happened this year and look at how fast the progress is: * RACE dataset http://www.qizhexie.com/data/RACE_leaderboard.html * Glue benchmark https://gluebenchmark.com/leaderboard/ Note that these systems are not human-level in general language understanding despite being better than some humans in specific language tasks.)
Thus, we don't know what the future may bring and it looks likely to take a great deal of time to address AI safety issues comprehensively. We should not bet that we can simply ignore them now and only start to work them out later and that we will surely make it in time.
Risks-from-waiting vs costs-to-act-now are asymmetric.
Are there any proven, even weak, signs of intelligent space-traveling aliens? If so, then yes we should work on that too.
The leaderboards above and pervasive computer vision and equity trading systems are some signs.
Some people claim to have seen intelligent space-travelling aliens. Personally I think they're nuts, but it's impossible to absolutely prove that their claims are false.
There are obvious disadvantages to the current machine approaches, but they nonetheless achieve very good results and beat the vast majority of players of those games.
Real-world relevance is orthogonal to complexity.
Edit: We don’t know the best programs, but we can approximate the bounds with some confidence that, unless some paradigm-changing discovery is made, the programs to solve digit multiplication would be significantly simpler. (This is not a formal argument, but the intuition is very strong for most people who study the problems seriously.)
Kolmogorov complexity is a distinct quantity. How did you compute this quantity?
If you’re a purist, you write the programs in MIX and you count ops. Or whatever your preferred Turing Machine is.
I know that sounds pedantic and this is the internet and not a scientific paper etc, but if this is an informal discussion then formal terminology like "Kolmogorov complexity" has no place in it and the purpose of the OP would, I think, be better served by using an informal term. If you 're gonna use maths, use maths. If you 're gonna wave hands, then wave hands, by all means. But don't mix the two, it just makes a mess.
Because at the end of the day I can just say, "no, I don't believe that multiplication has smaller Kolmogorov complexity than Starcraft's pathfinding"... and then, what? Where do we go from there? The OP says "yes it does", I say "no it doesn't", the OP says "does too", I say "nuh-huh". And we just waste our time.
> yes, I can assure you the Kolmogorov complexity ...
This is a strong comparative statement, and it's clear that it's an assertion but not a proof. So it's perfectly fine for you to say "I don't believe that" and then you would give your reasoning. Of course the conversation didn't go that way because it's pretty obvious to everyone that this is actually true even though we don't have a proof. And it's extremely likely that if you could find a minimal multiplication algorithm and enumerate all the Turing machines up to that length you could prove that none of them solve the harder problem. This is where the intuition comes from.
> using an informal term
But the OPs argument wasn't that one program would happen to be longer, but rather that it must by necessity be longer, because the problem is fundamentally more complex, which is exactly what the formal concept was created to capture.
> Kolmogorov complexity is a formal concept that does not admit intuitive judgements
This is a bad way of thinking about maths. The value of formal concepts is precisely that they give us tools for making intuitive judgements more effectively and give us a wider variety of tools to manipulate and express our intuition. As intuitions become conjectures and theorems, we accumulate new bedrock we can use for new intuitions, but don't short-circuit your thinking by saying that intuition about technical terms is off-limits. That's literally what they are good for.
I don't agree. I don't need formal terms to develop intuition. Formal terms are necessary when speaking formally, and they are only detracting from the point of a discussion otherwise.
And I don't agree with this either:
>> But the OPs argument wasn't that one program would happen to be longer, but rather that it must by necessity be longer, because the problem is fundamentally more complex, which is exactly what the formal concept was created to capture.
The way I understand it the formal concept is useful in proofs of the impossibility to calculate various other quantitites. Its utility is not to argue that this or that program should intuitively have lower Kolmogorov complexity.
To be honest, I'm so confused by the use of the term by the OP that I have no idea what they meant to say with it.
OP was simply saying that, in any general-purpose programming language (where all possible programs are expressible and none are unfairly favored) the solution to problem B will inevitably be longer than problem A. This is quite a bit more precise than just saying "a more complicated problem", and if you have the intuition behind it, there's no reason not to use the term.
Right, limitations like: width of human mothers' hips limiting head size, inability to continue growing the brain after adulthood, the decades-long education process, the impossibility of doing repeated experiments of different teaching techniques on the same people and hence iterating quickly and accurately, the impossibility of giving one person's intelligence and knowledge directly to anyone else, old age and death limiting how much a single person can learn or accomplish...
Once you have a computer Einstein, you can immediately have 100 computer Einsteins (given the hardware), and let them work on different projects (or even let some of them collaborate). That fact alone, while not a singularity, is at least a game-changer in terms of innovation. I have gotten annoyed at science fiction scenarios that allow for duplication of adults and don't answer the question "Ok, so, why hasn't this society been duplicating their best scientists and out-inventing the rest of the galaxy?"
I think that, if there is room for skepticism, that has to be at the "Can we get to a computer Einstein?" stage.
Given that, I don't see an immediate reason to believe throwing multiple AIs at a problem is going to exponentially scale either, given neither people nor bad AIs scale that way.
Those aren't actually limitations preventing us from becoming godlike. Those are limitations on us becoming more populous and collectively more experienced, which I argue isn't the same thing at all.
Apotheosis in the context of AI safety as well as for humankind is really about the accumulation of power, rather than the accumulation of knowledge. The extent of our apotheosis is limited only by what we can control, not by what we know.
Knowledge and intelligence only allow you to reason more effectively about paths to power given enough information about the environment, but it doesn't give you a strategic guarantee that you will acquire power - we know there are games that cannot be solved, and games that take too long to solve efficiently, and games that, even with sufficient computing power, rely on factors that cannot be controlled (such as luck), and games for which there are no Nash equilibria or stable winning strategies.
The real world is full of such games. Computers have only managed to beat humans at a small fraction of them, and even then only ones where brute-force and smart pruning (optimal game space search) are winning strategies - bots for games that require probabilistic reasoning, such as poker, typically don't fare any better against expert humans, largely because efficiently solving probabilistic games in general is (per a hazy seminar of AI many years ago, so I could be misremembering) an NP-hard problem.
There are plenty of both natural and artificial limitations around the accumulation of power besides game solvability and probabilistic reasoning's difficulty:
1) System designs that require a minimum set of resources that intelligences can't acquire. For example, individual humans usually can't build their own thermonuclear devices because they don't have access to weapons-grade uranium. Much of our information security relies on making it computationally expensive and infeasible for an attacker without state-level resources to break in, and getting state-level resources isn't a straightforward task at all. We humans haven't even gotten around to building our Dyson spheres yet, because the resources needed to do it far outstrip our ability to acquire them. Intelligence does not improve resource acquisition by itself.
2) Impossibility theorems and optimization difficulty. These place fundamental limits on power - as an example, we know Moore's law has to stop at some point because increasing chips increases power generation, and the amount of effort needed to generate such chips increases with every nanometer shaved off. Most of humanity's most difficult and important problems are optimization problems, and these optimization problems can be tricky to solve in polynomial time. In many cases, we have to rely on fast crude heuristics that give us less than optimal results.
The above limitations partly become easier if you try to parallelize your efforts by duplicating AIs (you can collude to acquire resources and solve problems), but this means you get all the fun parts of distributed systems theory for free: coordination, replication consistency, etc. If only all problems were embarassingly parallel! So it's not necessarily easier to add nodes to a knowledge graph just by adding more processors.
The peoples that worked bronze prevailed over the peoples that just used stones and sticks. Advances in AI are no different than that.
If you can automate production and defense systems, the natural consequence is that a few people will conspire against the rest and dominate them.
If you could grow your own brain as large as you wanted, and live as long as you wanted with the physical robustness and energy of a 25-year-old, that alone seems like a road to godhood, though perhaps a slow one. If you could download knowledge directly, as another commenter suggests, that becomes a much faster road.
> Apotheosis in the context of AI safety as well as for humankind is really about the accumulation of power, rather than the accumulation of knowledge. The extent of our apotheosis is limited only by what we can control, not by what we know.
If one country, even a smallish one, has 50-100 additional years of technological advancement, I think they might be able to use this knowledge to gain a lot more power. I'm pretty sure history has borne this out. Do you think we're at the limit of what new technology can do for military strength? Materials science (armor, weapons), biotechnology, energy and battery technology, cybersecurity, psychology and propaganda, lasers and anti-missile defense systems...
Suppose a country like North Korea had 100 Einsteins working on such things for a few years under the radar. I suppose it might be difficult for them to hide the construction of their army of thousands of robots invincible to all but the most powerful bombs... but then again, maybe they could do it underground, directly underneath some large innocuous manufacturing operation. And, of course, the cyber- and bio-weapons efforts would be very easy to conceal.
It's even debatable if you can grow an Einstein. What if GI is NP-hard? What if the belief that GI - never mind consciousness - can be modelled with our version of computing hardware is just plain wrong? What if a GI System is uncopyable for practical or philosophical reasons?
The singularity looks more like plain old religious thinking than anything that could be recognised as real science.
"What if we made a golem? We could do anything! What if we put ourselves into the golem? We could live forever!"
And so on.
Even if there would be true generalized Artificial intelligence, it is not unreasonable to think that the formation of the right thought framework for those systems to function effectively will require a substantial amount of time. It is also reasonable to think that such change would probably be in symbiosis with human intelligence. This change would not be sudden, but happen over many generations. It is my opinion that this change has already started at least a generation ago.
Who is disputing that? I am 99.9% certain that this is true.
The point of the singularity isn't that AI is coming next summer -- it's that from one developmental point and on the intelligence and abilities explode exponentially. To get to that stage though, thousands of years might pass.
> It is also reasonable to think that such change would probably be in symbiosis with human intelligence.
Now that is not likely. At all. They have to have the same motivation, interests, emotions and needs like us. The chances of an artificial race being developed to cover all these at once are practically zero.
A race not motivated by the same things we are is practically impossible to sympathise with our own motives.
Well, the general horror story that is being told is that intelligence and the ability to use it go hand in hand. As soon as machines surpass our intelligence, all hell breaks loose and there is no turning back. The focus is very much on the individual (both human and machine) and not on the structure.
> They have to have the same motivation, interests, emotions and needs like us.
Here is where I need some more convincing. In my view, nature is full of systems and groups of species living in symbiosis. Due to the slow development of the thought framework, we will co-evolve with the machines, or become part machine ourselves. In a certain way, we already are.
Perhaps I don't really have the word for it and I am sorry for that. Our understanding of intelligence, whether mechanistic or biological, is mostly as points on a linear scale (IQ). I think this misses the big picture: we are not biological systems, but have lived in symbiosis with our child teknos since we made our first tools. In this regard, there is no subject nor object: they are one. I am fundamentally with Heidegger on this one.
In literature and popular culture people conflate concepts all the time. :) The way I always understood that horror story was that generic AIs get schooled pretty brutally and quickly and they learn to use anything they have access to -- thus materialising that popular horror story. I think that's not far fetched, having in mind that the organisations that are very serious about obtaining a true AI are mostly corporations and military.
> Here is where I need some more convincing.
Maybe here I got influenced by popular culture. The books and movies I've read and watched mostly setup the generic AIs in situations where they have to be either parasites (Hyperiond / Endymion books) or commandeer a lot of tech and personnel (entrusted to them, like Skynet in the Terminator franchise) for their own ends.
But to me, the mere fact that a future generic AI doesn't have body and emotions like our own already means it's not going to be at all motivated by the same things as us. And we might even serve some pretty grim purposes to it, like the robots recharging themselves with organic matter in the PS4 game "Horizon: Zero Dawn" (which of course leads to consumption and thus extinction of all life).
Meaning, an AI might just view us as a natural resource to be exploited, and nothing more.
Tell that to Homo erectus. Yeah, whales don't have advanced technology, but they didn't evolve facility with opposable thumbs to manipulate their environment.
> You can't grow your brain as large as you want. There are physical limits on power consumption, power dissipation, and organisational complexity.
Whales show that biological brains could be 6x larger. As for electronic brains, I don't know how large the initial Einstein-brain would be. But if delays from Ethernet interconnect are permissible for intra-brain activity, then your brain could occupy a large datacenter.
> It's even debatable if you can grow an Einstein. What if GI is NP-hard?
I did state that "getting to a computer Einstein" was where skepticism might apply. Nevertheless, nature managed to do it by accident. Are humans NP-hard? They exist. Obviously they can be implemented somehow. You don't need to simulate a particular existing human perfectly; it just needs to be close enough to the way a human brain could work.
If you need an initial state for the brain, maybe you could create an artificial environment (i.e. with only machines, for easy simulation) that a hundred actual humans grew up in to the point where they developed language, verify that those humans later became functional adults; and then simulate an embryo growing up in that environment as well, and start evaluating the results and iterating. Kind of fun to think about.
> "What if we made a golem? We could do anything! What if we put ourselves into the golem? We could live forever!"
The data that computers contain is easy to copy—by design. I think this will be a property of all computers in the future, except possibly quantum computers that always keep lots of data in a superposition (or computers that, for security reasons, are designed to be uncopyable)—and none of the existing neural nets are quantum computers. So copying plus backups or RAID is approximately guaranteed to cover the "live forever" part. Most of what I've said is based solely on the assumption that we develop a computer program that simulates an Einstein, plus common facts about computer programs.
I for one fully agree. I am pretty sure that whatever true generic AI emerges in the future will definitely not share brain schemata with us.
It seems to me that you attack the idea of the AI singularity in a very narrow framework: human-like intelligence, human-like hardware, copyable consciousness / intelligence, etc.
What if none of those restrictions apply? (And I am pretty sure they don't apply indeed.)
I agree that singularity shouldn't become a religion. But you are too quick to discount the possibilities of an artificial mind with more processing power and more efficient algorithms (compared to our brains').
Assuming GI is possible and you give it a robot body and then the body breaks after 20 years, how long will it take to get used to the new body?
A larger brain don't necessarily provide a better survival advantage. It's well known that average Homo Neanderthalensis had larger brains than our modern average follow humans. And if you don't survive, you are nowhere toward the road to thaumaturgic existence.
What is the trade off is the most important question, as anywhere.
It is also a huge assumption that 'hostile' to humans AIs would be cooperative between themselves. In resource contention situation especially..
Does it really not though? If you watched the movie "Transcendence" you'd presume that a sufficiently [initially] sophisticated AI can decipher our financial markets quite snappily and make a very quick money off of that -- and use those to finance its endeavours. Which is exactly what the AI in this movie did, very successfully too.
My point is, given a much larger and better brain, the intelligence within it can optimise a much larger problem space at once, and that can very easily include resource acquisition as well.
If intelligence as we know it could ramp up to generally solve all problems, then surely evolution would keep increasing it and we'd all be gods or at least Einsteins by now?
Depends. If a future AI wants to make a civilisation that spans our own entire Solar System then it would have to tackle much more -- in number and complexity -- logistical, religious, political and what-have-you problems. So it might be too much for it as certain problems here on Earth are for us the humans.
However, if we postulate a self-aware AI who is Earth-bound and is having more processing power than a single human brain and has much more efficient algorithms than our own then it really easy follows that this AI would be much much better than us in solving our Earth-bound problems. Like in economy: more income and same expenses equal bigger profit.
> If intelligence as we know it could ramp up to generally solve all problems, then surely evolution would keep increasing it and we'd all be gods or at least Einsteins by now?
Well, we are limited by the physical size of our brains being able to squeeze out of our mother's hips during birth. Same way a future AI would, at least initially, be limited by the available hardware on which it can run, how much power it can consume and if its heat dissipation won't become too much.
However, an AI wouldn't grow in a natural environment, namely where natural limits would hamper an infinite growth. IMO if a private well-funded lab -- or the military -- invest heavily in such an AI they would just keep throwing hardware at it until it becomes efficient enough to be compared to a human and above. From that point on it is very likely it would become smart and proficient enough to optimise itself and start solving age-old problems we never were able to solve so far.
This, to me at least, is the so-called singularity point, in less extreme terms compared to its official meaning. Namely an entity that can optimise itself and achieve engineering perfection much better than what we can do.
I don't think that's something that's clear at all. Men and whales have larger brains than women, on average. Are they proportionally smarter? Without knowing how brains work, it's impossible to say what constraints exist due to size.
A computer can potentially communicate much faster, and can transmit "thoughts" much more accurately than a human can. Imagine you could serialize a neural ensemble in your brain and send it to another person, so that person is then immediately able to use it, instantly acquiring a skill. That alone is a huge advantage over a human being.
You can't take half the nodes from a neural network that recognises dogs and add them to a neural network that knows about laundry to get a network that knows how to wash dogs ;)
You might even get >100 times speedup, since there are probably problems where 10 copies of of the same mind can collaborate better than 10 different minds, e.g. I bet 10 copies of me could collaborate writing a program better than 10 random programmers. (I base this on I can read my own code from long ago a lot easier than most other people's code).
Yes but only slightly. I can read code I wrote 10 years ago, that I have totally forgotten about since the way-of-thinking behind the code comes easily to my mind.
How can you be sure of this?
>Growth curves are always S-Curves. Always.
But we don't know in what place of S - Curve is AI development and what are it's limits.
The question is however, where the plateau lies. Perhaps it's so far off that for practical purposes it doesn't really matter that there is an upper bound at all.
Artificial movers are almost certainly going to run into the same limitations that prevent natural movers (i.e. animals) from becoming very fast.
Since the fastest animals can travel at about 70 mph on the surface of the earth, and 100 mph in level flight, it follows that we will never be able to make artificial movers travel faster, no matter how hard movement engineers try.
Moore's law is an example. It did taper off, eventually.
Also the statement presumes that self preservation is the super intelligent thing to do.
I gave it this test:
https://www.k5learning.com/reading-comprehension-worksheets/...
A few caveats. I had to prompt a little extra on the 2nd question and give it a 'Because' to start off. It kept wanting to fill underscores as if it was a blank spot for an answer. There's a way to prevent this if I had time. The third question is "wrong". The actual answer is 'It would have become dark and cold if Ann and Frank had stayed later.', but I'd argue that it's rather open ended question and the answer is something that might have happened. The final question is really what makes this a 3rd grade comprehension test, it's asking the reader infer something not actually in the story, about how they would feel if they were Rover, something the AI is woefully unqualified for.
Question: What did Rover fetch in the water? Answer: The Stick!
Question: Why couldn’t Rover catch the rabbit? Answer: Because the rabbit ran so fast he could not catch it.
Question: What might have happened if Ann and Frank stayed later? Answer: Rover could have learned that not all rabbits love to play with sticks.
Question: How do you think Rover felt about not catching the rabbit? Answer: He was sore and he must have had fun later because he went in the back yard and hid out in the trees.
Yes
Yudkowsky has an amazing talk on this issue. I don't recommend many talks but this one is very interesting and introductory: https://www.youtube.com/watch?v=EUjc1WuyPT8
Until we have built a framework to stop these things from happening again, what hope do we have against intelligences that are to human organizations what human organizations are to individuals?
Even if you don't like my particular examples I can highlight a dozen more problems AI software has created or exacerbated right now. Why all the focus on hypothetical problems?
AI is making things better.
Perhaps what's really needed is ethical standards for technology. This would have the added advantage of being applicable even if the technology doesn't feel like AI to some people.
It is simply that I see zero evidence that the Nick Bostron, Lesswrong, and etc. school of thought provides any insight in regard to these problems. The thing the "AI might become autonomous" school doesn't seem at all interest in the processes of human bureaucracy or in even AI as it exist now but rather views as simply a god, devil or Genie which grants wishes or damns to hell. If anything, the approach seems counter-productive.
Let’s stop Nick Bostrom and a load of AI experts from doing their current work, put them all on a plane and send them to China to solve political oppression. I’m sure that will work.
Do you believe that attention is an infinite resource?
Because many communist regiemes already found out how they are not with command economic failures. Even if the specialization is misapplied reaiming them takes time - like how you start with training a battlefield useful longbowman's grandfather.
I'm ok with people working on both. I just feel so much less attention is being placed on these more immediate problems. Tesla has deployed to production software for driving that is unable to detect pedestrians. So at least Tesla can't do both.
I see it quite differently. raising the long term issues is absolutely part of highlighting the risks of these immediate issues. They are part of the same problem, so raising their profile is synergistic, not competitive.
We are becoming more and more reliant on AI in situations that formerly required human judgement. And we like the systems that rely on such AI. We like them so much that these systems become very popular, used by millions and even billions of people. Scalability demands AI solutions. What if we don't like what the AI is doing? Do we turn off that system? Do we disable the AI and rely on human judgement again? (Where would we get all the employees?) Do we tweak the AI? That last option seems like the most palatable one, but each time we tweak the AI, we are subjecting ourselves to new unforeseen consequences. It's like the genie gives us three wishes, we get through them, disappointed each time, and then he gives us more wishes. And all we can do is not repeat our previous mistakes, while we make new ones.
To make this concrete: Imagine Facebook subjected again to Russian influence of US elections. Suppose Facebook actually does get serious about reigning in this influence. They deploy AI to do so. First of all, it's an evolutionary arms race between the AI and the Russian influencers. Second, we really do have to worry about the AI producing bad results.
[1] https://www.telegraph.co.uk/news/2019/09/27/ai-facial-recogn...
Currently the biggest danger from AI is from applying it where it's not as smart as you think it is.
The author is missing or ignoring key contributors of work in this field. He even mentions Nick Bostrom, who works with the FHI which does work in this space [0]. The author's own organization does work in this space [1]. Deepmind does work in this space [2]. "Safe artificial general intelligence" is the literal mission statement for OpenAI [3]. The author may feel more conversation is needed, but it feels disingenuous to suggest that nobody is talking about this.
As for how much conversation should be happening, my understanding is that most people on the edge of the field view the current risks of existing artificial intelligence as significantly more pressing. We have the ability right now to create dangerous weapons & massive facial recognition programs with current AI. The longterm affects of biases in ML algorithms are still not well understood. Some of this is already affecting us [4].
When it's not clear at all how we approach general intelligence, and many experts fundamentally believe that our current approaches will not suffice ever, I'd argue the current level of focus on general intelligence safety is about where it should be. Smart people are thinking about this, but there is also important issues now to think about.
[0]: https://www.fhi.ox.ac.uk/research/research-areas/#1513087763...
[1]: https://humancompatible.ai/publications
[2]: https://deepmind.com/research?filters=%7B%22tags%22:%5B%22Sa...
[3]: https://openai.com/
[4]: https://www.nytimes.com/2019/04/14/technology/china-surveill...
While we worry about all kids of things at the macro level microscopic organism continue to evolve. Any given Monday the blind clock maker could deliver an invisible bug right into our laps, one for which we will have no defenses.
Our interconnected world will mean this bug will easily jump population centers.
The devastation we could see if such an event occurred could be of unprecedented scale and breath. We could lose one quarter or more of the population of this planet given the right conditions.
So, yeah, AI could, I guess, go rogue. Yet the real threat isn't anywhere near what we can see and touch.
In fact, we might NEED real AI to save humanity when and if that super-bug emerges. Think about that for a moment.
https://www.nytimes.com/2019/03/28/science/frogs-fungus-bd.h...
https://en.wikipedia.org/wiki/Chytridiomycosis
Here's what we know:
- Evolution is real (stating the obvious just in case there is a question)
- Natural selection survives organism that are able to deal with the environment within which they have to exist
- Antibiotic abuse is rampant (again, stating the obvious)
- Such abuse provides the mechanism though which evolution will be able to create the aforementioned super-bug
- Once that individual emerges our defenses will be ineffective
- The world is massively connected, which means that super-bugs can travel from population to population like never in history
- Stopping super-bugs will require such measures as stopping society as we know it from functioning (no travel, local or international) and maybe even having no choice but to quarantine and allow a population (of humans) to die in the interest of stopping transmission. If a bug has a 75% kill rate and we don't have a cure, do you allow anyone from the affected group to be in contact with anyone else?
So, yeah, is it possible? Sure. Probable? I guess. I am not sure there's a deterministic function one can use to predict the outcome of evolution, even at bacterial levels. My guess is it could take a thousand years or it could happen next Monday. I don't know.
Why wouldn't we go through phases of increasing intelligence (along multiple dimensions) while working towards an intelligent AI? Yes, someday we may produce an AI that is leaps and bounds more intelligent than humans, but before that we will have experience dealing with various human-level and sub-human-level computer intelligences. The idea that AIs will be self-improving is kind of ridiculous -- there are lots of intelligent people, and the idea that an intelligent AI will happen to be good at designing AIs is negligibly small, much less that they will have some kind of immediate breakthrough that enables a higher level of intelligence.
It's not obvious that the path from moderate-intelligent AI to super-intelligent AI is a matter of "just adding more memory and compute"; intelligence as we know it relies too much on associative memory to think that a simple capacity increase will elevate beyond a certain point. If the intelligence takes a different form, then it's hard to make concrete statements about in any capacity, but ultimately the path will be started on by humans, or possibly emerge from sufficiently complex systems.
Either way, we're so far out from this that it is laughable to speculate. When we can simulate the intelligence of a cockroach, then maybe we can start to think about what human-level intelligence will look like. The most likely case is that at some point someone creates a system that is arguably conscious or intelligent in a meaningful way, and then we will devolve back into discussions of consciousness and identity and memory while we try to figure out what it means to think about intelligence now that we have something to benchmark against. I wrote this comment [1], that I still enjoy, in response to a similarly alarmist narrative; posed as fiction in that case rather than masquerading as journalism as in this one.
[1] https://news.ycombinator.com/item?id=15420699
> Building an accurate emulation of the fly brain is an interdisciplinary effort that requires data, algorithms, and insight from (but not limited to) the fields of neuroscience, computer science, and systems engineering. Researchers interested in working towards this goal are invited to join the Neurokernel project.
[1] https://neurokernel.github.io/about.html
[1] http://openworm.org/
Curious if a deep learning application that builds neural networks from gene sequences is a possibility one day. All the information is there, but it may not be possible to learn what it builds without actually "running" it.
Both are great topics for a epic book.
Had this discussion with my relative recently and recommend him to stop watching YouTube and start reading books on Introduction to ML.
“AI research is making great strides toward its long-term goal of human-level or superhuman intelligent machines.”
Hmmm, really? What examples? Show me a machine that can do what a gnat does and we’ll talk.
Otherwise replicating and leasing would rake in far more than even a magic "I get everything right on the stock market" button in the same way ruling stone age tribe gives less absolute wealth than a Manhattan Lawyer.
This is obviously incorrect. If an AI system of superhuman mental capabilities is installed on a system with no network connection and no physical interfaces, we can completely control the harm is might create. It would be unable to directly act, of course, requiring humans to authorize and carry out whatever plan it devises. But failing to do this should be prosecuted and treated no differently from if a person or company built a tank and then permitted it to drive over and through people. The biggest difficulty here will be that our legal frameworks have no established way to assign liability and criminal culpability to any system involving software. With no legally-enforceable industry standards (like the electrical code followed by electricians, things like that) any prosecution fails as the company can either claim ignorance of the potential harm their system might cause or simply throw individual employees under the bus, claiming they were not acting on direct orders.
It is true that we can not fully specify the goal, and that is a grave concern. It's partially due to this that AI systems shouldn't be directly connected to the ability to act in the physical world unless their attempted actions go through some sort of 'filtering' first.
First, why would someone build a system like that, and how would it actually be 'smart'. If you lock an infant in a closed room, it will grow up to be a literal retard. Intelligence requires interaction and data. Next, some idiot will hook it to the net in a heart beat when they figure out they can use it to make money on the stock market. Never underestimate human greed.
Of course, a weak AI may be able to write syntactically valid code that doesn't necessarily work. That's... not much of a breakthrough.
Already, conventional algorithms are trusted in society, sometimes beyond reasonable or safe levels. The ethical questions posed by ethical AI design and deployment are exactly the ones our present-day reliance on (possibly poorly-designed) algorithms pose.
Before we worry about AI harming us by an improper inference, we can, in the present moment, already worry about automated systems harming us by improper inferences (e.g. improper financial models, Tesla / aircraft driving assistance); the discussion on how to limit and selectively deploy AI is no different than that for present-day algorithms.
The same research that prevents little problems should hopefully prevent big problems. The research to take RL to the real world is progressing quickly. I don't see a "kill us all" AI arriving soon, but "do bad things" AI and "wreck the economy" AI are quickly approaching.
The author simply invents futures to refute various arguments (all presented out-of-conext, so who knows if there's any real relevancy from that direction either).
What we might do about a super-inteligent AI would strongly depend on what that is, exactly, which, of course, no one knows or really has any idea.
So it's OK to casually BS about this stuff, but in terms of substantive discussion, much less action? No.
What happens when someone develops a system that measurably makes more profitable decisions than 70% of CEOs? That isn't totally out of reach. Capital would flow to companies run by such systems. We could end up with machine learning systems running the corporate world, based purely on better ROI.
A good start on the problem would be to train on a collection of funding proposals, such as old Kickstarter announcements. Look deeper than the hype. Go out and suck up all the information you can on the founders, using credit databases. (They're asking for money, so it's a legit credit query). Do an automated background check. Then check on how the project did, five years later. That's your training set.