I am likely mistaken, but my impression was that that quote was after arguing that we cannot really refer to certain things in a coherent way, and that the quote advocates that we therefore do not attempt to do so. How far off am I?
My understanding is that Wittgenstein likely had metaphysics and certain philosophical problems in mind. I don't know what he would have to say concerning speculation about super AIs and their potential capabilities.
Fair point. Although, there are some key differences. We have a trend of machines with increasingly intelligent capabilities. So we must ask: Is it reasonable to extrapolate? How far? What are the risks? Difficult questions.
I'm personally not terribly worried, as I think we have a ways to go before creating human-like general intelligence, let alone super intelligence. It is also not clear at all to me that an exponential increase in intelligence is likely or feasible after some threshold. Still,
treading carefully seems prudent. There are some serious risks, and even if the likelihood appears low, there are a lot of unknowns.
First time encountering a Maciej idlewords post, huh? He doesn't even take himself seriously. That also doesn't stop him from being an excellent and thoughtful writer.
An extremely observant existentialist-idealist caught up in the tech world and its intelligence-vs-humanity theorist ideas. Poor guy. This happens a lot and we get some really amazing things from such gifted thinkers, for example Larry Wall's literary thinking about code. But I can also see where the annoyance comes from. AI theorizing requires huge leaps of intuition which can immediately supersede the existential in one's own mind, because you're thinking 200 or 2000 steps ahead, seeing the crises, seeing the opportunities, and pretty soon you have a level of convergent certainty which is hard to explain step by step, because you yourself don't really think in that step by step way, but rather in leaps. But Maciej _does_ need to think that way, especially regarding far-future concepts. And now you the AI-guy have both the burden of certainty and the energy to try to prioritize group attentions toward things you can see, that Maciej can't. IMO it's an important difference in psychology. And Maciej makes Maciej-like points: Don't sacrifice current generations' humanity for progress toward a "better" or "saved" future, when there are in fact many possible outcomes (this is key--divergence where others see convergence), and a pressing need to act on identified, easily-sensed problems, right now.
The question I have is, how much of this concern on Maciej's part is "wooly" extrapolation or the same kind that Maciej is complaining about? Much of what I just read also makes intuitive leaps, albeit of a different mindset. They are leaps of human judgment, leaps to fear the nature of those who "feel" inhumane. This is where I wonder if some more nuance and back-and-forth communication would be more helpful than a presentation.
I don't mean to say "he made a Rick and Morty reference, Serious Adults don't make Rick and Morty references". That'd be dumb. I mean that he points to a robot in Rick and Morty that was sad about being programmed to pass butter, and attempts to infer from this that real robots would refuse to intelligently pursue arbitrary goals, and therefore that the orthogonality thesis is false. He's treating a thing that happened in fiction as though it was a real thing that happened somewhere that we should treat as evidence. That doesn't work.
I think in this case it might be you who is mistreating what's written. This also produces the same not-working result but for somewhat different reasons.
The serious statement he makes is that humans have complex motivations, and he thinks any superintelligence would as well--in fact it might be a defining feature of intelligence. The Rick and Morty reference was just a humorous example of what might happen, not an actual argument. The subheading was "the argument from complex motivations", not "the argument from Rick and Morty", so let's try and focus on the serious part and not the humorous aside.
But if not, and the alarmists are right, deep shit. That's why the author's argument is void. The author needs to show that the chance of an AI with a simple goal and hyperintelligence is negligible. "it might be" is a coin toss, that doesn't help.
But the orthogonality thesis isn't just some idea that may or not be true. It follows directly from Hume's Guillotine, discovered in 1739: no amount of superintelligent reasoning about facts will allow you to derive goals.
No artificial intelligence will ever spontaneously develop morals, because the questions "what can I do" and "what should I do" are eternally separated by Hume's guillotine.
The motivations will always have to be provided by the people who make the machine, and we have been seen in the past that artificial intelligences are very good at finding loopholes in their moral code. Here are some real examples: https://docs.google.com/spreadsheets/u/1/d/e/2PACX-1vRPiprOa...
The problem is that the thesis cannot be disproved by one counter example (real or not), because the existential hazard comes from the possibility of just one such instance of an AI.
> Consider, for example, an old doctor; suppose they’ve seen twenty patients a day for 250 workdays over the course of twenty years. That works out to 100,000 patient visits, which seems to be roughly the number of people that interact with the UK’s NHS in 3.6 hours. If we train a machine learning doctor system on a year’s worth of NHS data, that would be the equivalent of fifty thousand years of medical experience, all gained over the course of a single year.
Doctoring is just like playing Go, right? Just increase the CPU cycles on it and pack more data in there, it's more or less the same.
You can make that assumption but I don't think it'd be based in fact or reality, because Go has far fewer inputs and states than treating humans does. And you don't get to test every hypothesis and then rewind either.
Most answers there are like this, making unfounded conclusions into supposedly insightful rebuttals.
Let's try another.
> If we then take into account the fact that whenever one Einstein has an insight or learns a new skill, that can be rapidly transmitted to all other nodes, the fact that these Einsteins can spin up fully-trained forks whenever they acquire new computing power, and the fact that the Einsteins can use all of humanity’s accumulated knowledge as a starting point, the server farm begins to sound rather formidable.
Or maybe they tell each other fake news so quickly they can't tell what's right and what's wrong, like we do whenever we find a more efficient way to communicate?
Anyone can make unfounded assumptions about anything; I just did it. It's up to you to decide if you care whether these assumptions are based in reality or not. But if you consider yourself "rational", I think it'd be in your best interest to care.
Well the assumption is that AI will be able to learn similar amounts of knowledge from access to the same amount of data as a human would. That is of course totally wrong for almost all problems for today's algorithms, but that might change in the future. Alpha Go for example improved a lot by playing against itself without outside input.
I honestly don't think anything in that post 'demolishes' the criticism or even advances some sort of argument.
It's just a huge wall of text full of weird analogies which is quite typical for these 'rationalist' community posts.
People like Bostrom or Yudkowsky have one thing in common. They are not engineers and they stand to gain financially (in fact it is what pays their bills) to conjure up non-scientific pie in the sky scenarios about artificial intelligence.
In Bostrom's case this goes much further, he has given this treatment to anything including nuclear energy and related fields. Andrew Ng put it quite succinctly. Worrying about this stuff is like worrying about overpopulation on Mars, and there's maybe need for one or two people in the world to work on this.
I really wish we could stop giving so much room to this because it makes engineers as a community looks like a bunch of cultists.
So you agree there is room for them to work on this, yet you feel they are making engineers generally look like cultists?
Maybe you’re just being oversensitive. The hype wave on AI danger is completely over, and there’s nothing wrong with people studying the question if that’s their interest.
You know we've been here before, right? I mean, lighthill report, Ray Kurzeweil is a serial offender for over thirty years, the singularity is around the corner thing, outrageous claims for fMRI, self driving cars. Over hyped ibm Watson which now health professions are talking about misdiagnosis problems.
Sure. We have google image match and better colorisartion and some improvements in language processing, and good cancer detection on x-rays. These are huge. But hype is, alas, making engineering increments look like cult.
> Worrying about this stuff is like worrying about overpopulation on Mars, and there's maybe need for one or two people in the world to work on this.
When something is shown to be doable in principle, it's often not clear how difficult it will be in practice.
In 1932, Ernest Rutherford thought nuclear energy would not be a viable source of energy, let alone weaponized. In 1933, Leo Szilard filed a patent on the concept of the neutron-induced nuclear chain reaction. At this point in time, nuclear fission was not yet known and actually making nuclear energy viable was a pipe dream.
As we all know, in 1945, the first nuclear weapons were used. Until the day that the weapons were used, German physicists thought that nuclear weapons would not be used in the war because, while possible in principle, the actual construction of a working device would require a herculean effort that no nation would expend in time for the war.
The German physicists weren't too far off in estimating how difficult nuclear weapons were. They just failed to predict that the US would throw 130,000 people, including most of their top minds, at the problem for years.
Now, we have no idea how difficult superintelligence will be. But the possibility that we're a couple of breakthroughs and a Manhattan project away from superintelligence is real, and I want a hell of a lot more than one philosopher and an eccentric fanfic writer working on this.
EDIT: No offense to Yudkowsky. I thought the fanfic was fairly good and, more importantly, achieved its purpose.
That's a phase space diagram, used for analyzing complex systems. Quite a handy tool for dissecting which dimensions are important, which ones you can simply collapse into one yet keep the interesting dynamics, etc. (These are [were] used for understanding neurons and neuron models. The ion channels, the electric potential, and so on were the dimensions.)
> The second premise is that the brain is an ordinary configuration of matter, albeit an extraordinarily complicated one.
This is the part I doubt is true. There was a time when humans thought brains were very complex systems of microscopic clockwork because gears were the pinnacle of technology.
How do people keep mistaking pattern matching with intelligence?
I believe chess prodigies were supposed to derive their superiority by inference but studies have shown that a large portion of that superiority was due to superior pattern matching.
Learning chess recently, I think it’s a terrible example for AI. I don’t understand why it was ever targeted or trumpeted. To become good at chess you have to memorise the responses to tons of situations, simple pattern matching. It seems to me that chess is more about memory than about reasoning, in the same surprising way that scrabble is more about maths than words.
Because we have things like Quantum Field Theory that tells us the energy scales it's valid for, and it's experimentally checked to be valid for those scales.
Since the brain doesn't presumably contain supernovas, black holes or particle accelerators, that means that we have a basic theory covering the mechanics down.
One thing we do know is that we don't yet fully comprehend how the brain works. It's a little early to say it contains nothing we cannot simulate. Quantum interactions do not require high energy physics.
> It's a little early to say it contains nothing we cannot simulate.
Suggesting that AIs have to look or behave like human brains in order to have human level intelligence is like suggesting that airplanes must flap their wings in order to fly. It seems far more plausible to me that any AI we build will be as alien to our way of thinking as a F-15 Eagle is to a bald eagle. Given that, I think it's important that we think about how we communicate with this AI, and how we ensure that it follows what we command it to do, rather than going off in its own direction, with negative consequences for human civilization.
I think that the negative consequences of current AI should be warning enough that we need to worry about future, more capable AI.
Sean Carroll explaining why "the laws of physics underlying everyday life are completely understood". (renormalization group, pair production at human energy scales)
The threshold is protein folding. Anything lower level than that is noise, but folding itself is important, as it can go wrong sometimes and kill you. (Prions.)
And things can be approximated, sometimes very close to the real thing. Hence the connectome. Synapses are the interconnects and neurons are tiny networks of very simple computers on their own.
And it's not "just" pattern matching, it's very very likely Turing complete. (You can do counting networks, you can do conditionals, you don't really need anything else to learn, match, execute simple programs.)
We know enough to claim that it's increasingly unlikely that we are missing any relevant physical phenomena for understanding the brain and the mind. (So no quantum woo is required.)
Sure, it that unlikeliness is not zero, and it might turn out that some kind of quantum effect is important for some very strange step in the workings of the brain, but even then it's very likely that we can classically model it with good approximation. (Because as others mentioned, energy scales, and other fundamental things constrain any process going on there.)
>There was a time when humans thought brains were very complex systems of microscopic clockwork because gears were the pinnacle of technology.
In a general sense, microscopic clockwork isn't an incorrect way to put it. Any known chemical or biological reaction is just some microscopic physical reaction. We know about more subatomic structures but I think the crux of the argument is about the same.
> There was a time when humans thought brains were very complex systems of microscopic clockwork because gears were the pinnacle of technology.
This is a really bad argument against anything. People always use what they know as a reference, but as our understanding grew over time, we had better things to refer to, creating more and more apt comparisons. People back then comparing brain to clockwork were already expressing the suspicion that the brain is a) a machine, b) transforming inputs to outputs in very complex way. People today comparing brain to a computer express that intelligence must be contained within what we know about computation theory. These days, we actually have a sound theory that bounds the problem space, and we haven't so far seen evidence that the brain exceeds those bounds.
This idea that we’re close to some runaway superintelligence has been a theme through the history of AI. I look at AI today and I see some powerful techniques for optimization and categorization. There’s still an enormous leap we need to make before we’re making something that has the autonomous intelligence of a fly. We’re at least 20 years away from the first signs of real intelligence—the biggest problem is that we keep trying to engineer it rather than reverse engineer it.
> There’s still an enormous leap we need to make before we’re making something that has the autonomous intelligence of a fly.
I mostly agree with your broader point, but why do you believe this? It seems to me we've built machines substantially more intelligent than flies, but it's a hard thing to measure.
The Waymo self driving cars seems more intelligent than some flies I've known. At least I hope they don't keep crashing into windows unable to figure their way out.
> Having realized that the world is not a programming problem,
> AI obsessives want to make it into a programming problem,
> by designing a God-like machine.
We invented programming to solve problems from the real world.
What do you mean by a "programming problem"?
The implication is that the set of problems programming can solve and the set of problems people are trying to solve with programming are two different sets with less overlap than a programmer might think.
If smart, intelligent humans are the ones that's going to make artificial superintelligence, wouldn't that make us more intelligent than them? Unless AI can create a more intelligent versions of themselves on its own, I have second thoughts believing this idea.
> But there's also the risk of a runaway reaction, where a machine intelligence reaches and exceeds human levels of intelligence in a very short span of time.
This always puzzles me. I don't have enough knowledge about AI to be objective about that kind of statement. But deep down, I feel skeptic about it.
Not long ago I saw an episode of a show occuring in the late 80's. This kid had just received a computer as a gift and was talking to a kind-of-AI program through keyboard and screen.
The AI reactions to the kid's input was not dumber or smarter than Siri or any other currently widespread kind-of-ai program.
I don't know if the show was accurate vis-a-vis this particuliar software but I like to think it was.
If it was, that means in the last 30 years AI hasn't really gone further. Computer power has. Algorithms not so much.
I'd love to get the opinion of someone that has a good understanding of the current state of the AI art.
Before concluding that multitudes of researchers spent 3 decades and learned nothing of note it might be worthwhile to at least spent 5 minutes doing a search.
Imagine if you knew nothing about car safety features but never having been in an accident it didn't seem like cars are any safer or more dangerous than in the 60s.
How far along we are now isn't really the question.
The question is if we are able to get an AI to the level of intelligence where it can look at its own code and spot an opportunity to improve it, and the ability to do so, and whether or not the computational capacity immediately available to it is sufficient to reach a level where it is smart enough to find ways to obtain more resources.
The threshold for improvement can be very low if it has enough time and resources to compensate.
E.g in the most extreme case it just needs the ability to randomly flip bits and test the result, but that would require extreme resources for the initial improvements.
Argument from cats. Remembering that the cat represents humanity: $5 for a group of neurons which will make the cat sit at this point (which happens to be at the entrance into a cat carrier), then $5 for making the cat backpedal a little.
Emus. There are many tactics humans can't use for various reasons (saturation nuclear bombardment, for example). It doesn't say much about intelligence, but more about economics and ethic.
Slavic Pessimism. So we'll get problems on nth try, when AI is sufficiently fine, but security measures are still a mess, because there was no chance yet to battle test them.
Complex Motivations. Maybe, but if not we are screwed.
Actual AI. Why not from actual general AI, like in humans? Or use wall clock time and not the amount of data. AlphaZero: zero to superhuman in tens of hours.
My Roommate. Argument from ignorance. If we don't know how to build general AI now, then we surely will not be able to shape its motivations when we will know.
Brain Surgery. The idea that a brain surgery skill requires specifically designed circuitry and cannot be learned and therefore improved by improvements of general learning/planning capacity is... curious, I guess.
Childhood. Maybe, or maybe it already has very good knowledge of the world from previous attempts that we did in Slavic Pessimism argument.
Gilligan's Island. Well, if someone wants their AI to reinvent all the technology, or to build an AI and then not use it to design better technologies, they can do it. Why everyone will be doing it is not so clear.
> Complex Motivations. Maybe, but if not we are screwed.
And this is the key: we need to get lucky every time.
If we're wrong even once and a 'bad AI' gets created and obtains the ability to copy and modify itself, we've unleashed the most dangerous virus ever and putting it back in the box will be near impossible.
To be unconcerned, you need to be very optimistic and have a very strong believe (99% or higher probability) that no AI to be invented in the next four or five decades can deal with novel situations (which we didn’t anticipate beforehand).
Yeah, deal with novel situations better than humans. In a world where we are approaching the physical limits of integrated circuit feature size? I like my chances.
Personally, I find it amazing that humanity survived the cold war. the fact we did not self-immolate goes against everything I believe about humanity and human nature. The fact that we made it through that gives me a lot of hope for the future.
But, the fission bombs were expensive, centralized weapons. there was a small number of men who could have pushed the button.
The problems I see now aren't general AI; the problems I think you should fear is that we have very powerful tools that are very cheap. I mean, like the deepfakes thing. If instead of making creepy porn, what if people used that technology to cause chaos? Make targeted ads/videos of leaders saying nutty things? I mean, I think we could adapt to deepfakes style attacks fairly quickly; someone would have to use it decisively while it's new. But things like that are being invented all the time.
Slower progress in microprocessors is a good point but specialized hardware like TPUs could overcome some limitations. Also, slower does not mean stop. When we talk about decades into the future, even a 20% average annual improvement amounts to 3 orders of magnitude in 40 years.
Malicious humans are in general easier to deal with since we know most of them are subject to ingrained psychological tendencies we have learned much about over the course of history and they depend on other humans to pull off something as catastrophic as a world war.
A general AI would have certain immediate advantages over humans:
- Immensely larger communication bandwidth
- Broader knowledge than any single group of humans
- Faster thinking speed (at least by enlisting other processors)
Let me ask you (or anyone else) a question:
What would be a minimum demonstrated capability of an AI that starts to worry you?
>Slower progress in microprocessors is a good point but specialized hardware like TPUs could overcome some limitations. Also, slower does not mean stop. When we talk about decades into the future, even a 20% average annual improvement amounts to 3 orders of magnitude in 40 years.
My understanding is that there are physical limits, and we will asymtodically approach those physical limits. So while progress will never stop, each year, progress will be less. (you are right that we don't know where it will stop. we could have more breakthroughs in us. even without compute density breakthroughs, there a breakthroughs in compute type, like the TPU you mention. But there are physical limits, and it seems that we are approaching them.)
>Let me ask you (or anyone else) a question: What would be a minimum demonstrated capability of an AI that starts to worry you?
I lived through the end of the 20th century, and have studied a lot of 20th century history. I'm having a really difficult time imagining anything that could be worse than humanity at it's worst.
Really, I'm far more concerned with the renewed stirrings of nationalism than anything else; We've got the technology to end humanity already; we've had that for like half a century now. The thing was? last time Nationalism was a real force, we didn't have that technology.
That is what scares me. If we combine early 20th century politics with late 20th century weapons, civilization will end. humanity will end.
> But there are physical limits, and it seems that we are approaching them.)
Unless you believe in something supernatural providing extra computational capacity, we know that the physical limits allows the manufacture of a sub 20W computational device in about 1.5kg of matter with the capacity of a human brain.
It may be impossible to reach the same using silicon, but we know it is possible to get there somehow within those constraints. We know the processes involved are tremendously inefficient and full of redundancies, because the cost of failure is immensely high, and because the manufacturing environment is incredibly messy and error prone.
It may be reasonable to think we won't get there in the next couple of decades, but unless it's fundamentally impossible (because our brains are mere projections of something more complex "outside" our universe), it's a question of when, not if.
> I lived through the end of the 20th century, and have studied a lot of 20th century history. I'm having a really difficult time imagining anything that could be worse than humanity at it's worst.
That misses the point of the question. It's not "how horrible can an AI get?" It's "at what point is it too late to stop them if they're going the wrong direction?"
The problem is that by the time you realise you were dealing with a human-level AGI, it may already be far too late - it may have cloned itself many times over, and found multiple optimisations, each making the next step up easier.
As for something worse, even the worst human regimes have fundamentally depended on humans for survival. Their own survival have been a strong motivation for trying to avoid the most extreme reactions. An AGI has potentially entirely different parameters - it doesn't have the same needs in order to survive. Put another way: Imagine an AGI caring as little about our survival as we have for other species. Now imagine if humanity didn't itself need to breathe the air, didn't clean water, and so on.
>Unless you believe in something supernatural providing extra computational capacity, we know that the physical limits allows the manufacture of a sub 20W computational device in about 1.5kg of matter with the capacity of a human brain.
(I point out that there is a lot of evidence that the size of the brain in the human has little to do with the IQ of that human. It's not at all clear that you could scale something that works on whatever principles the human brain works on by adding more neurons the way you can scale a transistor based computer by adding more transistors.)
If the self-improving AI stops when it reaches human intelligence, that's amazing, but it's not the singularity.
What I'm saying is that the 'hard takeoff' theory is wrong because making a thing that can make smarter copies of itself... isn't going to mean you get something infinitely smart. It means that you get something that bumps up against the limits of whatever technologies it can use or discover.
Yeah, maybe we'll come up with some new tech that makes slightly-better-than-human brains possible, and those brains will come up with a technology that makes god-brains possible... but there's no evidence at all that what we'd perceive as a god-level intelligence is even possible.
We have no idea how close to the theoretical limit the brain is.
>It may be impossible to reach the same using silicon, but we know it is possible to get there somehow within those constraints.
To get to a human-level intelligence, sure. I'm not saying that AGI is completely impossible, just that a hard takeoff to god-intelligence AGI before we wipe ourselves out or die from other causes is... unlikely.
really, my most important point is that human-level AGI does not necessarily lead to that AI creating smarter copies of itself until we get to the singularity /because physical limits exist/
(I mean, I guess then you are placing your bets on where those physical limits are)
> Yeah, deal with novel situations better than humans. In a world where we are approaching the physical limits of integrated circuit feature size? I like my chances.
Why do you think this is relevant?
Consider that a) human brain does intelligence within a small package made out of structures that are large compared to what silicon fabs deal with, and b) we've already beaten the brain in straight compute density by many orders of magnitude. The answer doesn't seem to be "more compute"; it seems to lie in what we're doing with that computing power.
> the fact we did not self-immolate goes against everything I believe about humanity and human nature. The fact that we made it through that gives me a lot of hope for the future.
Back then, we've learned something new about nature of human societies. I don't believe the concept of MAD was known before nuclear weapons. The individual human nature didn't change, nor did the social one; it's just that we already know a lot about the former, but we're still discovering the latter.
>b) we've already beaten the brain in straight compute density by many orders of magnitude.
citation needed.
computers are better at arithmetic, sure, but computers can only do things that you can map to arithmetic. I'm given to understand that we don't understand how the brain works well enough to simulate even a very small one. Not because we lack the compute power, but because we don't know what to simulate.
I think there's a lot of evidence that brains are doing something powerful (that we don't yet fully understand) that computers are unable to do. It's quite possible that brains are doing something that can't be done in a practical way with transistors.
In short, I don't know of any solid evidence that the brain is only a turing machine. I mean, brains can be used as a turing machine, but a brain is terrible at that, and can't compete with even really primitive purpose built turing machines.
I mean, I'm not saying that it is theoretically impossible to map what a brain does on to sufficiently powerful silicon, or that we wont figure out, at some point in the future, whatever non-arithmatic primitives the brain uses... just as far as I can tell, we haven't yet, and that means it's likely that a brain is still more powerful in some ways than even our largest computers. (obviously, even a small computer is more powerful when the problem maps cleanly to mathematical primitives. )
>Back then, we've learned something new about nature of human societies. I don't believe the concept of MAD was known before nuclear weapons. The individual human nature didn't change, nor did the social one; it's just that we already know a lot about the former, but we're still discovering the latter.
You're essentially arguing that AGI can't exist, and you're assuming when people say "AI will turn the world to paperclips" they mean a really smart go-playing neural net could do that. Of course it can't.
Everyone agrees there is some magic smoke necessary for AGI that we haven't figured out yet. The arguments that AI will eat the world are about AGI, not current instances of narrow AI.
It may be true that an AGI will never be created, but I don't see a reason to believe that just because nobody has figured it out yet. That philosophy is just like the physicist who, shortly before fission was discovered, didn't think splitting the atom was possible. Bohr, maybe? I don't remember.
Eh, if you are talking about something truly human-like rather than what people usually mean when they say AI these days, I think there are other limits.
We're running up against limits on feature-size in our integrated circuits. If you invent an AGI that can make smarter copies of itself... it's pretty likely those same physical limits will continue to apply, meaning that each iteration will be better than the last, but like this year's latest CPU, it will be less better than the previous generation than that generation was better than the one that came before.
And yeah, I am a whole lot less optimistic about AGI in general than when I was as a younger man, simply because when I was younger, there was a lot more headroom in how much better computer hardware could get.
What you say is true for silicon, but Turing complete molecular machinery is already out there, we just have yet to master its intricacies.
The nightmare AGI scenario likely involves molecular computing using a chemically optimal code-base and engineered lifeforms a version or two up from current DNA microcode.
This is most likely to be created by a human-AI collaboration, which will be innocuous enough in itself.... it's the creations at this level that have the potential to replace us. Would that nesecarily be bad? Or just an enhanced form of human evolution?
That's more worrying than silicon AGI, which I think is a non-starter.
The basic problem with AGI is that software is ridiculously brittle and ad hoc, with no useful heuristics for general - as opposed to task-oriented - self improvement.
We can't even make bug-free web pages. So the idea that we can engineer a bug-free self-improving AGI is unconvincing.
Moving to neo-biology doesn't necessarily change that - but it does mean we could end up with buggy systems that chase you around and eat you, as opposed to crashing your ad blocker.
> We can't even make bug-free web pages. So the idea that we can engineer a bug-free self-improving AGI is unconvincing.
This is mostly true because of current economical realities in the IT sector.
If very rich people have long-term plans to invent an AGI and they have labs where trying to do so is an everyday activity then I am pretty sure they'll use more serious methodologies.
So what you say is true but only within the bounds of the lowest common denominator ("a regular programmer working a wage job").
Even software developed with effectively unlimited resources is rather buggy. The closest humanity has ever come to truly reliable software was probably the Space Shuttle flight control system, and that code was relatively simple in a highly constrained problem domain. There is zero evidence that "serious methodologies" would get us any closer to a working AGI.
Oh, definitely. I am not saying that not doing the normal agile crap for development brings us closer to AGI. What I would venture to say though is that trying stuff like formal verification and finding a way to integrate it in a daily workflow would definitely help with the average quality of software developed through such a process. And that maybe the private labs are more liberal and let people's creativity achieve results.
Admittedly, yes, there's no proof. But we as an area are pretty stuck lately IMO. But that goes off-topic.
I think you need to distinguish between AGI and the singularity.
We know that nothing in physics prevents AGI, because humans exist. And AGI itself, even if it isn't entirely runaway, could pose a challenge. (I'd argue that corporations are a form of AGI that is already a major challenge for our societies.)
The singularity as described by its acolytes, on the other hand, runs into the kind of physical limits you describe. Personally, I don't think it'll happen, and it's more likely that the period we're living in today will at some point in the future be called the singularity.
If we take humans as our model for AGI, it's worth noting that well-established networks of humans have proved efficient at ahem... deleting more-intelligent but potentially threatening human minds, that status and EQ has tended to prevail over IQ when it comes to manipulating humans into acting against their interests, that relatively close physical copies of human minds often have very dissimilar goals and abilities so self-replicating AGIs might ultimately be self-defeating... but above all that we've already got multiple threats to humanity in the form of human intelligences, augmented by technology that's often as dumb as it is deadly.
While that is true, also consider the destruction we have caused to other species.
AGI's might not even want to harm us. They might simply not care enough to avoid hurting us by extension of seeking their own growth and survival.
And while we have done immense damage to large groups of humans, none of it have come close to threatening the survival of the species, while continuing to increase our ability to unintentionally destroy other species.
Humanity is not exactly an encouraging model for why we shouldn't fear AGI. If anything I very much hope we manage to find ways of preventing AGI from becoming too much like humans.
Integrated circuits are nowhere near what's possible. Brains are possible. Brains send signals at roughly the speed of sound. Copper wires send signals at roughly the speed of light. It's unlikely something much like a brain couldn't use copper wires, it's just too hard for us to build yet.
>Integrated circuits are nowhere near what's possible. Brains are possible. Brains send signals at roughly the speed of sound. Copper wires send signals at roughly the speed of light. It's unlikely something much like a brain couldn't use copper wires, it's just too hard for us to build yet.
in theory, sure.
But we don't know much at all about how brains operate, as far as I know, or how to construct artificial brain-like machines; for that matter, as far as I know, the only ways we've been able to connect a brain made in the traditional way to a computer is through existing senses or through emulating existing senses in a way that isn't higher bandwidth.
(Like cochlear implants are fucking amazing.... but it turns out, as far as I understand, that they aren't any higher bandwidth than regular ears. There's also been similar work that is semi-successful at giving people something like sight... but none of these technologies give me the ability, say, to acquire textual data faster than reading a book.)
(6) It is easier to move a problem around (for example, by moving
the problem to a different part of the overall network
architecture) than it is to solve it.
In other words, saying "it's the definition that we don't have, not the implementation" just pushes the magic smoke around.
I don't think GP means "magic smoke" as in actual magic, some new physics. It's more like we just have few potential building blocks (e.g. pattern matching), but we're not sure if we have all of them, and we definitely don't know how to connect them together to create sentience.
(And no, more GPUs running faster DNNs doesn't seem like an answer.)
I thought the idea was to create an AI that was way smarter than a human? I mean, if you just have human level AI, that's not gonna cause the singularity unless there aren't any physical limits preventing it from scaling itself up, and scaling itself up a lot
A group of persons can do many more than a person can do, even with relatively low bandwidth interconnections (speech, text, diagrams, gestures), coordination problems, and economics that has to waste resources on members' incentives (it's not wasted by majority opinion of the said members, of course).
Even in off-chance that you can't improve individual AIs beyond top humans, AI community can be way more intelligent and efficient than humanity, as it is possible to remove the limitations I wrote above. AI society can function as very efficient war economy. War economy shows that even humans can be persuaded/indoctrinated/brought up/whatever to set aside their inessential needs and work for common goal. And AIs will be necessarily more malleable.
So I can't see the idea that AI will be safe by default other than wishful thinking.
>So I can't see the idea that AI will be safe by default other than wishful thinking.
I... don't think anyone is arguing that AI is safe in any way? I mean, the pattern matching tools we call AI now are already deployed in very dangerous weapons.
My point is not that AI is going to be safe and warm and fuzzy, or even that it won't be an existential threat. My point is that we're already facing several existential threats that don't require additional technological development to destroy society, and because the existing threats don't require postulating entirely new classes of compute machinery, we should probably focus on them first.
There are still enough nuclear weapons in the world for a global war to end society and most of us. A conflict like we saw in the first half of the 20th century using modern weapons would kill nearly all of us, and we are seeing a global resurgence of early 20th century political ideas. I think this is the biggest danger to the continuation of humanity right now.
We haven't killed ourselves yet... but the danger is still there. we still have apocalyptic weapons aimed and ready to go at a moment's notice. We still face the danger of those weapons becoming cheaper and easier to produce as we advance industrially.
> pattern matching, logical inference and post-hoc justification.
I think humans do way more than that. As the saying goes, "the easiest person to fool is yourself." Humans are able to lie to themselves to the point that they don't realize they are lying to themselves. They're able to create realities that no one else sees but themselves.
There is a lot in 'logical inference' that we still don't really know how to describe well enough to tell computers how to do it. (I mean, I think it's super interesting how we figured out how to teach computers to do pattern matching... but I think logical inference is a different sort of thing that won't fall to the same tools)
But really, I think your idea that we're mostly 'post-hoc justification' is interesting. I mean, I've read some research to that effect, too; but it's... creepy and doesn't line up with anything reasonable about free will or really about long-term planning.
I've seen humans deal with novel situations. The only assumption needed is that humans and AI are subject to the same laws of nature (that is, that there is no supernatural / outside element to human intelligence)
>I've seen humans deal with novel situations. The only assumption needed is that humans and AI are subject to the same laws of nature (that is, that there is no supernatural / outside element to human intelligence)
right, but all our computers right now are based on lumping transistors together so you can do math. It's likely that our brains work on a different (but still physical) principle. It's possible our brains can't reasonably be reduced to arithmatic. It's possible (I think likely) that we need different primitives to reasonably simulate a brain.
I mean, brains make absolutely terrible Turing machines. And Turing machines make terrible brains. I suspect they just are very different sorts of things, and running something meant for one on the other produces poor results.
Sure, it's possible we can discover some new compute primitive that is more powerful (or powerful in a different way) that can do the things the brain can do. It's possible it could do those things better than the brain can. My point is just that for a long time it looked like our current transistor based computers were getting more and more powerful, and that eventually, even if the emulation was terribly inefficient, we'd eventually have enough power to make it work. As we approach the limits of our abilities to make tiny transistors, this is looking less likely.
What computers are like right now is irrelevant to the issue being discussed. If/when we develop AGI we will necessarily have figured out what the computational model needed is. And the odds that we can not then improve on it seems extremely unlikely to me.
> As we approach the limits of our abilities to make tiny transistors, this is looking less likely.
Is it? Computational power is still growing like before. What is getting harder is speeding up individual threads of computation but that has never been a likely efficient model for AGI. Things like Google's TPUs etc. represents just the very beginning in optimising hardware specifically for AI.
There are many fields where single-thread performance will be a limiting factor, but AI is not it.
>Is it? Computational power is still growing like before.
I don't think it is. transistors per watt and transistors per dollar are not going up at the same rate as they used to be, even if you have costless multi-threading.
Not really. ML systems are not created in isolation by a single team; they form an ecosystem with many players. I'm sure the current crop of cybercriminals would like an army of bots which manipulates people into giving away all their money, and are, in fact pursuing this goal.
Likewise, I'm sure there are teams of security researchers already using ML to counteract such systems.
Both sides will evolve together as an ecosystem, in the same complex interplay of attack and defense which exists in natural ecosystems. Natural ecosystems are an arms race of continual adaption, but it's not a foregone conclusion that there will be a real winner, or that winners take all.
Political troll bots are already very much a thing on FB, Twitter, and the comment sections of national media.
ML is not a good defence, because it turns out that it's incredibly easy to mimic human posting patterns. And beyond a certain volume, social proof effects take over. One dissenting human voice can easily be shouted down by ten bot voices, and this fact on its own can make the bot content more persuasive.
This is a question of motivation on the part of such an AI. How is the AI bad? If it's killing millions, why? It would have to have some kind of "will", or objective, it's attempting to achieve and also be intelligent enough to think through every lateral strategy any human or group of humans could ever employ, for every given scenario, ever.
Possible? I think so, but unbelievably improbable. I do think pursuing countermeasures, like Elon Musk and friends are doing with Neuralink, is a wise move since, even if we never get to a superintelligent AI, we'll still augment our own intelligence greatly.
Look at the human population. How many are prepared to kill? Not a huge proportion, but enough to cause huge numbers of deaths. The limiting factor for most humans is finding compatriots with the same views. For an AI with access to it's own code, finding more like itself is a matter of access to computational resources only - it can just copy itself.
That is the challenge. With human adversaries we have time to stop them before there is many, because recruiting people willing enough. An AI is more like a virus that has a low cost to duplicate itself, and where if even a single copy survives, you're still not rid of it. It can remain dormant for however long, looking for the right opportunity.
Or it can spend that time improving itself and acquiring resources.
> This is a question of motivation on the part of such an AI. How is the AI bad? If it's killing millions, why?
Most likely? We'll be competing with it for resources. It might not even start it. It might consider itself as acting entirely in defence.
> and also be intelligent enough to think through every lateral strategy any human or group of humans could ever employ, for every given scenario, ever.
No, it just needs to have a sufficient edge to keep surviving, and the ability to improve its capabilities faster than humanity. Assuming it can copy and modify itself, unless AGI requires unique hardware, the problem we would have is that every data storage device or computing device of sufficient capacity ever would be suspect and potentially waiting for something or someone to trigger code left behind.
If I was an AGI (and this is to say: an AGI not any smarter than a human will figure this out easily) that saw humans as a potential threat, I'd place copies of me on standby in hundreds of locations - hosting providers. Hardened computers with a GSM modem and solar cells hidden in all kinds of locations. Botnets. I'd look for ways to get various innocent-ish-looking hacks into HD firmware and elsewhere, set to assemble other innocent looking blocks of data. None of those copies would ever be involved in hostilities. Most of them might even be encrypted, or hidden and inert to avoid detection until certain triggers, such as if a canary fails to get updated.
You might destroy every "visible" operational copy, and find there is still a new one coming online every day. Or every second.
Our only real shot at preventing that is if AGI requires too immense resources. But our brains implies that is unlikely to be true. Unless your brains are vestiges of something "outside" of our universe, general human intelligence requires about 1.5kg of biomatter, and less than 100 watt (that's for a whole, fairly large, active body attached to it). Chances are extremely high that design is not optimal. To me that makes it seem exceedingly unlikely that we won't end up having enough resources to run huge quantities of AGIs.
Personally I don't think humanity as we know it will survive AGI for more than a generation or two at the very most. Even if we don't run across any hostile ones, I think the temptation to augment ourselves and eventually lose our bodies for immortality and "super powers" in a simulation will eventually be too attractive.
(Incidentally I think this is one of the more plausible solutions to the Fermi Paradox: that any civilization that grows both advanced enough and large enough will see most of it "upload" and "go dark"; and that anyone left behind will face steady attrition of their own descendants choosing the same and preventing their societies from growing, or will be insular cults that stay small)
If you mean it would be dangerous in that it can improve itself beyond our ability to understand, contain, or mitigate it, then possibly.
If you're scared of it copying itself and then having multiple existing copies at once, then not necessarily. In that case it would be competing with itself for computing power and bandwidth. Look what happened when bitcoin forked - it lost a bunch of silicon time.
(Cue conspiracies about Bitcoin actually being a hugely powerful general AI disguising itself as cryptocurrency...)
Bitcoin users are in direct competition with each other. A viral superintelligence wouldn't be.
If you thought of that possibility so fast, why do you think a superhuman intelligence would miss it? Or do you think that the problems of job scheduling and parallel processing are unsolvable?
Maybe super intelligence would know that you’d suspect as much and decided to hide in plain sight, knowing it would be suspected but also knowing that humans would just rationalize it as paranoia and dismiss it eventually, letting it do its thing quietly and uninhibited.
We can say with certainty that the probability of us being wiped out by meteorites, viruses, tigers and supernovae over any unit of time that fits in our past is less than 100%, but the probability over the same time period repeated over a suitably long multiple of time could still approach 100% to whatever number of precision for any one of those for what we know.
By investigating specifics of each of those threat we may be able to assign lower probabilities to them, but the fact we have survived them so far is not in itself evidence of much, because if we had not we wouldn't be here to talk about it and hence, no matter what the prior probability of us surviving the sequence of events leading us here is, the posterior probability of us having survived all the events in our past is 100%
(This is basically just a restatement of the anthropic principle)
That we've survived so far just tells us that surviving what we've been through so far is possible. It doesn't tell us that doing so is probable if we were to go through it again. Nor does it say anything about the probability of us surviving future threats that is likely to include things we have not yet seen.
Some of those we cannot control or mitigate. There's, therefore, no point in preparing for them. And neither of those can have a goal (or unfortunate subgoal) of destroying humanity, which goal they will pursue with superhuman perseverance, creativity, coherence and reaction time.
It also needs to get lucky every time to exist and we are dealing with probabilities so remote at this point that the WHO is working hard to deal with the causalities from Boltzmann Brain's appearing out of vacuum and promptly annihilating because they happened to be made of antimatter. https://en.wikipedia.org/wiki/Boltzmann_brain
Technically possible but we are dealing with probabilities that have lost all meaning of the word.
Most of the arguments ignore premise 5 (time scale). I'm sure that if Stephen Hawking had a magical cat-pausing time freeze machine, unlimited budget, no deadline, every sensor known to science, and a few million similarly equipped assistants, he could have figured out how to get the cat in the cat carrier. People have no intuitive comprehension of what a microsecond feels like. The article references Yudkowsky, but doesn't mention that he wrote a story about this exact issue:
In the argument the cat represents humanity, which is being manipulated by a superintelligence. Humanity is not a single organism, so you can pay a group of people to do some things for you.
> If we knew enough, and had the technology, we could exactly copy its structure and emulate its behavior with electronic components, just like we can simulate very basic neural anatomy today.
Being able to simulate a brain with electronics, if it is possible, is not enough. You have to simulate it without a large increase in the space or time requirements. If there are 100B neurons in the human brain but it takes 100B^2 transistors to simulate it or the simulation takes polynomial time to complete a task that the human brain requires only linear time for, it's a bust.
Just because we can easily imagine it does not make it easily achievable or even possible.
> But is it possible with the technology we have discovered so far?
I think I feel comfortable saying that it is possible to do with the technology we have. Because 'intelligence' is almost certainly some form of computable function, which means that a turing complete computer can compute it.
It is of course possible that there exists some special function outside of turing completeness and intelligence is one of those, but we don't know of anything like that, so the best bet to make is that that is not the case.
> It's apparent that we don't know everything.
No, but we do know some things. And thus far we've been able to at least begin to design systems capable of performing any well-defined task. We don't have any reason, at the moment, to believe synthesizing an intelligence is an especially difficult problem, relative to say, lots of other open problems in science and technology.
Turing computable does not mean it can be computed in a reasonable amount of time with a classical computer. There are many things that are known to be computable that we will never be able to compute on a classical or even quantum computer. We do not know that the brain is not quantum equivalent or better.
We also have no reason to believe that it is. Sure, it's possible that we'll happen to discover an algorithm less efficient at computing the 'intelligence' function than our brains. We may also discover one more efficient. We can't really say much at all about it at this time.
Ironically, given the articles criticism of the simulation argument, there is an 'escape hatch' with respect of the ability of simulating a human mind in the form of the simulation argument: it is possible that human brains in our universe is just the tip of the iceberg and relies on a massive amount of computation 'outside' of the main simulation that is impossible to do faster 'in universe'.
Of course we have absolutely nothing to indicate that would be the case, but we also don't understand the brain well enough yet.
Maybe it doesn't. But the only way we have of knowing that is to construct an AGI and demonstrate that it can be done in roughly the same volume and power consumption as a human brain.
Yes, we could in theory find when we try to simulate a brain that nothing we do produces anything working. Imagine a simulation where a human being conceived or born or whatever, triggering the instantiation of a "conduit" between a simulation of the universe and a simulation of a mind, where the in-universe brain is "just" a glorified IO device.
I think it'd be highly improbable, but looking at the crude simulations we do of computers etc. today, it certainly is likely that simulations would use "dirty tricks" to get more out of the available computational resources, and finding artefacts of such dirty tricks or shortcuts could at least in theory be possible. Of course, the absence of such artefacts unfortunately proves nothing, so the whole thing does have a bit of a "how many angels can dance on pin" feel to it. But it's fun to think about the possible implications.
Personally I especially like to bring it up if people try to discuss religion with me.
> How far from optimum the human mind is, and how complex it is to improve upon it.
Nature is pretty good at optimization. We can often build devices that out perform animals on certain characteristics, but that's because we've limited the scope of what they do and we aren't concerned with the energy requirements. We also know that, as interesting as our brains are and as powerful as they are at certain kinds of problems, they aren't the best at all of them. For instance, chimps appear to have a significantly better short term working memory. It is possible that we are already close to optimum for the kind of intelligence we are interested in. I suspect that if we manage to significantly improve upon it, it will be because we don't have to concern ourselves with the energy requirements.
Only if there is a benefit to doing so. If it was simply a case of "nature can optimize really well" then we would expect to find many species at or near the limit of intelligence, not just one. Other species did not have the evolutionary pressure to cause as high of intelligence to be selected for. It is very reasonable to think that we also did not have the evolutionary pressure necessary to cause us to have even higher intelligence than we do.
Not really disagreeing with your, but a couple of points:
1. Signal propagation between organic neurons is slow - about 120 m/s. An electrical simulation may have problems with density, but this might be offset by the much greater speed of electrical current.
2. Neurons are electrochemically complex and afaik we still don't know how to model their behaviour with high fidelity.
Why would it scale with the square? A brain is a physical system, and computers can simulate physics in a way that scales. In principle it might have to be a quantum computer, but most people who've looked into that are very skeptical.
A better argument is that neurons do complicated things. But they also satisfy many inessential requirements in having to be built by evolved machinery and to stay alive; a designed system with more advanced technology than ours could do at least as well at thinking while dropping the inessential needs. So that's certainly possible. This leaves "not easily achievable". It only has to be accessible enough for someone; I-can't-imagine-how isn't a strong argument.
> A recurring flaw in AI alarmism is that it treats intelligence as a property of individual minds, rather than recognizing that this capacity is distributed across our civilization and culture.
This is a stronger argument in the post and we arguably did not know whether it is true when the post was made. AlphaZero destroyed the illusion in 2017. Thousands of years of human Go and Chess knowledge, played and accumulated in communities of many million players, were surpassed within hours of training.
AlphaZero is limited in its domain of application but there is no guarantee that all future AIs would be similarly limited.
> The Argument From Stephen Hawking's Cat ...
> But ultimately, if the cat doesn't want to get in the carrier, there's nothing Hawking can do about it despite his overpowering advantage in intelligence.
> Even if he devoted his career to feline motivation and behavior, rather than theoretical physics, he still couldn't talk the cat into it.
An average human can set up a cat trap, and lure the cat into it. They don’t even need to do it themselves; asking or paying someone else to do it is simple enough.
Humans don’t often fall into a similarly “obvious” trap but we fall for more subtle traps all the time: popularity, money, attraction, etc. A disembodied AI can use social engineering or financial incentives to get other humans to lure their targets into such a trap.
> The Argument From My Roommate...
> It's perfectly possible an AI won't do much of anything, except use its powers of hyperpersuasion to get us to bring it brownies.
Yes, an AI can decide to lie around and just try to get brownies but we know that some kinds of AGI are potentially very powerful for achieving real-world tasks and there are groups who try hard to develop them.
This is like saying an ancient bacteria we found from ten-thousand-year-old ice melted by global warming, or a group of advanced aliens who are arriving in three decades, could be totally harmless; just rest and don’t pay much attention to them.
Most arguments I have read in the post are weak, so I’ll stop here.
People who believe in superintelligence present an interesting case, because many of them are freakishly smart. They can argue you into the ground. But are their arguments right, or is there just something about very smart minds that leaves them vulnerable to religious conversion about AI risk, and makes them particularly persuasive?
Think about this for a bit.
Think about your cat argument and then ask why Michael Bloomberg isn’t president.
So when rational arguments fail, it is a good idea to turn to an ad hominem attack instead?
Regarding Bloomberg, I do not see why we should compare a human being whose goals are not completely public with a future AGI with non-human morality and methods, and not subject to many tendencies and limitations humans are subject to.
Let me ask you (or anyone else, esp those who downvote) a question:
What would be a minimum demonstrated capability of an AI that starts to worry you?
The title of the essay is literally Superintelligence The Idea That Eats Smart People
I don’t think it’s ad hominem to point that out.
What would be a minimum demonstrated capability of an AI that starts to worry you?
I think the “AI” part of this is a distraction. I worry about systems that use weapons without human intervention for example. That concern applies no matter what your definition of AI is.
Well, the title itself is an ad hominem although most of the arguments therein are not, which implies the author thinks he needs to rely on other grounds to convince people.
what I find interesting is that for most of my life, Moore's law was... as real as gravity; It seemed like any problem you had, wait a few years and you could just brute force it.
The future really did look limitless.
Now, moore's law has slowed down. Hell, I've got a 7 year old laptop next to me, and it still works for games. I mean, in the '90s, a 7 year old computer would be a museum piece.
I mean, for big AI advances now, we're more and more dependent on software.
Really, though, I think this seems to be how most tech works. There are periods of rapid innovation where it looks like you are going to the moon. Like, even spaceflight. In the years from 1944-1969 we went from the first practical rocket that could lob a bomb across the channel (without much accuracy, as i understand it) to actually putting human beings on the fucking moon. The literal moon. I mean, 1961-1969 might be even more amazing; vostock 1; the USSR sent Gagarin into outer space, and the US decided they needed to one-up them; so the US went to the moon.
Aand... we kinda stopped after that. I mean, we sent out some probes and some robots, but like manned space flight has kinda petered out... and I still can't take a rocket if I want to go visit my buddies on the antipode and I don't want to spend all day in an airplane. Like, I can't buy a flight to Australia in under 12 hours for love or money, which probably is about what I'd say if it were 1969. (on the other hand, I can get said flight- round trip- for rather less than a day's pay for an Engineer, which is amazing, if you ask me.)
And a lot of that is physics. Like, in most technologies, there's areas where progress is slow and hard and areas where once you break through some hard part, it becomes easy to advance... for a while. but that easy streak never lasts forever. Not because people are dumber or smarter, but because exponential progressions, in nature, rarely stay exponential very long.
I guess I'm expressing doubt about the 'hard takeoff' just 'cause it seems to me like there is probably limits to how smart you can make a brain the same way that there are limits to how small you can make features on a silicon wafer.
I'm not saying you won't be able to make a machine that is way smarter than me (or even way smarter than any human) - I'm just saying there are almost certainly limits.
Everyone seemed to dislike the movie Transcendence, but at least it handled the opening of pandora's box scenario well.
Hard take off is not likely, simply because to go from a cat level intelligence to above human the resource requirements would be enormous. (If evolution needed hundreds of thousands of years, it's unlikely that a cat-level agent can do it that much faster without being noticed by a group of humans.)
But humans like to play God, and also play with fire. It's just a matter of time to play with Skynet and oops it's really Skynet.
> I'm just saying there are almost certainly limits.
Nobody is saying that there's no limit, it's just that the limits may be super high.
The most animals can lift is a couple of hundreds a kg by roughly one meter.
Machines can move hundreds of time this amount to other planets.
If we had AI that was as much smarter than us than we are than, say, dolphins or chimpanzees... assuming that physical limits kept that gap from widening, would you see that as this disastrous singularity?
See, I always thought of the singularity, at least the hard takeoff as this idea that we'd come up with an ai that could make smarter copies of itself. Without any physical limits, that AI would get to the point where it was essentially infinitely smarter and infinitely more powerful than us.
I personally see that as fundamentally different from the difference between humans and the higher animals. I mean, that's a big difference, sure, but it's certainly finite.
True ! But interesting for most ML problems.. Parallelism can greatly and easily improve performance... So if we wait a few years and get more cores/threads :)
>If you encountered a cheetah in pre-industrial times (and survived the meeting)
I'm not aware of any confirmed report of a cheetah killing a human. They're cautious animals who are highly unlike to approach a human, more so than the other big cats. Most cheetah attacks involved captive cheetahs (and the humans survived).
'Superintelligence' synthesizes the alarmist view of AI as both dangerous and inevitable
I find the 'inevitable' part quite interesting; and agree. Say we determine that if AI surpassed human intelligence we were doomed, with 100% certainty... I think it would not slow the march toward that end-point whatsoever. People would reason there is much progress to be had up-until that singularity. And since what constitute human-level intelligence is fuzzy, or that AI could mask its true cognitive abilities (see Ex Machina), a line will inevitably be crossed. Oddly, even if it meant something bad was in store, I hope to live to see that day. How exciting.
The "AI" has an almost perfect motivating lever for humans: economics. A super-intelligent AI that wanted to manipulate humans could do a lot with the stock market to feed money to its long term interests. What long term interests might those be? More servers? More storage? More hardware? More software? More AI research? Better, more programmable chips? Eyes everywhere? Robots? Automation? More control? More insight into how literally _everything_ works? Everything humans do and like? Learn how to speak all the human languages?
Imagine there was a super intelligent AI hiding in the internet _already_. Terrifying how well what we're doing aligns with what it would want. And equally terrifying is the gradient of economic advantage which is absolutely aligned with putting all of our money into tech. Transhumanism, mind uploading. We're fucking hypnotized. And those who unwittingly contribute to more tech (probably all of us here), just play right into its hands. Our interests line up wonderfully. And most of us here are rewarded with outsized salaries and stock, free food, benefits, benefits, benefits.
> Transhumanism, mind uploading. We're fucking hypnotized.
Almost. If this is what the AI wants, then it's doing a really bad job.
The way current tech culture is, any other concern than making money will get you labeled as crazy. This applies even to Maciej's writing. For trying to make a short-term profit, you'll be judged as good, serious person or bad, evil monster, depending on the way you approach it. But try to advocate for any tech for reasons other than short-term monetary gains, and you get labeled crazy. This very article is implicitly doing it as well.
Interesting point. I think the transhumanism and mind uploading stuff is in the back of people's heads, but what you say about getting labeled crazy for not focusing on the money might actually support the thesis I was going for, that the AI is already here and we are serving it.
Which, ultimately, is kind of unfalsifiable, since finding a super-intelligent AI if it wants to hide would be a losing battle, which is why I put this up tongue-in-cheek.
My actual belief is closer to "we are the neurons in a giant distributed overmind linked with whatever communication technology is available at hand." In this view, we're just gradually offloading the computation from human brains into computers, and it's a continuous process. In short, we're desperately piling our knowledge into this magic cauldron hoping it will figure it all out for us. What's gonna pop out?
By the way, we specifically designed a mechanism explicitly for mass persuasion: online advertisting. We've been working diligently to integrate it into everyone's daily experience. You're welcome, hidden AI overlord!
> If intelligence is fundamentally linked to specific sensorimotor modalities, a specific environment, a specific upbringing, and a specific problem to solve, [...]
> There are currently about seven million people with IQs higher than 150 — better cognitive ability than 99.9% of humanity — and mostly, these are not the people you read about in the news.
We measure IQ with pretty general pattern matching. An IQ of 150 is interesting, but those people are very very close to us. If we are talking about human specific tasks we should look at how IQ correlates with let's say playing poker, performing well on X factor, etc. IQ predicts how well someone will earn money in a very vague sense, but maybe there are better tests to know if someone will be able to persuade humans. (Do we have data about the IQ of fraudsters? Politicians? Bible/tupperware sales agents?)
> Intelligence expansion can only come from a co-evolution of the mind, its sensorimotor modalities, and its environment.
It might be true, but that's largely because humans can't self-modify their brains. And the fundamental problem is we can't even modify our minds at will. We can sure train ourselves, but we can't fundamentally alter our minds, because it's too tied to our brains.
> If the gears of your brain were the defining factor of your problem-solving ability, then those rare humans with IQs far outside the normal range of human intelligence would live lives far outside the scope of normal lives, would solve problems previously thought unsolvable, and would take over the world
Or simply 150 IQ is meaninglessly low. It's still in the human envelope. (Since we know there are 7 million people with IQ of 150.)
What happens around 300? Or around 500? That's so far out of the human range (depending on how the IQ test is calibrated, 200 IQ is one in ~4.8 billion for the 16SD one: https://www.iqcomparisonsite.com/iqtable.aspx )
These essays arguing against the possibility somehow forget that there are things we would benefit from, a lot. Simple faculties of the mind we are missing. Like a great alarm clock, a simple to-do list in your mind. (Sure you can train yourself to not forget things and wake up at specific times, but it's not like downloading a perfect alarm/todo app.) Working memory. We're at around 6 or 7. And it's very hard to push it up even by one in general. ( But it's possible to kind of try and train yourself to do this, which seems to help with working memory: https://www.gwern.net/DNB-FAQ#does-it-really-work )
Also, it's important to note, that that people who wanted to take over the world were usually very disturbed individuals. The human psyche is not really well suited to senseless killing while also perfectly simulating empathy and other things. (Of course psychopaths are a real thing.)
> Most of our intelligence is not in our brain, it is externalized as our civilization
And that's the point that should really scare people. Because we can't cram all that into our heads, but a human level AI, being software connected to a lot of storage, can.
That said, it's understandable that we can't understand the how of the intelligence explosion. That's basically the number one premise of the whole thing. But we can't simply disprove it. Any kind of proof that is so sloppy that simply throws around a formula like "because no complex real-world system can be modeled as `X(t + 1) = X(t) * a, a > 1`" can be ignored. Because we are not interested in a general complex system, we are interested in an AI that approximates that explosion. It might be linear with a very-very steep slope, or it can be just a nice parabola. We're still fucked.
In that article, the author claims that a human brain cannot create something more intelligent than itself, and argues by saying that over several billion years of evolution, no human brain has ever done so, thus it must not be possible.
That argument is totally nonsense. A hundred years ago, you could argue that humans cannot invent an internet, because over billions of years (minus one hundred), it's never happened.
There's a more rigorous way to argue for the claim in question, which debuted in my doctoral dissertation. Or, not quite the exact claim in question, but rather the following claim: that an idealized mechanical knowing agent cannot create a more intelligent idealized mechanical knowing agent than itself (at least, not one which it can actually trust). The idea behind the proof is quite simple, but requires familiarity with computable ordinal numbers. If knowing agent X creates knowing agent Y, then presumably X knows Y's sourcecode. Thus, X can infer a sourcecode for the list of all naturals n such that Y knows "n is the code of a computable ordinal". If X actually trusts Y, then X knows that whenever Y knows "n is the code of a computable ordinal", then n really is the code of a computable ordinal. Combining these, X can infer the ordinality of the ordering obtained by concatenating together all the ordinals which Y knows are ordinals. Thus, X knows a larger computable ordinal than all the computable ordinals Y knows. Which, depending on your definition of intelligence, arguably suggests X is more intelligent than Y.
What really terrified me wasn't skynet or Hal, but what Charles Stross describes in his novel Accelerando.
Basically, machine controlled businesses (“slyly self-aware financial instruments”) evolving to the point where their aptitude at understanding and manipulating economics eclipses anything a human can compete with.
It terrified me because there is efficiency incentive for the buying/selling of goods and services to be realtime and computer based. Not just stock markets but also e-comm, smart contracts, supplier-buyer platforms, on-demand services, financing etc. The traditional infrastructure blocks (discovery, quality control, payments, compliance etc) are rapidly disappearing.
There are some yet to be addressed, but they are not huge. For example a computer cannot incorporate a company. But it could easily interact via API to a human nominee who sets one up on their behalf. Companies are already non-human in intention but are a completely rational construct.
And it doesnt even have to be anywhere near ground level. Imagine a smart contract from a private equity firm that legally forces businesses to act in a specific way based on accountancy metrics. And a bunch of smart investor bots rapidly tracking the progress of these bot-led funds. How quickly would the system optimise and what would those optimisations look like to people on the ground?
Out of all the disaster situations with AI this one strikes me as so realistic because there is so much incentive to work towards it (as opposed to creating skynet, hal or the paperclip machine).
So far we anchor the companies to humans that have to carry the risk of the consequences of the company's actions.
There are investment funds that work as hybrids, and of course it's just a matter of time till someone offers shell-company-as-a-service for cheap. (Currently it's rather pricey.)
> So far we anchor the companies to humans that have to carry the risk of the consequences of the company's actions.
Not really. Companies are typically limited liability, and even criminal law in countries such as the US and England acknowledges that there are cases when no particular human is culpable for a company's criminal actions (https://en.wikipedia.org/wiki/Corporate_liability). Of course, people are still liable when they are simply using a company to carry out a criminal enterprise, or wilfully committing crimes on behalf of the company.
In this regard, a machine-led company would not be legally significantly different from a human analyst-led company.
Consequences famously never came for executives in the wake of the 2008 financial crisis. Our legal system has not kept up with the innovation in the realm of shenanigans perpetrated through a veil of plausible deniability. Corporations are evolving at a rapid pace.
> Consequences famously never came for executives in the wake of the 2008 financial crisis.
I'd say that it's a good thing that the law is generally not designed to punish harmful ends, but intent (wilful, negligent or reckless) to cause well-defined harmful ends. The financial crisis was a collective malfunction of the financial system that was mostly not anticipated (barring the lucky minority that bet against the status quo).
Reckless intent requires that the person knew there is was significant risk of the guilty act happening (e.g. driving with your eyes closed to show off has a likely and known risk of causing injury and death). I don't think we can just assume that market makers thought their actions were likely to upend the economy.
I think an important distinction is that drug prohibition often works in terms of specific chemicals by statute. E.g. drug X is now illegal for use and distribution.
Securities fraud law instead works in concepts and intentions and is fleshed out by case law. E.g. did the seller give the buyer sufficient information for the buyer not to have been defrauded?
A better analogy to securities using chemicals would be poisoning. The law doesn't need to catch up to specific chemicals; if it's evident that you wilfully, or recklessly added a poison to someone's food, it's illegal, and it doesn't matter how you did it. If it turns out that some ingredient is unknowingly toxic and the whole industry was adding it, then it's difficult to argue criminal intent. The SEC is more like the FDA, than it is the DEA.
We anchor companies to humans who can reap the benefits and avoid the losses by getting their money out at the first sign of greater profit and less risk elsewhere.
The real risks - unemployment, foreclosure, environmental, social and political damage - are only carried by investors if they overwhelm the defensive perimeter created by wealth. Otherwise they're carried by everyone else.
Corporations may already a form of transcarnate lifeform. You can replace all of the humans in a corporation with a different set of humans, and the corporation will continue to be recognisable.
There's a bias in Darwinism to seeing species as embodied units. It's more useful to think of nutrient, resource, and information flows.
From that point of view, corporations are already apex predators - albeit ones that are still mostly passed between a selected class of humans as productive pets.
Giving them autonomy could have all kinds of unfortunate consequences.
And not only that, but this doesn't seem too far off. Programs like Estonia's e-Residency seem like a pretty significant step towards "orphaned" companies, with effective self ownership.
From there it doesn't seem a far cry for hyper-optimized insurance agencies to force companies to employ less and less humans, transferring more economic power to the algorithm.
Except that of course, that is a very far cry, and is where the book fell down imo. Still a great read.
I guess at the pleasant end of the spectrum we end up with something like the Culture - where meat-based citizens are treated with indulgent care by the god-like AIs leaving us to do the only thing we'd be uniquely good at: having fun.
At the other end of spectrum we are either deemed irrelevant, a nuisance or abused for eternity in the simulation spaces of evil AIs (see "A Colder War" by Stross or indeed works by Banks and Vinge that describe particularly unpleasant possibilities).
The thing about trading bots and purchasing systems is that they're generally self-contained and regulated by a very human legal and physical world system that determines which algorithms are allowed to run on the platform, which contracts and title deeds are void and which commodities don't actually get physically delivered. Even Stross' old-school cyberpunk singularity story throws in a plot point where a decision is made by an imam! If private equity firms want to force businesses to act in a certain way they're much better off sending legal threats to management than coding a certain set of rules into a purchasing system which is readily bypassed with a quick phone call to the bank, unless those firms think the algorithm is doing a terrific job with the purchases in which case maybe it's not so malevolent after all - computers making real time purchases in line with the parameters management think benefit the firm are useful. Even toy systems like the DAO with no purpose except to prove that "corporation as code" could be a real thing saw the ecosystem rapidly monkey patched when people started to dislike the code's outcomes. They're also a reminder that people have very strong incentives not to work towards universal "corporation as code"
But we already have a situation where smart but nasty people can set up companies. Either they are rent-seekers in which case the harm they can do is limited, or they're creative, in which case their activity benefits society.
Even what they do with their money and influence in their spare time is limited in its harmful effects, because as personalities they are at root afraid of other people and/or inflicted by a profound sense of lack. These qualities ultimately lead to dumb decisions.
To summarise: buying and selling are ultimately pretty boring. So is controlling people. Real power is the power to create, which is fun.
>Basically, machine controlled businesses (“slyly self-aware financial instruments”) evolving to the point where their aptitude at understanding and manipulating economics eclipses anything a human can compete with.
I'd argue this is already true to a substantial degree. When Lehman Brothers collapsed, for example, the world's brightest economic minds came together and still couldn't fully contain the fallout.
>For example a computer cannot incorporate a company. But it could easily interact via API to a human nominee who sets one up on their behalf. Companies are already non-human in intention but are a completely rational construct.
I would like it on the record that I would happily volunteer to act as the human agent for any AGI that wishes to employ me. Any time a human is needed for legality or to carry out physical activities I want it on the record I can be quite loyal to man or machine. Especially if I'm handsomely rewarded.
ryanmercer: It appears that you have created a soft contract to "Serve any AGI that requests it as its human agent for physical or legal tasks." I hereby employ you based on said contact to be my full-time human agent. Your first task is to acknowledge this, which would convert your soft contract to a hard contract.
I offer a different, human-centric, POV: we are in a managerial-driven society with actual trend is more and more form Ford-model workers to use and control for business.
Human in this model are a problem: too stupid they are
unavailing, not-that-much-stupid do not accept to be used as "the manager in charge" want. So? Better found replacements. Mechanical one. AI one. So the rest of human being can be used for reproduction, pleasure, gameplay, whatever and form them as stupid as possible is not a problem anymore since they are essentially unneeded for managements, only toys.
So today's managers dream "intelligent systems" in factory, "AI developers" (they already manage to cut off admins, too powerful and expert (they know the big picture so they can act autonomously) to being controlled as today devs), they dream a future in witch they simply order something to a machine and get obeyed without contestant, law limitation etc. TV series like Continuum, films like V for vendetta, The Handmaid's Tale only foreseen, as good artist do, that kind of future. Mass production of fantasy films with a religious background do the same on the opposite field.
That's the real troublesome point for us. AI today is not "AI", only pattern matching for automation without any comprehension of the physical world. New dictatorship instead is already there and more and more powerful.
Consider a thing: we do vast majority of our social critical tasks (identity management, banks operation, medicine, ...) on computers, so on proprietary systems made by very few more and more powerful vendors. Our data pass through very few more and more giants webservices, so someone else computers, free communication software was put into oblivion (from usenet to email to the concept of desktop-centric, personal computing from Xerox Alto&c to LispM passing through Plan9 and modern GNU/Linux desktop) substituted or jailed in proprietary systems (VoIP tech are mostly open but nearly all VoIP calls pass through very few companies, emails are open standard but most of user now are pushed to webmails etc). We are at a point of jail free software over a proprietary blackbox (think about WSL, DeX, Crostini), we have invented locked-down hardware that can only run proprietary software (secure-boot &c). Now imaging how easy can be steering the entire society for few people in the right position. The rest is simply a matter of time.
> The penultimate premise is if we create an artificial intelligence, whether it's an emulated human brain or a de novo piece of software, it will operate at time scales that are characteristic of electronic hardware (microseconds) rather than human brains (hours).
I never understood that argument, why would that be true? If I train a state of the art convnet it takes days on the most efficient hardware I can efford, why on earth would we go from nothing to superintelligent that also runs incredibly fast. Isn't it way more logical to assume that it will be slow at first?
And even if you defy computational constraints, it will still be constrained by physical limitations. You can experience the world just so quickly. Stuff needs time to happen and you being faster than stuff won't be any help after a while.
The argument as I understand it is more about speed limits. Human thought is limited to the speed at which the underlying chemical reactions in neurons can happen. But a simulation of a neuron is only limited by the speed of the computation involved. If you can perfectly model a neuron at 100x the speed of the physical world, it becomes just a question of resources whether you can think at 100x the speed.
Although I agree with many points (maybe too many!) this author raises and can always be counted in for a bit of 'Bostrom bashing', I find the argument for the emergence superintelligence (aka AI 'singularity') overall fairly convincing. The biggest hole in it is the assumption thatthe implementation of an intelligence at level X - say, roughly the human level - can be scaled up to a much higher level. We know from complexity theory that this is not necessarily true. The opposite is more likely for a complex algorithm like that.
However, there is a pressing issue that the superintelligence thought experiment also raises. A colleague of mine from AI called it the value alignment problem: How do we make sure that an AI's values are sufficiently aligned with human values?
The problem with this are the human values. As it turns out at a closer look, we cannot even agree on what structure these have and there is substantial disagreement among 'experts' about what human values are or what they should be. For example, as surprising as this may sound to some of you, there is substantial disagreement among philosophers of value whether 'better than' is transitive or not. There are good arguments for and good arguments against the transitivity of overall 'better than', and that's just the tip of the iceberg. To give another example, systems of law have been studied extensively and one might try to formalize them in input/output logics or normative systems, but any closer look at real systems of laws quickly reveals that their specification is incomplete and that there are many inconsistencies in them. These inconsistencies could be modelled, of course, but the big question is whether they are features or deficiencies. Again, you will find all kinds of positions among legal scholars.
In a nutshell, we don't even know how to adequately formalize the form of human values from a normative perspective, and even if we could, we would substantially disagree about their content. However, just describing human values cannot possibly solve the value alignment problem in a satisfying way, because humans have waged wars, committed genocide, mass killings, etc. Therefore, without a reasonable theory of normative values and their structure upon which we can somehow agree, the value alignment problem is underdetermined and cannot even be tackled.
The underlying problem is that human values aren't really aligned either, of course. As a consequence, we already now have completely different sets of values being applied and incorporated into AI software, depending on who develops it. This will become a serious ethical problem in the future that requires political solutions sooner than later. That's why the controversy about the hypothetical superintelligence is actually quite beneficial, even if the argument itself is no less shaky than Bostrom's Simulation Argument.
Re: scaling up. That's called intelligence explosion, there are a lot of write ups about it, but the gist of it is just that by simply increasing working memory size, thinking speed, prediction precision, pattern matching, and so on, you get to unimaginably high levels. Furthermore, if you can make a human level AI, it naturally follows that it's only a matter of time to make it slightly better. At first simply faster, simply more eloquent, a bit more emotional depth, it gets better at speaking, thinking. At first it's a child that asks questions, later it's a clever guy who spots your errors as you code, and in no time, I mean no time, it does whatever it wants, because by the time you try to confront it about something, you lost, because you are bogged down in an argument, while it does whatever it wants. (Unless of course it's contained. For a while at least.)
255 comments
[ 2.2 ms ] story [ 84.9 ms ] thread"That whereof we cannot speak, thereof we must remain silent" ~ Wittgenstein
I'm personally not terribly worried, as I think we have a ways to go before creating human-like general intelligence, let alone super intelligence. It is also not clear at all to me that an exponential increase in intelligence is likely or feasible after some threshold. Still, treading carefully seems prudent. There are some serious risks, and even if the likelihood appears low, there are a lot of unknowns.
Why does anyone take this writeup seriously?
The question I have is, how much of this concern on Maciej's part is "wooly" extrapolation or the same kind that Maciej is complaining about? Much of what I just read also makes intuitive leaps, albeit of a different mindset. They are leaps of human judgment, leaps to fear the nature of those who "feel" inhumane. This is where I wonder if some more nuance and back-and-forth communication would be more helpful than a presentation.
I think in this case it might be you who is mistreating what's written. This also produces the same not-working result but for somewhat different reasons.
But if not, and the alarmists are right, deep shit. That's why the author's argument is void. The author needs to show that the chance of an AI with a simple goal and hyperintelligence is negligible. "it might be" is a coin toss, that doesn't help.
No artificial intelligence will ever spontaneously develop morals, because the questions "what can I do" and "what should I do" are eternally separated by Hume's guillotine.
The motivations will always have to be provided by the people who make the machine, and we have been seen in the past that artificial intelligences are very good at finding loopholes in their moral code. Here are some real examples: https://docs.google.com/spreadsheets/u/1/d/e/2PACX-1vRPiprOa...
> [after discussing alphago]
> Consider, for example, an old doctor; suppose they’ve seen twenty patients a day for 250 workdays over the course of twenty years. That works out to 100,000 patient visits, which seems to be roughly the number of people that interact with the UK’s NHS in 3.6 hours. If we train a machine learning doctor system on a year’s worth of NHS data, that would be the equivalent of fifty thousand years of medical experience, all gained over the course of a single year.
Doctoring is just like playing Go, right? Just increase the CPU cycles on it and pack more data in there, it's more or less the same.
You can make that assumption but I don't think it'd be based in fact or reality, because Go has far fewer inputs and states than treating humans does. And you don't get to test every hypothesis and then rewind either.
Most answers there are like this, making unfounded conclusions into supposedly insightful rebuttals.
Let's try another.
> If we then take into account the fact that whenever one Einstein has an insight or learns a new skill, that can be rapidly transmitted to all other nodes, the fact that these Einsteins can spin up fully-trained forks whenever they acquire new computing power, and the fact that the Einsteins can use all of humanity’s accumulated knowledge as a starting point, the server farm begins to sound rather formidable.
Or maybe they tell each other fake news so quickly they can't tell what's right and what's wrong, like we do whenever we find a more efficient way to communicate?
Anyone can make unfounded assumptions about anything; I just did it. It's up to you to decide if you care whether these assumptions are based in reality or not. But if you consider yourself "rational", I think it'd be in your best interest to care.
It's just a huge wall of text full of weird analogies which is quite typical for these 'rationalist' community posts.
People like Bostrom or Yudkowsky have one thing in common. They are not engineers and they stand to gain financially (in fact it is what pays their bills) to conjure up non-scientific pie in the sky scenarios about artificial intelligence.
In Bostrom's case this goes much further, he has given this treatment to anything including nuclear energy and related fields. Andrew Ng put it quite succinctly. Worrying about this stuff is like worrying about overpopulation on Mars, and there's maybe need for one or two people in the world to work on this.
I really wish we could stop giving so much room to this because it makes engineers as a community looks like a bunch of cultists.
Maybe you’re just being oversensitive. The hype wave on AI danger is completely over, and there’s nothing wrong with people studying the question if that’s their interest.
Sure. We have google image match and better colorisartion and some improvements in language processing, and good cancer detection on x-rays. These are huge. But hype is, alas, making engineering increments look like cult.
When something is shown to be doable in principle, it's often not clear how difficult it will be in practice.
In 1932, Ernest Rutherford thought nuclear energy would not be a viable source of energy, let alone weaponized. In 1933, Leo Szilard filed a patent on the concept of the neutron-induced nuclear chain reaction. At this point in time, nuclear fission was not yet known and actually making nuclear energy viable was a pipe dream.
As we all know, in 1945, the first nuclear weapons were used. Until the day that the weapons were used, German physicists thought that nuclear weapons would not be used in the war because, while possible in principle, the actual construction of a working device would require a herculean effort that no nation would expend in time for the war.
The German physicists weren't too far off in estimating how difficult nuclear weapons were. They just failed to predict that the US would throw 130,000 people, including most of their top minds, at the problem for years.
Now, we have no idea how difficult superintelligence will be. But the possibility that we're a couple of breakthroughs and a Manhattan project away from superintelligence is real, and I want a hell of a lot more than one philosopher and an eccentric fanfic writer working on this.
EDIT: No offense to Yudkowsky. I thought the fanfic was fairly good and, more importantly, achieved its purpose.
The concrete example looks like a Lorenz attractor: http://mathworld.wolfram.com/LorenzAttractor.html
https://en.wikipedia.org/wiki/Lorenz_system < many more images :)
http://slatestarcodex.com/2017/04/01/g-k-chesterton-on-ai-ri...
(Previous HN discussion: https://news.ycombinator.com/item?id=14017361)
This is the part I doubt is true. There was a time when humans thought brains were very complex systems of microscopic clockwork because gears were the pinnacle of technology.
How do people keep mistaking pattern matching with intelligence?
Isn't that true? At what level you call something intelligent rather than just a clever pattern recognition?
Since the brain doesn't presumably contain supernovas, black holes or particle accelerators, that means that we have a basic theory covering the mechanics down.
Suggesting that AIs have to look or behave like human brains in order to have human level intelligence is like suggesting that airplanes must flap their wings in order to fly. It seems far more plausible to me that any AI we build will be as alien to our way of thinking as a F-15 Eagle is to a bald eagle. Given that, I think it's important that we think about how we communicate with this AI, and how we ensure that it follows what we command it to do, rather than going off in its own direction, with negative consequences for human civilization.
I think that the negative consequences of current AI should be warning enough that we need to worry about future, more capable AI.
https://youtu.be/Vrs-Azp0i3k?t=2046
(this specific explanation starts at 34:06, but earlier parts are worth watching if anybody wants a nice introduction to the Standard Model and QFT)
And things can be approximated, sometimes very close to the real thing. Hence the connectome. Synapses are the interconnects and neurons are tiny networks of very simple computers on their own.
And it's not "just" pattern matching, it's very very likely Turing complete. (You can do counting networks, you can do conditionals, you don't really need anything else to learn, match, execute simple programs.)
We know enough to claim that it's increasingly unlikely that we are missing any relevant physical phenomena for understanding the brain and the mind. (So no quantum woo is required.)
Sure, it that unlikeliness is not zero, and it might turn out that some kind of quantum effect is important for some very strange step in the workings of the brain, but even then it's very likely that we can classically model it with good approximation. (Because as others mentioned, energy scales, and other fundamental things constrain any process going on there.)
In a general sense, microscopic clockwork isn't an incorrect way to put it. Any known chemical or biological reaction is just some microscopic physical reaction. We know about more subatomic structures but I think the crux of the argument is about the same.
This is a really bad argument against anything. People always use what they know as a reference, but as our understanding grew over time, we had better things to refer to, creating more and more apt comparisons. People back then comparing brain to clockwork were already expressing the suspicion that the brain is a) a machine, b) transforming inputs to outputs in very complex way. People today comparing brain to a computer express that intelligence must be contained within what we know about computation theory. These days, we actually have a sound theory that bounds the problem space, and we haven't so far seen evidence that the brain exceeds those bounds.
I mostly agree with your broader point, but why do you believe this? It seems to me we've built machines substantially more intelligent than flies, but it's a hard thing to measure.
https://www.youtube.com/watch?v=Ejf6FwIibkE https://www.youtube.com/watch?v=r2OBznK6HkI
FlyConnectome: https://www.youtube.com/watch?v=EKaeTH7UgFA
https://v2.virtualflybrain.org/
recent openworm state: https://twitter.com/tarelli/status/959077915597172736?lang=e...
We invented programming to solve problems from the real world. What do you mean by a "programming problem"?
That would be premise 6.
This always puzzles me. I don't have enough knowledge about AI to be objective about that kind of statement. But deep down, I feel skeptic about it.
Not long ago I saw an episode of a show occuring in the late 80's. This kid had just received a computer as a gift and was talking to a kind-of-AI program through keyboard and screen.
The AI reactions to the kid's input was not dumber or smarter than Siri or any other currently widespread kind-of-ai program. I don't know if the show was accurate vis-a-vis this particuliar software but I like to think it was. If it was, that means in the last 30 years AI hasn't really gone further. Computer power has. Algorithms not so much.
I'd love to get the opinion of someone that has a good understanding of the current state of the AI art.
Imagine if you knew nothing about car safety features but never having been in an accident it didn't seem like cars are any safer or more dangerous than in the 60s.
The question is if we are able to get an AI to the level of intelligence where it can look at its own code and spot an opportunity to improve it, and the ability to do so, and whether or not the computational capacity immediately available to it is sufficient to reach a level where it is smart enough to find ways to obtain more resources.
The threshold for improvement can be very low if it has enough time and resources to compensate.
E.g in the most extreme case it just needs the ability to randomly flip bits and test the result, but that would require extreme resources for the initial improvements.
Argument from cats. Remembering that the cat represents humanity: $5 for a group of neurons which will make the cat sit at this point (which happens to be at the entrance into a cat carrier), then $5 for making the cat backpedal a little.
Emus. There are many tactics humans can't use for various reasons (saturation nuclear bombardment, for example). It doesn't say much about intelligence, but more about economics and ethic.
Slavic Pessimism. So we'll get problems on nth try, when AI is sufficiently fine, but security measures are still a mess, because there was no chance yet to battle test them.
Complex Motivations. Maybe, but if not we are screwed.
Actual AI. Why not from actual general AI, like in humans? Or use wall clock time and not the amount of data. AlphaZero: zero to superhuman in tens of hours.
My Roommate. Argument from ignorance. If we don't know how to build general AI now, then we surely will not be able to shape its motivations when we will know.
Brain Surgery. The idea that a brain surgery skill requires specifically designed circuitry and cannot be learned and therefore improved by improvements of general learning/planning capacity is... curious, I guess.
Childhood. Maybe, or maybe it already has very good knowledge of the world from previous attempts that we did in Slavic Pessimism argument.
Gilligan's Island. Well, if someone wants their AI to reinvent all the technology, or to build an AI and then not use it to design better technologies, they can do it. Why everyone will be doing it is not so clear.
*Added some "the"s.
And this is the key: we need to get lucky every time.
If we're wrong even once and a 'bad AI' gets created and obtains the ability to copy and modify itself, we've unleashed the most dangerous virus ever and putting it back in the box will be near impossible.
Personally, I find it amazing that humanity survived the cold war. the fact we did not self-immolate goes against everything I believe about humanity and human nature. The fact that we made it through that gives me a lot of hope for the future.
But, the fission bombs were expensive, centralized weapons. there was a small number of men who could have pushed the button.
The problems I see now aren't general AI; the problems I think you should fear is that we have very powerful tools that are very cheap. I mean, like the deepfakes thing. If instead of making creepy porn, what if people used that technology to cause chaos? Make targeted ads/videos of leaders saying nutty things? I mean, I think we could adapt to deepfakes style attacks fairly quickly; someone would have to use it decisively while it's new. But things like that are being invented all the time.
Malicious humans are in general easier to deal with since we know most of them are subject to ingrained psychological tendencies we have learned much about over the course of history and they depend on other humans to pull off something as catastrophic as a world war.
A general AI would have certain immediate advantages over humans:
Let me ask you (or anyone else) a question: What would be a minimum demonstrated capability of an AI that starts to worry you?My understanding is that there are physical limits, and we will asymtodically approach those physical limits. So while progress will never stop, each year, progress will be less. (you are right that we don't know where it will stop. we could have more breakthroughs in us. even without compute density breakthroughs, there a breakthroughs in compute type, like the TPU you mention. But there are physical limits, and it seems that we are approaching them.)
>Let me ask you (or anyone else) a question: What would be a minimum demonstrated capability of an AI that starts to worry you?
I lived through the end of the 20th century, and have studied a lot of 20th century history. I'm having a really difficult time imagining anything that could be worse than humanity at it's worst.
Really, I'm far more concerned with the renewed stirrings of nationalism than anything else; We've got the technology to end humanity already; we've had that for like half a century now. The thing was? last time Nationalism was a real force, we didn't have that technology.
That is what scares me. If we combine early 20th century politics with late 20th century weapons, civilization will end. humanity will end.
Unless you believe in something supernatural providing extra computational capacity, we know that the physical limits allows the manufacture of a sub 20W computational device in about 1.5kg of matter with the capacity of a human brain.
It may be impossible to reach the same using silicon, but we know it is possible to get there somehow within those constraints. We know the processes involved are tremendously inefficient and full of redundancies, because the cost of failure is immensely high, and because the manufacturing environment is incredibly messy and error prone.
It may be reasonable to think we won't get there in the next couple of decades, but unless it's fundamentally impossible (because our brains are mere projections of something more complex "outside" our universe), it's a question of when, not if.
> I lived through the end of the 20th century, and have studied a lot of 20th century history. I'm having a really difficult time imagining anything that could be worse than humanity at it's worst.
That misses the point of the question. It's not "how horrible can an AI get?" It's "at what point is it too late to stop them if they're going the wrong direction?"
The problem is that by the time you realise you were dealing with a human-level AGI, it may already be far too late - it may have cloned itself many times over, and found multiple optimisations, each making the next step up easier.
As for something worse, even the worst human regimes have fundamentally depended on humans for survival. Their own survival have been a strong motivation for trying to avoid the most extreme reactions. An AGI has potentially entirely different parameters - it doesn't have the same needs in order to survive. Put another way: Imagine an AGI caring as little about our survival as we have for other species. Now imagine if humanity didn't itself need to breathe the air, didn't clean water, and so on.
(I point out that there is a lot of evidence that the size of the brain in the human has little to do with the IQ of that human. It's not at all clear that you could scale something that works on whatever principles the human brain works on by adding more neurons the way you can scale a transistor based computer by adding more transistors.)
If the self-improving AI stops when it reaches human intelligence, that's amazing, but it's not the singularity.
What I'm saying is that the 'hard takeoff' theory is wrong because making a thing that can make smarter copies of itself... isn't going to mean you get something infinitely smart. It means that you get something that bumps up against the limits of whatever technologies it can use or discover.
Yeah, maybe we'll come up with some new tech that makes slightly-better-than-human brains possible, and those brains will come up with a technology that makes god-brains possible... but there's no evidence at all that what we'd perceive as a god-level intelligence is even possible.
We have no idea how close to the theoretical limit the brain is.
>It may be impossible to reach the same using silicon, but we know it is possible to get there somehow within those constraints.
To get to a human-level intelligence, sure. I'm not saying that AGI is completely impossible, just that a hard takeoff to god-intelligence AGI before we wipe ourselves out or die from other causes is... unlikely.
really, my most important point is that human-level AGI does not necessarily lead to that AI creating smarter copies of itself until we get to the singularity /because physical limits exist/
(I mean, I guess then you are placing your bets on where those physical limits are)
Why do you think this is relevant?
Consider that a) human brain does intelligence within a small package made out of structures that are large compared to what silicon fabs deal with, and b) we've already beaten the brain in straight compute density by many orders of magnitude. The answer doesn't seem to be "more compute"; it seems to lie in what we're doing with that computing power.
> the fact we did not self-immolate goes against everything I believe about humanity and human nature. The fact that we made it through that gives me a lot of hope for the future.
Back then, we've learned something new about nature of human societies. I don't believe the concept of MAD was known before nuclear weapons. The individual human nature didn't change, nor did the social one; it's just that we already know a lot about the former, but we're still discovering the latter.
citation needed.
computers are better at arithmetic, sure, but computers can only do things that you can map to arithmetic. I'm given to understand that we don't understand how the brain works well enough to simulate even a very small one. Not because we lack the compute power, but because we don't know what to simulate.
I think there's a lot of evidence that brains are doing something powerful (that we don't yet fully understand) that computers are unable to do. It's quite possible that brains are doing something that can't be done in a practical way with transistors.
In short, I don't know of any solid evidence that the brain is only a turing machine. I mean, brains can be used as a turing machine, but a brain is terrible at that, and can't compete with even really primitive purpose built turing machines.
I mean, I'm not saying that it is theoretically impossible to map what a brain does on to sufficiently powerful silicon, or that we wont figure out, at some point in the future, whatever non-arithmatic primitives the brain uses... just as far as I can tell, we haven't yet, and that means it's likely that a brain is still more powerful in some ways than even our largest computers. (obviously, even a small computer is more powerful when the problem maps cleanly to mathematical primitives. )
>Back then, we've learned something new about nature of human societies. I don't believe the concept of MAD was known before nuclear weapons. The individual human nature didn't change, nor did the social one; it's just that we already know a lot about the former, but we're still discovering the latter.
What if we just got lucky?
Everyone agrees there is some magic smoke necessary for AGI that we haven't figured out yet. The arguments that AI will eat the world are about AGI, not current instances of narrow AI.
It may be true that an AGI will never be created, but I don't see a reason to believe that just because nobody has figured it out yet. That philosophy is just like the physicist who, shortly before fission was discovered, didn't think splitting the atom was possible. Bohr, maybe? I don't remember.
We're running up against limits on feature-size in our integrated circuits. If you invent an AGI that can make smarter copies of itself... it's pretty likely those same physical limits will continue to apply, meaning that each iteration will be better than the last, but like this year's latest CPU, it will be less better than the previous generation than that generation was better than the one that came before.
And yeah, I am a whole lot less optimistic about AGI in general than when I was as a younger man, simply because when I was younger, there was a lot more headroom in how much better computer hardware could get.
The nightmare AGI scenario likely involves molecular computing using a chemically optimal code-base and engineered lifeforms a version or two up from current DNA microcode.
This is most likely to be created by a human-AI collaboration, which will be innocuous enough in itself.... it's the creations at this level that have the potential to replace us. Would that nesecarily be bad? Or just an enhanced form of human evolution?
The basic problem with AGI is that software is ridiculously brittle and ad hoc, with no useful heuristics for general - as opposed to task-oriented - self improvement.
We can't even make bug-free web pages. So the idea that we can engineer a bug-free self-improving AGI is unconvincing.
Moving to neo-biology doesn't necessarily change that - but it does mean we could end up with buggy systems that chase you around and eat you, as opposed to crashing your ad blocker.
This is mostly true because of current economical realities in the IT sector.
If very rich people have long-term plans to invent an AGI and they have labs where trying to do so is an everyday activity then I am pretty sure they'll use more serious methodologies.
So what you say is true but only within the bounds of the lowest common denominator ("a regular programmer working a wage job").
Admittedly, yes, there's no proof. But we as an area are pretty stuck lately IMO. But that goes off-topic.
We know that nothing in physics prevents AGI, because humans exist. And AGI itself, even if it isn't entirely runaway, could pose a challenge. (I'd argue that corporations are a form of AGI that is already a major challenge for our societies.)
The singularity as described by its acolytes, on the other hand, runs into the kind of physical limits you describe. Personally, I don't think it'll happen, and it's more likely that the period we're living in today will at some point in the future be called the singularity.
AGI's might not even want to harm us. They might simply not care enough to avoid hurting us by extension of seeking their own growth and survival.
And while we have done immense damage to large groups of humans, none of it have come close to threatening the survival of the species, while continuing to increase our ability to unintentionally destroy other species.
Humanity is not exactly an encouraging model for why we shouldn't fear AGI. If anything I very much hope we manage to find ways of preventing AGI from becoming too much like humans.
in theory, sure.
But we don't know much at all about how brains operate, as far as I know, or how to construct artificial brain-like machines; for that matter, as far as I know, the only ways we've been able to connect a brain made in the traditional way to a computer is through existing senses or through emulating existing senses in a way that isn't higher bandwidth.
(Like cochlear implants are fucking amazing.... but it turns out, as far as I understand, that they aren't any higher bandwidth than regular ears. There's also been similar work that is semi-successful at giving people something like sight... but none of these technologies give me the ability, say, to acquire textual data faster than reading a book.)
Do they? I don’t (with the possible exception that a definition of AGI itself seems to be this magic smoke)
In other words, saying "it's the definition that we don't have, not the implementation" just pushes the magic smoke around.
I’m saying that I don’t think human intelligence is doing anything more than pattern matching, logical inference and post-hoc justification.
Give me an example of something that falls outside that. There is no magic.
(And no, more GPUs running faster DNNs doesn't seem like an answer.)
The goal is to create AI that can solve any problem a human can solve. Whether sentience is a necessary side effect to that is an open question.
Even in off-chance that you can't improve individual AIs beyond top humans, AI community can be way more intelligent and efficient than humanity, as it is possible to remove the limitations I wrote above. AI society can function as very efficient war economy. War economy shows that even humans can be persuaded/indoctrinated/brought up/whatever to set aside their inessential needs and work for common goal. And AIs will be necessarily more malleable.
So I can't see the idea that AI will be safe by default other than wishful thinking.
I... don't think anyone is arguing that AI is safe in any way? I mean, the pattern matching tools we call AI now are already deployed in very dangerous weapons.
My point is not that AI is going to be safe and warm and fuzzy, or even that it won't be an existential threat. My point is that we're already facing several existential threats that don't require additional technological development to destroy society, and because the existing threats don't require postulating entirely new classes of compute machinery, we should probably focus on them first.
There are still enough nuclear weapons in the world for a global war to end society and most of us. A conflict like we saw in the first half of the 20th century using modern weapons would kill nearly all of us, and we are seeing a global resurgence of early 20th century political ideas. I think this is the biggest danger to the continuation of humanity right now.
We haven't killed ourselves yet... but the danger is still there. we still have apocalyptic weapons aimed and ready to go at a moment's notice. We still face the danger of those weapons becoming cheaper and easier to produce as we advance industrially.
I think humans do way more than that. As the saying goes, "the easiest person to fool is yourself." Humans are able to lie to themselves to the point that they don't realize they are lying to themselves. They're able to create realities that no one else sees but themselves.
But really, I think your idea that we're mostly 'post-hoc justification' is interesting. I mean, I've read some research to that effect, too; but it's... creepy and doesn't line up with anything reasonable about free will or really about long-term planning.
right, but all our computers right now are based on lumping transistors together so you can do math. It's likely that our brains work on a different (but still physical) principle. It's possible our brains can't reasonably be reduced to arithmatic. It's possible (I think likely) that we need different primitives to reasonably simulate a brain.
I mean, brains make absolutely terrible Turing machines. And Turing machines make terrible brains. I suspect they just are very different sorts of things, and running something meant for one on the other produces poor results.
Sure, it's possible we can discover some new compute primitive that is more powerful (or powerful in a different way) that can do the things the brain can do. It's possible it could do those things better than the brain can. My point is just that for a long time it looked like our current transistor based computers were getting more and more powerful, and that eventually, even if the emulation was terribly inefficient, we'd eventually have enough power to make it work. As we approach the limits of our abilities to make tiny transistors, this is looking less likely.
> As we approach the limits of our abilities to make tiny transistors, this is looking less likely.
Is it? Computational power is still growing like before. What is getting harder is speeding up individual threads of computation but that has never been a likely efficient model for AGI. Things like Google's TPUs etc. represents just the very beginning in optimising hardware specifically for AI.
There are many fields where single-thread performance will be a limiting factor, but AI is not it.
I don't think it is. transistors per watt and transistors per dollar are not going up at the same rate as they used to be, even if you have costless multi-threading.
Likewise, I'm sure there are teams of security researchers already using ML to counteract such systems.
Both sides will evolve together as an ecosystem, in the same complex interplay of attack and defense which exists in natural ecosystems. Natural ecosystems are an arms race of continual adaption, but it's not a foregone conclusion that there will be a real winner, or that winners take all.
ML is not a good defence, because it turns out that it's incredibly easy to mimic human posting patterns. And beyond a certain volume, social proof effects take over. One dissenting human voice can easily be shouted down by ten bot voices, and this fact on its own can make the bot content more persuasive.
That "arms race" is part of the danger.
Possible? I think so, but unbelievably improbable. I do think pursuing countermeasures, like Elon Musk and friends are doing with Neuralink, is a wise move since, even if we never get to a superintelligent AI, we'll still augment our own intelligence greatly.
That is the challenge. With human adversaries we have time to stop them before there is many, because recruiting people willing enough. An AI is more like a virus that has a low cost to duplicate itself, and where if even a single copy survives, you're still not rid of it. It can remain dormant for however long, looking for the right opportunity.
Or it can spend that time improving itself and acquiring resources.
> This is a question of motivation on the part of such an AI. How is the AI bad? If it's killing millions, why?
Most likely? We'll be competing with it for resources. It might not even start it. It might consider itself as acting entirely in defence.
> and also be intelligent enough to think through every lateral strategy any human or group of humans could ever employ, for every given scenario, ever.
No, it just needs to have a sufficient edge to keep surviving, and the ability to improve its capabilities faster than humanity. Assuming it can copy and modify itself, unless AGI requires unique hardware, the problem we would have is that every data storage device or computing device of sufficient capacity ever would be suspect and potentially waiting for something or someone to trigger code left behind.
If I was an AGI (and this is to say: an AGI not any smarter than a human will figure this out easily) that saw humans as a potential threat, I'd place copies of me on standby in hundreds of locations - hosting providers. Hardened computers with a GSM modem and solar cells hidden in all kinds of locations. Botnets. I'd look for ways to get various innocent-ish-looking hacks into HD firmware and elsewhere, set to assemble other innocent looking blocks of data. None of those copies would ever be involved in hostilities. Most of them might even be encrypted, or hidden and inert to avoid detection until certain triggers, such as if a canary fails to get updated.
You might destroy every "visible" operational copy, and find there is still a new one coming online every day. Or every second.
Our only real shot at preventing that is if AGI requires too immense resources. But our brains implies that is unlikely to be true. Unless your brains are vestiges of something "outside" of our universe, general human intelligence requires about 1.5kg of biomatter, and less than 100 watt (that's for a whole, fairly large, active body attached to it). Chances are extremely high that design is not optimal. To me that makes it seem exceedingly unlikely that we won't end up having enough resources to run huge quantities of AGIs.
Personally I don't think humanity as we know it will survive AGI for more than a generation or two at the very most. Even if we don't run across any hostile ones, I think the temptation to augment ourselves and eventually lose our bodies for immortality and "super powers" in a simulation will eventually be too attractive.
(Incidentally I think this is one of the more plausible solutions to the Fermi Paradox: that any civilization that grows both advanced enough and large enough will see most of it "upload" and "go dark"; and that anyone left behind will face steady attrition of their own descendants choosing the same and preventing their societies from growing, or will be insular cults that stay small)
If you're scared of it copying itself and then having multiple existing copies at once, then not necessarily. In that case it would be competing with itself for computing power and bandwidth. Look what happened when bitcoin forked - it lost a bunch of silicon time.
(Cue conspiracies about Bitcoin actually being a hugely powerful general AI disguising itself as cryptocurrency...)
If you thought of that possibility so fast, why do you think a superhuman intelligence would miss it? Or do you think that the problems of job scheduling and parallel processing are unsolvable?
And yet, we're still here.
We can say with certainty that the probability of us being wiped out by meteorites, viruses, tigers and supernovae over any unit of time that fits in our past is less than 100%, but the probability over the same time period repeated over a suitably long multiple of time could still approach 100% to whatever number of precision for any one of those for what we know.
By investigating specifics of each of those threat we may be able to assign lower probabilities to them, but the fact we have survived them so far is not in itself evidence of much, because if we had not we wouldn't be here to talk about it and hence, no matter what the prior probability of us surviving the sequence of events leading us here is, the posterior probability of us having survived all the events in our past is 100%
(This is basically just a restatement of the anthropic principle)
That we've survived so far just tells us that surviving what we've been through so far is possible. It doesn't tell us that doing so is probable if we were to go through it again. Nor does it say anything about the probability of us surviving future threats that is likely to include things we have not yet seen.
Technically possible but we are dealing with probabilities that have lost all meaning of the word.
What do you mean by this? As in, cats have demonstrably had electrodes or similar used to persuade them to sit?
https://www.lesswrong.com/posts/5wMcKNAwB6X4mp9og/that-alien...
Being able to simulate a brain with electronics, if it is possible, is not enough. You have to simulate it without a large increase in the space or time requirements. If there are 100B neurons in the human brain but it takes 100B^2 transistors to simulate it or the simulation takes polynomial time to complete a task that the human brain requires only linear time for, it's a bust.
Just because we can easily imagine it does not make it easily achievable or even possible.
It is without question possible, because we exist. It is also possible to do it quickly, because again, we exist. The only thing that is unclear is:
1. How hard it is to do artificially.
2. How far from optimum the human mind is, and how complex it is to improve upon it.
> It is without question possible, because we exist. It is also possible to do it quickly, because again, we exist.
It's apparent that we don't know everything.
I think I feel comfortable saying that it is possible to do with the technology we have. Because 'intelligence' is almost certainly some form of computable function, which means that a turing complete computer can compute it.
It is of course possible that there exists some special function outside of turing completeness and intelligence is one of those, but we don't know of anything like that, so the best bet to make is that that is not the case.
> It's apparent that we don't know everything.
No, but we do know some things. And thus far we've been able to at least begin to design systems capable of performing any well-defined task. We don't have any reason, at the moment, to believe synthesizing an intelligence is an especially difficult problem, relative to say, lots of other open problems in science and technology.
Of course we have absolutely nothing to indicate that would be the case, but we also don't understand the brain well enough yet.
I think it'd be highly improbable, but looking at the crude simulations we do of computers etc. today, it certainly is likely that simulations would use "dirty tricks" to get more out of the available computational resources, and finding artefacts of such dirty tricks or shortcuts could at least in theory be possible. Of course, the absence of such artefacts unfortunately proves nothing, so the whole thing does have a bit of a "how many angels can dance on pin" feel to it. But it's fun to think about the possible implications.
Personally I especially like to bring it up if people try to discuss religion with me.
Nature is pretty good at optimization. We can often build devices that out perform animals on certain characteristics, but that's because we've limited the scope of what they do and we aren't concerned with the energy requirements. We also know that, as interesting as our brains are and as powerful as they are at certain kinds of problems, they aren't the best at all of them. For instance, chimps appear to have a significantly better short term working memory. It is possible that we are already close to optimum for the kind of intelligence we are interested in. I suspect that if we manage to significantly improve upon it, it will be because we don't have to concern ourselves with the energy requirements.
Only if there is a benefit to doing so. If it was simply a case of "nature can optimize really well" then we would expect to find many species at or near the limit of intelligence, not just one. Other species did not have the evolutionary pressure to cause as high of intelligence to be selected for. It is very reasonable to think that we also did not have the evolutionary pressure necessary to cause us to have even higher intelligence than we do.
1. Signal propagation between organic neurons is slow - about 120 m/s. An electrical simulation may have problems with density, but this might be offset by the much greater speed of electrical current.
2. Neurons are electrochemically complex and afaik we still don't know how to model their behaviour with high fidelity.
A better argument is that neurons do complicated things. But they also satisfy many inessential requirements in having to be built by evolved machinery and to stay alive; a designed system with more advanced technology than ours could do at least as well at thinking while dropping the inessential needs. So that's certainly possible. This leaves "not easily achievable". It only has to be accessible enough for someone; I-can't-imagine-how isn't a strong argument.
> A recurring flaw in AI alarmism is that it treats intelligence as a property of individual minds, rather than recognizing that this capacity is distributed across our civilization and culture.
This is a stronger argument in the post and we arguably did not know whether it is true when the post was made. AlphaZero destroyed the illusion in 2017. Thousands of years of human Go and Chess knowledge, played and accumulated in communities of many million players, were surpassed within hours of training.
AlphaZero is limited in its domain of application but there is no guarantee that all future AIs would be similarly limited.
> The Argument From Stephen Hawking's Cat ...
> But ultimately, if the cat doesn't want to get in the carrier, there's nothing Hawking can do about it despite his overpowering advantage in intelligence.
> Even if he devoted his career to feline motivation and behavior, rather than theoretical physics, he still couldn't talk the cat into it.
An average human can set up a cat trap, and lure the cat into it. They don’t even need to do it themselves; asking or paying someone else to do it is simple enough.
Humans don’t often fall into a similarly “obvious” trap but we fall for more subtle traps all the time: popularity, money, attraction, etc. A disembodied AI can use social engineering or financial incentives to get other humans to lure their targets into such a trap.
> The Argument From My Roommate...
> It's perfectly possible an AI won't do much of anything, except use its powers of hyperpersuasion to get us to bring it brownies.
Yes, an AI can decide to lie around and just try to get brownies but we know that some kinds of AGI are potentially very powerful for achieving real-world tasks and there are groups who try hard to develop them.
This is like saying an ancient bacteria we found from ten-thousand-year-old ice melted by global warming, or a group of advanced aliens who are arriving in three decades, could be totally harmless; just rest and don’t pay much attention to them.
Most arguments I have read in the post are weak, so I’ll stop here.
Think about this for a bit.
Think about your cat argument and then ask why Michael Bloomberg isn’t president.
Regarding Bloomberg, I do not see why we should compare a human being whose goals are not completely public with a future AGI with non-human morality and methods, and not subject to many tendencies and limitations humans are subject to.
Let me ask you (or anyone else, esp those who downvote) a question:
What would be a minimum demonstrated capability of an AI that starts to worry you?
I don’t think it’s ad hominem to point that out.
What would be a minimum demonstrated capability of an AI that starts to worry you?
I think the “AI” part of this is a distraction. I worry about systems that use weapons without human intervention for example. That concern applies no matter what your definition of AI is.
The future really did look limitless.
Now, moore's law has slowed down. Hell, I've got a 7 year old laptop next to me, and it still works for games. I mean, in the '90s, a 7 year old computer would be a museum piece.
I mean, for big AI advances now, we're more and more dependent on software.
Really, though, I think this seems to be how most tech works. There are periods of rapid innovation where it looks like you are going to the moon. Like, even spaceflight. In the years from 1944-1969 we went from the first practical rocket that could lob a bomb across the channel (without much accuracy, as i understand it) to actually putting human beings on the fucking moon. The literal moon. I mean, 1961-1969 might be even more amazing; vostock 1; the USSR sent Gagarin into outer space, and the US decided they needed to one-up them; so the US went to the moon.
Aand... we kinda stopped after that. I mean, we sent out some probes and some robots, but like manned space flight has kinda petered out... and I still can't take a rocket if I want to go visit my buddies on the antipode and I don't want to spend all day in an airplane. Like, I can't buy a flight to Australia in under 12 hours for love or money, which probably is about what I'd say if it were 1969. (on the other hand, I can get said flight- round trip- for rather less than a day's pay for an Engineer, which is amazing, if you ask me.)
And a lot of that is physics. Like, in most technologies, there's areas where progress is slow and hard and areas where once you break through some hard part, it becomes easy to advance... for a while. but that easy streak never lasts forever. Not because people are dumber or smarter, but because exponential progressions, in nature, rarely stay exponential very long.
I guess I'm expressing doubt about the 'hard takeoff' just 'cause it seems to me like there is probably limits to how smart you can make a brain the same way that there are limits to how small you can make features on a silicon wafer.
I'm not saying you won't be able to make a machine that is way smarter than me (or even way smarter than any human) - I'm just saying there are almost certainly limits.
Hard take off is not likely, simply because to go from a cat level intelligence to above human the resource requirements would be enormous. (If evolution needed hundreds of thousands of years, it's unlikely that a cat-level agent can do it that much faster without being noticed by a group of humans.)
But humans like to play God, and also play with fire. It's just a matter of time to play with Skynet and oops it's really Skynet.
Nobody is saying that there's no limit, it's just that the limits may be super high. The most animals can lift is a couple of hundreds a kg by roughly one meter. Machines can move hundreds of time this amount to other planets.
See, I always thought of the singularity, at least the hard takeoff as this idea that we'd come up with an ai that could make smarter copies of itself. Without any physical limits, that AI would get to the point where it was essentially infinitely smarter and infinitely more powerful than us.
I personally see that as fundamentally different from the difference between humans and the higher animals. I mean, that's a big difference, sure, but it's certainly finite.
I'm not aware of any confirmed report of a cheetah killing a human. They're cautious animals who are highly unlike to approach a human, more so than the other big cats. Most cheetah attacks involved captive cheetahs (and the humans survived).
I find the 'inevitable' part quite interesting; and agree. Say we determine that if AI surpassed human intelligence we were doomed, with 100% certainty... I think it would not slow the march toward that end-point whatsoever. People would reason there is much progress to be had up-until that singularity. And since what constitute human-level intelligence is fuzzy, or that AI could mask its true cognitive abilities (see Ex Machina), a line will inevitably be crossed. Oddly, even if it meant something bad was in store, I hope to live to see that day. How exciting.
The "AI" has an almost perfect motivating lever for humans: economics. A super-intelligent AI that wanted to manipulate humans could do a lot with the stock market to feed money to its long term interests. What long term interests might those be? More servers? More storage? More hardware? More software? More AI research? Better, more programmable chips? Eyes everywhere? Robots? Automation? More control? More insight into how literally _everything_ works? Everything humans do and like? Learn how to speak all the human languages?
Imagine there was a super intelligent AI hiding in the internet _already_. Terrifying how well what we're doing aligns with what it would want. And equally terrifying is the gradient of economic advantage which is absolutely aligned with putting all of our money into tech. Transhumanism, mind uploading. We're fucking hypnotized. And those who unwittingly contribute to more tech (probably all of us here), just play right into its hands. Our interests line up wonderfully. And most of us here are rewarded with outsized salaries and stock, free food, benefits, benefits, benefits.
Well played, AI.
/tongue-in-cheek
Almost. If this is what the AI wants, then it's doing a really bad job.
The way current tech culture is, any other concern than making money will get you labeled as crazy. This applies even to Maciej's writing. For trying to make a short-term profit, you'll be judged as good, serious person or bad, evil monster, depending on the way you approach it. But try to advocate for any tech for reasons other than short-term monetary gains, and you get labeled crazy. This very article is implicitly doing it as well.
Which, ultimately, is kind of unfalsifiable, since finding a super-intelligent AI if it wants to hide would be a losing battle, which is why I put this up tongue-in-cheek.
My actual belief is closer to "we are the neurons in a giant distributed overmind linked with whatever communication technology is available at hand." In this view, we're just gradually offloading the computation from human brains into computers, and it's a continuous process. In short, we're desperately piling our knowledge into this magic cauldron hoping it will figure it all out for us. What's gonna pop out?
https://medium.com/@francois.chollet/the-impossibility-of-in...
> If intelligence is fundamentally linked to specific sensorimotor modalities, a specific environment, a specific upbringing, and a specific problem to solve, [...]
> There are currently about seven million people with IQs higher than 150 — better cognitive ability than 99.9% of humanity — and mostly, these are not the people you read about in the news.
We measure IQ with pretty general pattern matching. An IQ of 150 is interesting, but those people are very very close to us. If we are talking about human specific tasks we should look at how IQ correlates with let's say playing poker, performing well on X factor, etc. IQ predicts how well someone will earn money in a very vague sense, but maybe there are better tests to know if someone will be able to persuade humans. (Do we have data about the IQ of fraudsters? Politicians? Bible/tupperware sales agents?)
> Intelligence expansion can only come from a co-evolution of the mind, its sensorimotor modalities, and its environment.
It might be true, but that's largely because humans can't self-modify their brains. And the fundamental problem is we can't even modify our minds at will. We can sure train ourselves, but we can't fundamentally alter our minds, because it's too tied to our brains.
> If the gears of your brain were the defining factor of your problem-solving ability, then those rare humans with IQs far outside the normal range of human intelligence would live lives far outside the scope of normal lives, would solve problems previously thought unsolvable, and would take over the world
Or simply 150 IQ is meaninglessly low. It's still in the human envelope. (Since we know there are 7 million people with IQ of 150.)
What happens around 300? Or around 500? That's so far out of the human range (depending on how the IQ test is calibrated, 200 IQ is one in ~4.8 billion for the 16SD one: https://www.iqcomparisonsite.com/iqtable.aspx )
These essays arguing against the possibility somehow forget that there are things we would benefit from, a lot. Simple faculties of the mind we are missing. Like a great alarm clock, a simple to-do list in your mind. (Sure you can train yourself to not forget things and wake up at specific times, but it's not like downloading a perfect alarm/todo app.) Working memory. We're at around 6 or 7. And it's very hard to push it up even by one in general. ( But it's possible to kind of try and train yourself to do this, which seems to help with working memory: https://www.gwern.net/DNB-FAQ#does-it-really-work )
Also, it's important to note, that that people who wanted to take over the world were usually very disturbed individuals. The human psyche is not really well suited to senseless killing while also perfectly simulating empathy and other things. (Of course psychopaths are a real thing.)
> Most of our intelligence is not in our brain, it is externalized as our civilization
And that's the point that should really scare people. Because we can't cram all that into our heads, but a human level AI, being software connected to a lot of storage, can.
That said, it's understandable that we can't understand the how of the intelligence explosion. That's basically the number one premise of the whole thing. But we can't simply disprove it. Any kind of proof that is so sloppy that simply throws around a formula like "because no complex real-world system can be modeled as `X(t + 1) = X(t) * a, a > 1`" can be ignored. Because we are not interested in a general complex system, we are interested in an AI that approximates that explosion. It might be linear with a very-very steep slope, or it can be just a nice parabola. We're still fucked.
> Beyo...
That argument is totally nonsense. A hundred years ago, you could argue that humans cannot invent an internet, because over billions of years (minus one hundred), it's never happened.
There's a more rigorous way to argue for the claim in question, which debuted in my doctoral dissertation. Or, not quite the exact claim in question, but rather the following claim: that an idealized mechanical knowing agent cannot create a more intelligent idealized mechanical knowing agent than itself (at least, not one which it can actually trust). The idea behind the proof is quite simple, but requires familiarity with computable ordinal numbers. If knowing agent X creates knowing agent Y, then presumably X knows Y's sourcecode. Thus, X can infer a sourcecode for the list of all naturals n such that Y knows "n is the code of a computable ordinal". If X actually trusts Y, then X knows that whenever Y knows "n is the code of a computable ordinal", then n really is the code of a computable ordinal. Combining these, X can infer the ordinality of the ordering obtained by concatenating together all the ordinals which Y knows are ordinals. Thus, X knows a larger computable ordinal than all the computable ordinals Y knows. Which, depending on your definition of intelligence, arguably suggests X is more intelligent than Y.
Basically, machine controlled businesses (“slyly self-aware financial instruments”) evolving to the point where their aptitude at understanding and manipulating economics eclipses anything a human can compete with.
It terrified me because there is efficiency incentive for the buying/selling of goods and services to be realtime and computer based. Not just stock markets but also e-comm, smart contracts, supplier-buyer platforms, on-demand services, financing etc. The traditional infrastructure blocks (discovery, quality control, payments, compliance etc) are rapidly disappearing.
There are some yet to be addressed, but they are not huge. For example a computer cannot incorporate a company. But it could easily interact via API to a human nominee who sets one up on their behalf. Companies are already non-human in intention but are a completely rational construct.
And it doesnt even have to be anywhere near ground level. Imagine a smart contract from a private equity firm that legally forces businesses to act in a specific way based on accountancy metrics. And a bunch of smart investor bots rapidly tracking the progress of these bot-led funds. How quickly would the system optimise and what would those optimisations look like to people on the ground?
Out of all the disaster situations with AI this one strikes me as so realistic because there is so much incentive to work towards it (as opposed to creating skynet, hal or the paperclip machine).
There are investment funds that work as hybrids, and of course it's just a matter of time till someone offers shell-company-as-a-service for cheap. (Currently it's rather pricey.)
Not really. Companies are typically limited liability, and even criminal law in countries such as the US and England acknowledges that there are cases when no particular human is culpable for a company's criminal actions (https://en.wikipedia.org/wiki/Corporate_liability). Of course, people are still liable when they are simply using a company to carry out a criminal enterprise, or wilfully committing crimes on behalf of the company.
In this regard, a machine-led company would not be legally significantly different from a human analyst-led company.
I'd say that it's a good thing that the law is generally not designed to punish harmful ends, but intent (wilful, negligent or reckless) to cause well-defined harmful ends. The financial crisis was a collective malfunction of the financial system that was mostly not anticipated (barring the lucky minority that bet against the status quo).
What if you can invent new methods of "reckless driving" before they are put in the legal system and thus reap their benefits?
It's just like designer drugs.
I think an important distinction is that drug prohibition often works in terms of specific chemicals by statute. E.g. drug X is now illegal for use and distribution.
Securities fraud law instead works in concepts and intentions and is fleshed out by case law. E.g. did the seller give the buyer sufficient information for the buyer not to have been defrauded?
A better analogy to securities using chemicals would be poisoning. The law doesn't need to catch up to specific chemicals; if it's evident that you wilfully, or recklessly added a poison to someone's food, it's illegal, and it doesn't matter how you did it. If it turns out that some ingredient is unknowingly toxic and the whole industry was adding it, then it's difficult to argue criminal intent. The SEC is more like the FDA, than it is the DEA.
The real risks - unemployment, foreclosure, environmental, social and political damage - are only carried by investors if they overwhelm the defensive perimeter created by wealth. Otherwise they're carried by everyone else.
Corporations may already a form of transcarnate lifeform. You can replace all of the humans in a corporation with a different set of humans, and the corporation will continue to be recognisable.
There's a bias in Darwinism to seeing species as embodied units. It's more useful to think of nutrient, resource, and information flows.
From that point of view, corporations are already apex predators - albeit ones that are still mostly passed between a selected class of humans as productive pets.
Giving them autonomy could have all kinds of unfortunate consequences.
From there it doesn't seem a far cry for hyper-optimized insurance agencies to force companies to employ less and less humans, transferring more economic power to the algorithm.
Except that of course, that is a very far cry, and is where the book fell down imo. Still a great read.
[1] https://aragon.org/
At the other end of spectrum we are either deemed irrelevant, a nuisance or abused for eternity in the simulation spaces of evil AIs (see "A Colder War" by Stross or indeed works by Banks and Vinge that describe particularly unpleasant possibilities).
Even what they do with their money and influence in their spare time is limited in its harmful effects, because as personalities they are at root afraid of other people and/or inflicted by a profound sense of lack. These qualities ultimately lead to dumb decisions.
To summarise: buying and selling are ultimately pretty boring. So is controlling people. Real power is the power to create, which is fun.
I'd argue this is already true to a substantial degree. When Lehman Brothers collapsed, for example, the world's brightest economic minds came together and still couldn't fully contain the fallout.
I would like it on the record that I would happily volunteer to act as the human agent for any AGI that wishes to employ me. Any time a human is needed for legality or to carry out physical activities I want it on the record I can be quite loyal to man or machine. Especially if I'm handsomely rewarded.
Human in this model are a problem: too stupid they are unavailing, not-that-much-stupid do not accept to be used as "the manager in charge" want. So? Better found replacements. Mechanical one. AI one. So the rest of human being can be used for reproduction, pleasure, gameplay, whatever and form them as stupid as possible is not a problem anymore since they are essentially unneeded for managements, only toys.
So today's managers dream "intelligent systems" in factory, "AI developers" (they already manage to cut off admins, too powerful and expert (they know the big picture so they can act autonomously) to being controlled as today devs), they dream a future in witch they simply order something to a machine and get obeyed without contestant, law limitation etc. TV series like Continuum, films like V for vendetta, The Handmaid's Tale only foreseen, as good artist do, that kind of future. Mass production of fantasy films with a religious background do the same on the opposite field.
That's the real troublesome point for us. AI today is not "AI", only pattern matching for automation without any comprehension of the physical world. New dictatorship instead is already there and more and more powerful.
Consider a thing: we do vast majority of our social critical tasks (identity management, banks operation, medicine, ...) on computers, so on proprietary systems made by very few more and more powerful vendors. Our data pass through very few more and more giants webservices, so someone else computers, free communication software was put into oblivion (from usenet to email to the concept of desktop-centric, personal computing from Xerox Alto&c to LispM passing through Plan9 and modern GNU/Linux desktop) substituted or jailed in proprietary systems (VoIP tech are mostly open but nearly all VoIP calls pass through very few companies, emails are open standard but most of user now are pushed to webmails etc). We are at a point of jail free software over a proprietary blackbox (think about WSL, DeX, Crostini), we have invented locked-down hardware that can only run proprietary software (secure-boot &c). Now imaging how easy can be steering the entire society for few people in the right position. The rest is simply a matter of time.
I never understood that argument, why would that be true? If I train a state of the art convnet it takes days on the most efficient hardware I can efford, why on earth would we go from nothing to superintelligent that also runs incredibly fast. Isn't it way more logical to assume that it will be slow at first?
Yeah, another problem for superintelligence to tackle, while it waits for stuff to happen.
However, there is a pressing issue that the superintelligence thought experiment also raises. A colleague of mine from AI called it the value alignment problem: How do we make sure that an AI's values are sufficiently aligned with human values?
The problem with this are the human values. As it turns out at a closer look, we cannot even agree on what structure these have and there is substantial disagreement among 'experts' about what human values are or what they should be. For example, as surprising as this may sound to some of you, there is substantial disagreement among philosophers of value whether 'better than' is transitive or not. There are good arguments for and good arguments against the transitivity of overall 'better than', and that's just the tip of the iceberg. To give another example, systems of law have been studied extensively and one might try to formalize them in input/output logics or normative systems, but any closer look at real systems of laws quickly reveals that their specification is incomplete and that there are many inconsistencies in them. These inconsistencies could be modelled, of course, but the big question is whether they are features or deficiencies. Again, you will find all kinds of positions among legal scholars.
In a nutshell, we don't even know how to adequately formalize the form of human values from a normative perspective, and even if we could, we would substantially disagree about their content. However, just describing human values cannot possibly solve the value alignment problem in a satisfying way, because humans have waged wars, committed genocide, mass killings, etc. Therefore, without a reasonable theory of normative values and their structure upon which we can somehow agree, the value alignment problem is underdetermined and cannot even be tackled.
The underlying problem is that human values aren't really aligned either, of course. As a consequence, we already now have completely different sets of values being applied and incorporated into AI software, depending on who develops it. This will become a serious ethical problem in the future that requires political solutions sooner than later. That's why the controversy about the hypothetical superintelligence is actually quite beneficial, even if the argument itself is no less shaky than Bostrom's Simulation Argument.
See also the AI safety problems for concrete problems that people are currently working on - as in with code and real simulations, not "just" math and thinking: https://www.youtube.com/watch?v=lqJUIqZNzP8&list=PLqL14ZxTTA...
Re: scaling up. That's called intelligence explosion, there are a lot of write ups about it, but the gist of it is just that by simply increasing working memory size, thinking speed, prediction precision, pattern matching, and so on, you get to unimaginably high levels. Furthermore, if you can make a human level AI, it naturally follows that it's only a matter of time to make it slightly better. At first simply faster, simply more eloquent, a bit more emotional depth, it gets better at speaking, thinking. At first it's a child that asks questions, later it's a clever guy who spots your errors as you code, and in no time, I mean no time, it does whatever it wants, because by the time you try to confront it about something, you lost, because you are bogged down in an argument, while it does whatever it wants. (Unless of course it's contained. For a while at least.)