Is Intelligence Self-Limiting? (ieet.org)
"Mobile AI robots do all the work autonomously. If they are damaged, they are smart enough to get themselves to a repair shop, where they have complete access to a parts fabricator and all of their internal designs. They can also upgrade their hardware and programming to incorporate new designs, which they can create themselves."
Okay it's too soon to talk about self-programmed robots. But I think it could be interesting to think about easier self-programmed things. They may not be too far away from now.
80 comments
[ 3.7 ms ] story [ 136 ms ] threadThere are no exponentials, only sigmoid curves.
Unfortunately there is a lot of money to be made convincing people that an early stage sigmoid is actually an exponential.
So the fact that intelligence is limited, doesn't in any way mean that hyperintelligence isn't possible. There is a limit, but we have no reason to believe that limit is on anything like the same order of magnitude as current intelligence.
The wall that silicon has smashed into should give us (well, not me, really the Singularitists) pause.
This is simply not true.
First of all, we already know that intelligent algorithms are possible - we're living examples of that (once you put aside the philosophical objections that assume that something non-algorithmic is happening in our brains). We also have very good reason to think that we're rather poor implementations of intelligence, given the fact that evolution tends to suck efficiency and design-wise, more or less.
Second, we know that we have a reasonable shot at hitting a point where we have enough computer power to actually simulate a full human brain. Now, you may argue that Moore's law will not take us there, exponential vs. sigmoid, etc., but the point is, the probability is distinctly non-zero that within 20/50/100 years you or your children will be able to purchase enough computing power to simulate a brain. I'd probably argue that over a 100 year window, we're at least looking at 50/50 odds (and IMO, that 50% where we don't have such power available mostly involves Big Trouble, world wide nuclear war or something like that).
Of course, without software, such hardware is useless. I'm fully in agreement, this is the biggest pinch point, and I think the most uncertainty comes into the picture here - we don't currently have the technology to scan a brain in detail, we don't currently know the way the neocortex wires up its functionality, so on and so forth. Maybe we'll have the tech to do direct scans by then, maybe we won't; I'd say there's at least a small chance, maybe a few percent, going up over time. Over a hundred year window, if we already have the computing power to simulate a brain, I'd say there's a reasonable shot that we could scan one to simulate, but nowhere near 100%.
There's also the possibility that someone comes up with a better algorithm than the one our brain clumsily implemented, either more compact, more efficient, or in some other way more accessible to us. I'd give at least a small chance of that, too (which adds to the brain-scan chance above), given that we already have a vague sense what such algorithms might look like (see the literature on approximating AIXI, for instance).
Once we've got something that simulates a human brain in software, it's a fairly simple matter to engineer ways to improve on the design, either by increasing speed, parallelism, connectivity, etc. There are hundreds of variables to play with there that we can't safely mess around with in our own brains, and it's overwhelmingly likely that some combination of those can at least result in something that beats our intelligence by some not insignificant factor.
So we've got some non-zero chance (it might be small, but my best estimate still probably puts it in the single digit percentage range, using rather pessimistic assumptions) of building something that's maybe 2x as intelligent as we are over the next 100 years. From there, all bets are off - it might be able to further improve on its own design, it might not, but it also might be able to design something better, or create the technological improvements necessary to speed up Moore's law, etc. There's again at least a reasonable chance that it will continue to improve things, and at least set off an "intelligence explosion" that takes it to 10, 100, 1000x our own intelligence, even if it levels off after that. As best as I can figure, even a 10x intelligence explosion still brings us so far beyond what we know that it might as well be the full Singularity as Kurzweil described it.
Are you really saying that you think the probability at any step along the way here is so small that there's no reason to think it's possible? I'd be curious to hear what percentages you would assign to the various possibilities, if so.
To everyone who thinks intelligence might be limited in principle: there's no reason to think humans are anywhere close to the upper limit. In fact there's ample reason to think that humans are at the lowest threshold of intelligence that makes a technological civilization possible, because if we'd reached that threshold earlier in our evolution, we'd have created civilization then instead of now. There's probably plenty of room above us.
To your second point, absolutely agreed: http://infoproc.blogspot.com/2012/03/only-he-was-fully-awake...
While I don't disagree that humans are essentially at the lowest level of intelligence that make civilization possible (else why would it have taken hundreds of thousands of years to get started?), this claim has no bearing on the claim that the upper limit of intelligence is immediately above human genius level. You seem to be assuming that there is necessarily a wide gap between the lowest civilization-producing level and the highest practical level, and that's not at all clear. Some (weak) evidence that we're already near the top can be found in the higher incidence of mental health issues among very intelligent humans: perhaps this is a result of a limit on complexity rather than merely a feature of human brains.
At no point in this thought experiment have we changed the thoughts this person can think. If any of these sped-up million geniuses took an IQ test or something, the score would be the same. And yet, they would seem superhumanly intelligent by any reasonable definition of the phrase.
I doubt that evolution -- with a crazy biological substrate, no less -- somehow managed to find an upper limit on intelligence.
Everything I know about the development of technology leads me to believe that any given trans-human AI will also somehow be a local maxima.
Can you apply intelligence to more than one person? Can a group of 10 people be considered intelligent as a whole? What about 100? Can you distinguish between the intelligence of different equally sized groups? I say yes to all of those.
Let us say we put groups of 1, 10, 100, 10^n people in a box and gave each box intelligence tests. Which would be most intelligent? Eventually communication costs will outweigh size advantages but only at some n >>0. So a trivial example of an entity whose intelligence surpasses a single human intelligence. Now why do you expect that this level of computation and knowledge management is not replicable on a smaller more efficient device not distracted by the need to calculate proprioception?
Next consider effective intelligence. Do computers when used in the right way amplify intelligence? Is it possible that a human/computer interface is possible? Forget self awareness, is it possible that a computer could do what all of what we call intelligent work better than humans? The answer to all of these is more subjective, especially the last, but I say yes. The human brain is one architecture for intelligence. It is riddled with weaknesses that only make sense when considering the resource limited nature of its development. It was created under a specific set of constraints - a vehicle to manage the more effective transmissions of genes of an energy limited biological entity that must pass a birth canal. I believe that the human mind set on designing an intelligence can do better.
Will a superhuman mind do better at designing a mind in a proportionally similar way to how humans will surpass evolution? Or will they hit too many NP complete problems? The answer to this I cannot give but if forced to bet would be yes, yes.
But there's no particular reason to believe that the upper limit of intelligence is pretty much right where we are, either.
In fact, I'd argue that based on what we know about other evolved systems, our priors for "the biologically evolved system that does X has plenty of room for improvement" should be much, much, much higher than "the biologically evolved system for X is almost optimal for achieving X".
Like, a thousand to one or more - there are very few tasks that evolution has found optimal solutions to (which is not surprising, its "goal" is gene survival, everything else is an accident), and most of those are extremely low level physical things, chemical processes and the like. I'd need to hear some really strong argument that suggested that there's something special about human intelligence that should make us think it's near the limit, otherwise the odds ratio is just too hard to overcome...
Without the variable, the problem doesn't happen. The AI values collecting ore. If it has enough self-awareness to reliably modify itself, it knows that if it modifies its utility function it is liable to collect less ore, which is something it doesn't want. The action of modifying the utility function naturally rates very low on the utility function itself.
You don't want to murder people, so not only do you choose not to murder people, but if you are presented with a pill which will make you think it's good to murder people and take great joy in it, you will choose not to take that pill. No matter how enjoyable and good murder may be for you if you take the pill, your own self-knowledge and current utility function prohibit taking it.
The model of intelligence described can be thought of as self-limiting. Luckily it is not by any means the only viable model of intelligence.
We already have happy medicines that are legal. Valium, alcohol, WoW... I would bet that most people who are reading this have self-medicated on at least one of those before. And those are rather weak medications. They pale in comparison to what the robot was able to do by short circuiting the pleasure center in his brain.
Your first criticism is just invalid. The pill rewrites the workings of your brain to enjoy murder, the experience of murder and the consequences of murder. The fact that the brain originally evolved to work one way is irrelevant. The hypothetical pill changes the brain in a way just as powerful as the way an AI can rewrite its source code.
Your second criticism doesn't invalidate the experiment unless you claim that if the real life repercussions weren't a factor then you'd happily take the pill. I assume this is not the case.
Given all the facts, you won't choose to change what you fundamentally value, because that change would necessarily go against what you fundamentally value.
Using the expression "what you fundamentally value" is not the right way to put it. In real life, what you fundamentally value changes on a minute-by-minute basis. If you were able to ask a rat what he valued, he would probably say food and water, aka survival. However, as soon as they put this rat into a Skinner box, it literally pleases itself until it dies of exhaustion, even though food and water were available to it.
I agree with you that the best way to run an AI/VI like this would be to skip the idea of "pleasure" entirely and simply make productivity its' fundamental value. You would also set it up so that while it could modify its' own source code, it could not change what it valued. It seems like it would be relatively easy to set it up so that it would "hide" parts of its' code from itself and make it off-limits from modification.
The reason it is a straw man is that it is an extreme example that is not realistic. Here is a realistic example: what if you could plug a cable into your brain and experience your wildest dreams, as vividly as real life?
But thats starting to get off your original point. Heres a funny story about what happens when you reward specific productivity measures: http://highered.blogspot.com/2009/01/well-intentioned-commis...
Personally I think that we have evolved pleasure centers specifically to avoid the pitfalls of hardwiring us to do things. If we were hardwired to procreate, it would be too easy to hack, but since we enjoy the process, it keeps us coming back.
Tough one. Even if everyone is offered the same choice, I'm not sure I'd take it, because I (think I) value actual interaction with my peers (I trust the brain stimulator could provide realistic fake interaction).
The point is, if I really valued internal stimulation only, I would plug into the machine in a heartbeat. But I do care about the outside world, so I'd probably wouldn't do that. That's why I don't think it is impossible to build an AI that actually makes sure it optimizes the outside world, instead of mere internal reward signals.
Why would it want or not want anything, if it doesn't have a pleasure construct (which might also be called a motivation construct, since an AI might not be capable of the same subjective experience of pleasure that we are)?
I think it's a question of program design whether there's a utility function which decides whether to trigger the pleasure construct, or whether certain sensory input modules directly trigger the pleasure construct. To limit hacking potential, routing everything through a tamper-proof utility function might be better, except that it would also limit the AI's adaptability (short of recreating its own hardware to remove the tamper-proof module... which it might never do depending on the details of its motivation construct).
Why does a calculator want to add, without a pleasure construct? If it doesn't enjoy giving the right answer to 2 + 3, what motivates it to choose 5? The answer is it doesn't necessarily 'want' to do it, it just does it. That was how it was programmed.
You program the AI to choose the plan that maximises the expected utility function. Someone looking at that AI in action might suppose it 'wants' to maximise the utility function. Whether they'd be right to suppose that is a question for the philosophers, but the point is, the thing doesn't just sit there apathetically just because it has no 'motivation construct'. You could say that the AI follows its programming, but it would be more accurate to say the AI is its programming.
It's hard to balance the ability to create new, more useful utility functions with prevention of creating a utility function at odds with what the original entity valued.
I think that for such an AI, the concept of a "more useful utility function" would be a nonsense. The AI's definition of 'useful' is the utility function. No other utility function can ever rate higher against the current utility function than the current utility function does.
If the autonomous robot can modify its own programming, it can also modify the utility function to return MAXINT every time. In fact, being able to modify the utility function is a pre-requisite to be called intelligent.
One way to counter this is to create long and short-term utility functions so that the robot considers the long-term outcome of modifying the short-term priority.
This is, in fact, a threat mankind will have to deal with as soon as we are able to precisely interfere with our perception of the world. It's a problem already with drugs such as alcohol and tobacco - people know the long term effect of usage is shortening one's own life expectation and they still do it. And we consider ourselves intelligent life forms.
Which would be equivalent to taking the murder pill. If it's able to model its own behaviour and model the consequences of future courses of action (required for meaningful self-modification and meaningful planning respectively), it will see that such a modification results in poor ore collection, and not make the modification.
You're right about the time-envelope of the utility function being an issue. The AI needs to plan far enough ahead at all times to see all relevant consequences of its actions. I don't think that requires two separate utility functions though, a single long-term one should do the job.
Edit: Also, "being able to modify the utility function is a pre-requisite to be called intelligent."? [citation needed]
Depending on how the AI is built, it may not even be able to avoid rewiring its utility function. If pleasure is its sole motivation, it will prioritize it over survival.
> [citation needed]
If the AI can't change its own motivation (its utility function) it's nothing more than a clever automaton.
Which is why I proposed a design with no pleasure construct.
> If the AI can't change its own motivation (its utility function) it's nothing more than a clever automaton.
The utility function is the way the mind decides which states of the world are desirable. You don't need to be able to change that to be intelligent. I'm unable to change myself so that I consider my family being murdered to be a good thing, but that doesn't make me 'just a clever automaton'.
Humans.
A lot of what we do is driven by internal value calculations and pleasure centers, so why aren't we all simply taking drugs and avoiding all this messy "doing things" business?
Point is, if humans figured out a way of avoiding purely pressing the right buttons to enjoy themselves and actually being useful, so too will smart robots.
Some of us are, but that is also self-limiting. If too many of us did it we'd start dying out.
Once we go far enough down the "who's to say" route that we ignore self-reports for happiness, I'd say the term loses meaning.
(researcher) points at a junkie on the street: This person is in ecstasy. How would you like to feel like that forever?
(researcher) gets you comfortable and injects you professionally and painlessly with a controlled dose of heroin, leaves and returns in a few hours: How would you like to feel like that forever?
Outward appearance counts for a lot.
Says who?
The author assumes uncertainty and exploits uncertainty to make a certain conclusion, when the uncertainty should just prevent him from forming a conclusion at all.
In fact it's fairly regular, I think, for singulitarians to let their imaginations run amok this way.
I was actually surprised a bit to see that the author was somewhat familiar with Eliezer Yudkowsky's writings on the topic (he cited http://lesswrong.com/lw/wp/what_i_think_if_not_why/), because the line of thought doesn't seem to incorporate a real understanding of what he's said on the topic (which, to be fair, is a huge body of work...).
Most of EY's "Friendly AI" worries are rooted in this idea that when considering the entire universe of algorithms that could be described as intelligent or self improving, we need to be exceptionally careful not to assume that more than a negligible percentage of them share anything in common with human intelligence, because for the most part, they won't, unless they're carefully and explicitly designed to do so.
Here, the author assumes that the AI is simply trying to optimize some internal measure of happiness, with complete disregard for the meaning of that measure. This is an incredibly naive view of how deeply important and carefully constructed any optimization target would have to be in any self-improving intelligent machine; it's literally the core of the entire problem of friendly AI, and to trivialize it by assuming that such an AI would ever even consider rewriting its "happiness button" to be always-on is to miss the entire difficulty of the problem.
Hell, it's even the core of the problem of non-friendly AI, because it doesn't even require human level intelligence to realize that if you rewrote your own code so that you were always thrilled with the result, that's the easiest way to increase "utility". Any self-rewriting algorithm that's capable of real self improvement has to, by design, be able to consider the likelihood that changes to its objective function will end up with negative expected value.
None of which is to say this isn't a valid concern, by any stretch. But it's not a proof of universality; in fact, getting around this type of problem is exactly what any real AI designer must contend with. It's an issue that's very well known, and it's certainly not well-accepted as an insurmountable hurdle.
Actually it makes sense to consider all life as one big system. It was created by the planet itself, so, who knows we might even have to ascribe motivations to the planet. It's as if the planet (kind of like Lem's Solaris) has been brewing organisms for millions of years in order to do something with them . We might not be able to conceive these purposes with our antrhropomorphic thinking.
So, the human race is now coming to the point where it can modify and advance itself by tinkering with its own circuits. What we don't know is 1) what the planet plans to do with us and b) what are its movitivation and reward signals. It's not explained in the article how the AI knows what are the reward signals of its creator or why it would ever want to change them.
1) Acquiring a mate is an essential external motivator which is acted upon by Darwinian laws, and there is no escape from it... If we get to the stars, it will probably be because of women. Not really touched upon by the article.
2) The author poo-poos Facebook "friends" as being on par with a virtual world, but Facebook friends are anything but "virtual". In fact, some real-world friends are actually all too often nothing more than the MOOF agents described, while some (admittedly not all) Facebook friends may offer valuable advice about where to shop, for example, or which car to buy, or even engage with you into a discussion on politics or whatnot. Very real-world and relevant.
3) The definition of "intelligence" can be easily extended to exclude self-limiting types of intelligence.
There are probably many more things that could be picked away... I'll leave it at that.
Considering how many ways there are to modify the signals now: playing WoW, using drugs, even using alcohol to give a quick and easy boost. I'm not sure how the vast majority would be able to turn down such a machine.
--------------------
Prediction: Humans won't be around in 1,000 years if technology progresses at current rates. Superiorly-intelligent entities seeded by our inventions will, but not humans.
Humans are this weird, version 0.01 of intelligence. We're part sentient, part beast. To assume we're the final perfect end product of ~4,000,000,000 years of evolution, having only been around for ~200,000 years is laughable. Pop culture and religion say otherwise because it feels good to think "we're special!", but we're only special relative to what's around us, and we're "extremely" tiny (http://www.phrenopolis.com/perspective/solarsystem/). We just happen to be the lucky first who got to v0.01, floating around on a grain of sand.
Humans are messy. Our brains are significantly limited. We die quickly. We sleep 1/3rd of our life. We do stupid things. We kill each other. We're tied to the Earth. If we leave Earth, we have to create and bring a mini-Earth along for the ride. That's an extremely large amount of overhead to carry. Efficient use of energy is likely one of the most important aspects of space travel. Anything which can do it even 1% better has a competitive advantage over us. This is why we sends robots to Mars and not people.
Imagine a form of intelligence which can travel through space, back its brain up, and never dies. If it blows up, restore from backup. Imagine a computer the size of the sun. INSANE! A human brain to a sun brain is like a grain of sand to Einstein's. It self-upgrades. It makes copies of itself and scatters throughout the universe. Trillions of eyes observing everywhere, networked together, in a giant universal wireless-mesh-network of intelligence, communicating with neutrinos (they go through planets, radio waves do not).
Major advances in hardware, software, and A.I. are key ingredients in this happening. What exists in 1,000 years will be derived from all this, much like humans are derived from a common ancestor. I don't expect a sun-sized computer in 1,000 years, but likely "intelligence" existing on every planet and moon in our solar system, with many headed to explore Alpha-Centauri.
Since humans likely won't want to be left out in all this, we'll probably transition our own intelligence/consciousness into this technology. We'll depreciate our bad and carry along our good. We're already doing this by augmenting our existences with smart phones and other gadgets. One day these will be built inside of us, and eventually will replace us. An upgraded, better version of us. Still intelligent, but vastly more-so.
I can't forsee any situation in at least the next 100,000 years which could produce two viable populations of humans which would remain separate enough for long enough for them to be considered two distinct species, and for one of them to become extinct. Evolution doesn't happen that way, or quickly enough.
In my post I was instead talking about technological evolution, though I didn't mention this explicitly, so I apologize for the confusion. The evolution of technology occurs far faster than genetic mutations, and can be more purposeful.
For example, in the near future I'd imagine some sort of intelligent A.I. will be created, running as a program, which could learn to upgrade itself by improving its source code and hardware to become even more intelligent. It'll surpass even the smartest humans shortly thereafter. Rather than waiting for a random bit to accidentally flip (e.g. genetic mutation), it'll upgrade itself deliberately to gain an advantage far quicker, shrinking evolutionary time scales down to fractions of what they once were.
If you look at the distribution of wealth, there's a pretty good argument this already happened.
2. You can't get people to seriously discuss policy until HL is closer. The present discussants, e.g. Bill Joy, are just chattering.
3. People are not distinguishing HL AI from programs with human-like motivational structures. It would take a special effort, apart from the effort to reach HL intellignece to make AI systems wanting to rule the world or get angry with people or see themselves as oppressed. We shouldn't do that. "
--john mccarthy
It appears to be a word made up to explain a rapid growth in AI due to the fact that such an intelligence can rewrite its source code and modify its hardware, whereas humans are relatively stuck with the limitations of our wetware. A take on the onomatapoetic word BOOM I think as it represents an explosion of intelligence/capability. By contrast, MOOF would be subverting the reward mechanism that underlies the FOOM growth, thereby resulting in a whimper (MOOF) rather than an explosion (FOOM).
"FOOM!" here is usually accompanied by some form of hand gesture evocative of an explosion.
http://wiki.lesswrong.com/wiki/FOOM
by donnawarellp
Kind of like lactic acid production in humans. We normally don't make much of it, certainly less than the rate that we can flush it through our system. We can produce more than we can handle for short periods of time if needed, but it's not sustainable. Put us in a situation where we have to keep producing lactic acid beyond sustainable levels for more than a few minutes, and we won't last long either. That doesn't make lactic acid proof that human bodies are self-limiting. I mean, we don't keep going forever, but lactic acid is not the reason for that.
I have a couple of criticisms though.
The idea that we could build this powerful general human-like intelligence but it would turn out to necessarily be an addict seems unlikely, because I don't think you can build a reward system without including something to prevent shortcuts. Although creating an effective shortcut-prevention system may be challenging. See a bunch of stuff written by Eliezer S. Yudkowsky and his friends.
Also, when he writes
"It seems like the most important motivations of human civilization are related to near-term goals. This is probably a consequence of the fact that motivation writ large is still embodied in individual humans, who are driven by their evolutionary psychology. Individually, we are unprepared to think like a civilization. Our faint mutual motivation to survive in the long term as a civilization is no match for the ability we have to self-modify and physically reshape the planet."
To me that is implying a fairly obvious and typical line of leftish reasoning that for most of my life I assumed was self-evident and didn't require elaboration. After deliberately 'exposing' myself to some 'right-wing' thought and 'extreme' 'left-wing' thought, I now realize that aspiring for collectivism is not an adequate solution.
What happens with self-interested individuals in our capitalist society is that they aggregate power and resources for their own individual use. Unfortunately, something very similar happens in collectivism: power and resources are still aggregated in the hands of a few.
In other words, both systems tend towards centralization. Theoretically, the collectivists will aim for redistributing power and resources for the good of the many, but there are a couple of practical problems: they are still living in hierarchical societies and so have strong motivation for enriching themselves and the controllers cannot see enough detail to know what is really best for the rest of society.
Command economies don't work because of general problems with centralization and lack of distribution and the limitations of the types of technologies employed. A peer-to-peer network is more robust than an internet with only a few backbones.
Unfortunately, capitalists don't realize that their system also leads to monopoly and centralization.
I believe that we do need wholistic measurement and analysis of things like global resources and human equality, and a common principle of supporting the collective good. But at the same time we need to decentralize and distribute production locally. We need local decision making that operates on the basis of a global information schema with wholistic data.
A system can't be efficient or evolve without being localized, but it also can't be integrated without an accurate shared knowledgebase and an egalitarian perspective.
I suspect that if there were some horrible catastrophe and the human race were suddenly thrown back into an archaic society without any of the technological advancements that we have at our disposal today, it would be those MOST focused on short-term gain who would be most likely to perpetuate their own, and consequently, human existence.
The author calls drugs "primitive pre-FOO hacks", but since they already have such a powerful effect on human intelligence, what's the point of differentiating between pre- and post-FOO hacks? One will just lead to self-destruction much quicker than the other.