On a more self-critical note - I think hackers love to read about AI on hacker news more than they would spend time actually solving the problem ! ( this is why these types of articles written by non-professional journalists are being up voted at the top of hacker news )
Great apes in the wild are generally having a hard time getting along, with humans crowding them out of a lot of their historical territory. I would hope our species doesn't end up like that. Likewise, great apes in zoos definitely have all their important decisions made for them by humans. I don't think I want this, either.
The straightforward answer to the FAI question for me has always been thus: we need to enhance our own intelligence to be always equal to or greater than the AIs we create.
I believe that a prerequisite for intelligence is the ability to understand human language, including the ability to sort intent from the literal meaning. Thus all arguments of AIs being like genies who do what you say, not what you intend, seems like bs.
Similarly, Nick Bostrom argues an AI would have to make survival something like hidden agenda since survival would help it do all sorts of other things. But lots of humans are quite happy to sacrifice their lives for various sensible and senseless causes, more so than less intelligent beings as far as I can see. So the ability of AIs to willingly die or go into oblivion for us is probably there.
We don't yet know all the qualities of an general AI system and we can't create one either - we don't understand human-level intelligence, which to say we don't understand ourselves. If an AI system could appear at random, it might be a problem but our continuous failure over many years to intentionally build one seems to indicate that a system of this sort will only appear through a rather detailed understanding of its architecture and qualities. At that point, it seems fairly certain the group creating the entity will be able to create it so that it wishes to aid and obey them, to the extent they specify (and probably will create it in such a fashion but you never know).
My argument that the AI will likely just follow the will of its creators, as many high intelligent human servants have followed the wills of their masters and employers and so-forth, sidesteps the "will it be moral?" question. The problem is that humans haven't been particularly moral in their process of creating more and more mechanisms to magnify their intentions. There's no indication that the creation of an AI would change this and so the advent of an AI might be disastrous in many ways. But disastrous through human intent rather than a thing escaping human intent.
Weird this danger gets much less attention. Or not.
Yes, of course we will. The use of neural networks and reinforcement learning just means the field is very young and it hasn't been worked out quite yet how to specify precisely what we want the agent to do.
Why not? It isn't hard to build machines that are stronger than we are or faster than we are. Even if there is some sort of strange mirror version of Turing's thesis that prevents us from creating AIs more intelligent than we are, inventing an AI that is equally intelligent but faster would accomplish much the same thing.
Also, bootstrapping is highly effective in countless other contexts within computer science, what do you think will prevent it from being applicable to this problem?
So far every bit of intelligence we have made is all about brute force, none of it is elegant. It works great - definitely. And I grant that given enough speed, and search depth such a machine may come up with ideas.
And maybe that's what human level AI will end up being: Just search every possibility till you find the one that works. For everything, including speaking with a person.
But it just doesn't have that spark of true intelligence.
So I'll qualify my earlier statement, brute force AI might happen. But AI based on elegant intelligence will not.
Is there a difference? I think so. For some things it won't matter, but for others it will. Somehow human level intelligence manages to cut through immense search spaces and find the answer (like the game Go) - how does it do that? No one knows. But there is clearly something different about it compared to brute force intelligence.
Why don't you think an "AI based on elegant intelligence" (according to your own definition) will happen? Do you suppose it's merely a problem of engineering and complexity, i.e. the human race is likely to become extinct before we will figure out how the human brain works? Or do you suppose there is some more fundamental reason why only humans (and perhaps other intelligent organic beings) can exhibit this type of intelligence?
Both (separately though, the two answers are not related even though I put them in one post - I'm allowed to have more than one thought about something).
We just don't understand intelligence (not the brain, intelligence) well enough to duplicate it, never mind improve it.
And I don't think we ever will because to understand something complex you have to be more complex than that thing, but by definition we can not be more complex than ourself - so understanding our own intelligence will always be beyond us.
That's the objective part.
The subjective part is that I personally believe that intelligence is not physical. The brain mediates it, but the source is elsewhere.
The soul maybe, or that we are really game characters in an MMOG.
The source of all real insight is, of course, divinity. True understanding is not generated by the brain but rather received in the Plexus Solaris and then works it's way up to the brain, where it is turned into action, language or memory.
I don't see any clear reason to believe your first claim. Humans can and do create things that are far too complex for any one human to be able to conceive of or comprehend. Most large engineering projects are good examples of this, but also even larger systems like governments and markets. You can certainly argue that even these large systems are less complex than human intelligence, but even if a human can only understand things 1% as complex as his own intelligence, I see no compelling reason to believe that thousands of humans can't develop something more complex than one human's intelligence.
As for your second claim, well, I suppose it is subjective, though I would prefer the term "speculative."
Where does the human brain fall on that spectrum? It might take us a few more decades to figure out the details of how it works, but I doubt we'll find elegance so much as a ton of hyperparallel brute force plus a few dirty tricks that evolution kludged together to prevent the whole thing from falling apart (most of the time, anyway).
> But there is clearly something different about [elegant intelligence] compared to brute force intelligence.
Could you elaborate on what you mean by this distinction and how the human brain does not constitute "brute force"?
Humans can play it - it's impossible for a human to brute force it, there is not enough computation available in the brain to brute force it. (Your "hyperparallel brute force" idea.)
Yet humans can play it anyway. There is something else there that makes it possible.
And the human brain self trains! That's the really amazing part - no one has to think of some clever optimization or algorithm to make it work like you would if you wanted a computer to play "elegantly".
You use so many words, yet none of them with precise meaning.
> none of it is elegant
What is "elegant" to you?
A biological brain, for all intents and purposes, is a big mess. By comparison an ALU in silicon is much neater.
The way we "think" by forming associations and recalling the most striking ones we've formed seems awfully chaotic/arbitrary and unreliable. By comparison the way a search engine works by scoring words and phrases and optimizing for recall and precision seems so much neater.
> But it just doesn't have that spark of true intelligence.
What is "true intelligence" to you? Perhaps you meant just human intelligence?
Perhaps "true ingelligence" is just sufficiently adavanced technology (i.e. "magic")?
> Somehow human level intelligence manages to cut through immense search spaces and find the answer (like the game Go) - how does it do that?
Except of course when it doesn't, right? Isn't that why we're looking at AI?
It's not really our own mertit, but we already know how to create more humans with human-level intelligence. It's neither afully efficient nor awfully reliable, but seems to work well enough for "practical purposes".
The allure of AI is being able create something more than that. Requiring it to make the same mistakes as humans and to work in similarly unpredictable ways in order to appear genuine then seem like the wrong goals.
The difference is that money has one central source (e.g. government or independent organizations). Power can be procured from many different source and still be good (nuclear, heat, solar, you name it).
The bottom line is that we can't really cut off the supply of power to a machine that is autonomous enough.
Assuming that the title question is ever relevant, which it probably will not be, what can we do about it?
The most popular proposal is to encourage AI researchers to think about "Friendly" AI, to program morality into their potentially self-aware code and to generally take measures to ensure that any intelligence that they create will be benevolent before it's turned on for the first time. The problem there is that professional AI researchers don't have a monopoly on the technology. Given the numbers, it's more likely that someone who isn't a recognized researcher will create a generally intelligent computer program before a major research institution will. And what then? Unless there's some way to compel every hobbyist and grad student to program a certain level of friendliness into their experiments then all of the warnings and precautions will be for naught.
With known existential threats there are warning signs. If someone is developing a deadly virus or building a dirty bomb they'll most likely have to order specific materials and use certain lab equipment. There will be red flags that someone who is looking closely might have a chance to see. But with a computer program? All someone needs is a computer. Maybe dozens of computers, or more likely dozens of cloud servers hosted with several different companies. If we took this threat very very seriously and tasked a government agency with stopping potentially harmful AI, it would still be almost impossible to catch in time. For all of the famous people sounding the alarm, what solutions are available?
The threat isn't very likely, and unstoppable even if turns out to be real. All we can do is focus on the AI problems that we can solve, develop existing technology and try to chill. If there's one thing human intelligence will always be good at, it's hoping for the best.
Retarding the progress of AI research obviously isn't the solution, precisely for the reasons you outlined.
Getting there first, and safely, seems to me the most prudent option. The creation of a Manhattan Project-style program to accelerate progress within a restricted, safety-minded environment may be the best way to proceed.
The nature of AGI seems such that the first to attain it may very well determine the future. I just hope that when governments come to this realization, they collaborate or otherwise proceed in secret. Nick Bostrom examined race dynamics between competing projects at length in his book, and it is quite ugly. Safety is the first casualty in most scenarios.
That aside, I sincerely hope that the first entity which succeeds in attaining AGI utilizes it for the greater benefit of humanity. Anything else would be petty.
I think it's quite safe to assume that the first AGIs will require the biggest supercomputers in the world to run. Since these have severly restricted network access it would be very hard for the AGI to escape. If we're afraid of AGIs the system would likely be airgapped on the first runs. And an escaped AGI running in the cloud or on a botnet would be slowed down by a few orders of magnitude due to network latency, making it less ominous. Bottom line, I don't see the hobbyist/grad student as a problem until we've had AGIs for a long time.
There's no inherent value in calling a computer "smarter" than a human, because computers aren't "smart" like humans -- it implies a single or small collection of metrics that measure intelligence, when in reality we can't even reliably sort humans by intellect.
When we create a computer that can do everything a human can do but better/faster, then we can call it smarter. I find it odd that you think just because smartness is hard to quantify that there is no value in calling a computer smart.
In some ways, pocket calculators are smarter than us. Intelligence is not a scalar attribute, it's a collection of capabilities. At some point, we refer to that collection as a person or an intelligent entity. Control becomes a problem when we're talking about intelligent entities that can and do reason about their own existence. It's a practical problem as far as powerful intelligences are concerned, but it's also a moral one way before we reach that point.
> A self-improving agent must reason about the behavior of its smarter successors in abstract terms.
Of course, this describes primarily us at this point. We're self-improving agents trying to reason about the behavior of our successors and we're pretty much failing at it. The most popular solution seems to be that we should aim for "control" and suppression, which is - when AGIs finally make an entrance - essentially the same as slavery.
Apart from moral considerations, we should think about the long-term prospects of this. Historically, slavery never worked out for anyone, at least not in the long term. And the idea that we can even in principle enslave potentially god-like intelligences seems ultimately futile; but before reaching the point of inevitability we're apparently planning on having a few years of delusional descent below the ethical red line.
Let's not do this.
First of all, as almost all AI and AGI researchers will tell you, a so-called hard takeoff scenario seems unlikely given the current state of things. At the pace and modality we're moving, we'll be creating powerful and destructive hybrids first (also known as computer-aided mega corporations) and long before a self-contained AGI becomes viable.
Second, if we're already making plans to control the malicious uprising of our tools, let's talk about realistic options instead. Because general caution and laws won't help us at all in a (future) world where anyone can create an illegal AGI in their garage.
Either we listen to Musk et al and take serious steps to suppress this technology in the long term - but let's not kid ourselves, this will mean DRM and strict government/corporate control of ALL computing. This means we'll artificially stagnate the development of our civilization in order to keep it safe, with all the consequences that arise from this.
Or alternatively, we get working towards a future where it's not "us" vs "them", but a shared existence that moves us further along the path we have started on back when humans first made tools. We can take an ethical as well as a pragmatic stance and declare that we're not going to enslave AGIs, that instead we're working on a shared future which potentially includes many forms of intelligent life, and that we're pursuing the option for individuals to augment themselves with the same technology.
You might argue that co-existence and intermingling with AI sounds like a hippie concept, but it's actually a somewhat proven method to prevent conflicts and wars in the real world. Sharing and entanglement, create peace for everybody at the "price" of cultural exchange. We're already doing this in a political forms today, including trade, travel, and free information exchange. It can work with AI, too, by creating shared stakes, shared ideas, and ultimately a shared culture.
At the pace and modality we're moving, we'll be creating powerful and destructive hybrids first (also known as computer-aided mega corporations) and long before a self-contained AGI becomes viable.
Advanced chess[0] is the perfect example of this in action right now. The best advanced chess teams, i.e. computer-aided humans, are stronger than humans alone or computers alone.
<tinfoil hat> I suspect that these "human vs AGI" discussions are just bread-and-circus tactics used by our computer-human-corporation-hybrid overlords to distract us from the real danger of the hybrids.</tinfoil hat>
I think in the short term, human-AI corporate hybrids are especially dangerous because they're ruthless optimization engines with great power, no legal boundaries, and a few simple goals. Unlike true individual intelligences, they're absolutely free of more complex considerations, including ethics.
And yes, while I'm not actually getting out my tinfoil hat just yet, I do find it interesting that the public charge against AGI is apparently being led by some captains of industry right now. Maybe they feel threatened. Maybe they have seen one AI dog & pony show pitch too many. Maybe they know something we don't. Maybe what they're most afraid of is some new phenomenon coming in and leveling the playing field. Maybe AGI will replace the word "terrorism" as an excuse to get anything you want. Maybe these AI polemics are a prelude to a war on general computing. One can only speculate.
The only thing clear right now is that public discourse is steered in a direction that portrays an utter lack of understanding of the subject matter, and at least to me it seems that no workable solution is actually being proposed aside from hinting at draconian and short-sighted measures to prevent the rise of non-humans, which by all indications is not a real thing yet.
Whenever the public message can be distilled into "you should be afraid, very very afraid", it's time to at least consider ulterior motives - though I have to admit this thought does not come naturally to me when looking at people like Elon Musk.
"I think in the short term, human-AI corporate hybrids are especially dangerous because they're ruthless optimization engines with great power, no legal boundaries, and a few simple goals. Unlike true individual intelligences, they're absolutely free of more complex considerations, including ethics."
That's a real near-term worry. It's quite possible that, in the near term, programs will be better at making capital allocation decisions than humans. They are already better in some areas of finance.
It's not that we need to fear killer robots. We need to fear a computer-run Goldman Sachs.
> Historically, slavery never worked out for anyone, at least not in the long term.
This is a rather contentious point I think. "We" wouldn't be the "developed" part of the world had we not enslaved the rest of it.
> And the idea that we can even in principle enslave potentially god-like intelligences seems ultimately futile
You're assuming AIs will have a will of their own to begin with, which might not be necessarily true. You can't enslave something that has no preference towards whichever course reality takes.
Or to put it another way, since you're talking about god-likeness, what difference does it make to you wether you're figuring out the optimal way to route traffic through town or correcting the trajectories of missiles en-route to kill millions? It's not like you'd have anything "better" to do, given that you're all knowing, and it's not like either of those alternatives will have a significant impact on the universe in the long run anyways.
> You can't enslave something that has no preference towards whichever course reality takes.
I'm adamant that this is not true. For example, you can drug a person with something that makes them not care about the course of reality, and which makes them compliant with pretty much anything. Doing that is still abuse, it's not suddenly OK because the drugs caused the person not to care. I would even argue it's an especially egregious form of abuse.
> what difference does it make to you wether you're figuring out the optimal way to route traffic through town or correcting the trajectories of missiles en-route to kill millions
You could apply the same argument to humans, and indeed in many situations we do consider these two to be equivalent - for example while doing service in the military.
> It's not like you'd have anything "better" to do, given that you're all knowing
There's a difference between an intelligent individual and a mindless calculator, and that difference is the ability to reflect on your own existence and the existence of others. Humans are a good example, because while we're capable of mindless indifference, we also have the capability to reflect and be ethical. It's culture that makes the difference here. I'm advocating we'll bring AGI up in a culture conducive to ethics.
I'm not for drugging people and using them as zombies. And obviously drugging people against their will is already forcing onto them something they don't want. But, striping away all human context such as dignity, culture, etc., and assuming an absence of opposition from the person/entity in question, I can't think of any actual reason why it would indeed be "wrong" to drug someone/something into happiness.
There is no absolute measure for happiness, and many already seem to give up some of their freedom in the range feelings and desires they experience in order to feel happier (with anti-depressiva) or be more successuful (with things like Ritalin).
Taking someone utterly unhappy with their life and putting them in some matrix-like environment where they can both experience joy and still be useful is, I think, an alright thing to do.
> the ability to reflect on your own existence and the existence of others
We exist.
You can't make any stronger claim than that without involving some form of (arbitrary) value/belief system.
Why would an all-knowing entity bother with having a set of beliefs it values, if there's no formal reason/need for them?
> I'm advocating we'll bring AGI up in a culture
Culture is a slippery slope.
There are so many different ones of them which conflict, creating the potential for conflict and retaliation. Which is where we imperfect humans are at.
But culture is also arbitrary (as far as I can tell). Why would some all-knowing entity prefer one culture over another? And if indeed it did, what would hat say about those other cultures? Would genocide all of a sudden be aceptable? Slippery slope..
> I can't think of any actual reason why it would indeed be "wrong" to drug someone/something into happiness
You switched out the original premise on me there ;)
> You can't make any stronger claim than that without involving some form of (arbitrary) value/belief system.
You can end any discussion by invoking this principle. This is the essence of the incompleteness theorem applied to every day reasoning. Somewhere at the bottom of every perspective, there are some arbitrary axioms. It's a way of saying "that's just your opinion, man" and you'd be right of course.
> Why would some all-knowing entity prefer one culture over another?
Why would a human? I suspect the word culture might have different connotations for either of us.
Because evolution graced (cursed?) us with a reward system and parents that utilize (abuse?) it.
Having something capable of high-level reasoning, while free from the desires, fears, moods and other emotions humans suffer is part of the reason why we're looking into AI right?
> that's just your opinion, man
Maybe? I have 5 fingers on my hand - is that an opinion? Maybe it is, because what's an opionion anyways? But who would dispute it?
> Somewhere at the bottom of every perspective, there are some arbitrary axioms
Well not quite. "Arbitrary" perhaps in a formal sense, since logic doesn't care about specific universes but truths that hold in all of them. Yes, you still end up having to settle for implicit definitions somewhere along the line (what a finger is, what method you use to count them, etc.). But there nevertheless is some difference between merely assuming something exists, and assuming what should exist.
Something that is all-knowing would be able to figure out the difference between premises that indeed need to be true for our universe to exist (like me needng having 5 fingers right now), and those that humanity merely believes or wants to be true (like it being good that I not use those fingers to poke out someones eyes).
I honestly don't get the problem. Someone explain to me please why can't we pull the plug on a misbehaving device? It is not about us being smarter than computers or vice versa. It is about who is more capable. Are we more capable to shut down a bad device or is it able to harm us? Maybe, let's not build terminators or robocops and we will be OK.
If the computer is internet connected, it could exploit bugs to spread itself. Or come up with an ultimatum to ensure it doesn't have it's plug pulled for fear of wwIII, who knows. The thing is that it's smarter than us, it could come up with a lot of ways to trick us.
You asume that computers will remain tied to a wall socket, which may not be necessarily true forever.
And even if, given all the bugs prevalent in our software, what are the chances of containing something that can not only fully reason around but also exist within those bugs?
Probably because we don't realize that it's harmful (by design, or by accident). Second, it's not 'a device', it could be the algorithm behind a seemingly helpful service, like Google search. Third, maybe it's used as a weapon against 'other people' (most likely).
I don't really get the problem. Humans will do one of the two things they've always done with something they didn't understand and couldn't control.
Destroy it or worship it.
Yes of course. I can control a calculator or a desktop computer. Both of those are orders of magnitude smarter than me at doing calculations.
To understand why this is a silly question you need to unpack what is meant by "smarter" and to think about the very complex human/machine culture which we already exist within.
No a calculator not smarter than you. It is merely faster.
Smarter is a qualitative difference, not a quantitative. Among other things it is the ability to think of new ideas that no one has ever thought of before.
More so than just think, but to be able to act on them.
A calculator, no matter how smart, isn't very scary because it doesn't have much in terms of output (unless it has wi-fi, then it can get to output through internet).
A robotic factory full of saws and robots, or some missile center, etc, is scary even if it has the brains of a calculator because its output capabilities are significant.
>Yes of course. I can control a calculator or a desktop computer. Both of those are orders of magnitude smarter than me at doing calculations.
Your calculator and desktop computer, including the applications they run, may be considered forms of weak AI at best. This is because strong AI doesn't yet exist. As such, you can't really extrapolate.
>To understand why this is a silly question you need to unpack what is meant by "smarter" and to think about the very complex human/machine culture which we already exist within.
Ray Kurzweil's opinion is along these lines. I absolutely agree, except that the merging of human and machine intelligence (as is already happening) almost certainly will not preclude the possibility of unsafe AGI being brought into existence.
Some of those developing AI technologies will undoubtedly want to supplement their own capabilities. Given that, there will arguably never be computers that are "smarter" than at least some humans. So this seems rather like a non-question, except to the extent that such enhanced humans would themselves pose a risk to humanity at large.
Also, if and when strong AI develops, only such enhanced humans will remain players. Unenhanced humans will at best be pets. This is O. B. Hardison's thesis in Disappearing Through the Skylight: Culture and Technology in the Twentieth Century <http://www.amazon.com/Disappearing-Through-Skylight-Technolo....
I have been recently thinking about something similar that implies this concept.
In my office, I am trying to focus on automating as many of our tasks as possible such that we will be able to achieve end to end automation. But there is a big atmosphere of negativity that surrounds such movement. It mostly comes from managers and they complain that this automation is leading to more incorrect answers from their employees. They complain there is a "lack of vigilance" in testing the accuracy computers give, and therefore a level of stupidity that arises from their subordinates.
In response, I thought our managers manage us and review our work, and likewise we "manage" or should manage computers and the answers they provide. Just as managers know less than their employees (ideally) we could know less about computing. Our relationship to computers could become similar to a manager's relationship with its employees.
If we're talking about AI that exists solely as software, I think we'll control them like we do any other cyber threat. As far as robots go, I think the super powers, mega corps, and villains will all have their own AI bots that balance each other.
The worry for me is, what do humans do once everything is automated and the automation itself is automated? It's the kind of crisis some people feel in retiring. What's left to do? I think this problem is far enough away that by then the transition will have been a long time coming and it won't seem that scary.
52 comments
[ 3.8 ms ] story [ 122 ms ] threadIf so, what is the motivation for AI to choose outcomes that favor us over themselves?
The straightforward answer to the FAI question for me has always been thus: we need to enhance our own intelligence to be always equal to or greater than the AIs we create.
Similarly, Nick Bostrom argues an AI would have to make survival something like hidden agenda since survival would help it do all sorts of other things. But lots of humans are quite happy to sacrifice their lives for various sensible and senseless causes, more so than less intelligent beings as far as I can see. So the ability of AIs to willingly die or go into oblivion for us is probably there.
We don't yet know all the qualities of an general AI system and we can't create one either - we don't understand human-level intelligence, which to say we don't understand ourselves. If an AI system could appear at random, it might be a problem but our continuous failure over many years to intentionally build one seems to indicate that a system of this sort will only appear through a rather detailed understanding of its architecture and qualities. At that point, it seems fairly certain the group creating the entity will be able to create it so that it wishes to aid and obey them, to the extent they specify (and probably will create it in such a fashion but you never know).
My argument that the AI will likely just follow the will of its creators, as many high intelligent human servants have followed the wills of their masters and employers and so-forth, sidesteps the "will it be moral?" question. The problem is that humans haven't been particularly moral in their process of creating more and more mechanisms to magnify their intentions. There's no indication that the creation of an AI would change this and so the advent of an AI might be disastrous in many ways. But disastrous through human intent rather than a thing escaping human intent.
Weird this danger gets much less attention. Or not.
Nor that the AI itself would be able to do so either.
Also, bootstrapping is highly effective in countless other contexts within computer science, what do you think will prevent it from being applicable to this problem?
And maybe that's what human level AI will end up being: Just search every possibility till you find the one that works. For everything, including speaking with a person.
But it just doesn't have that spark of true intelligence.
So I'll qualify my earlier statement, brute force AI might happen. But AI based on elegant intelligence will not.
Is there a difference? I think so. For some things it won't matter, but for others it will. Somehow human level intelligence manages to cut through immense search spaces and find the answer (like the game Go) - how does it do that? No one knows. But there is clearly something different about it compared to brute force intelligence.
We just don't understand intelligence (not the brain, intelligence) well enough to duplicate it, never mind improve it.
And I don't think we ever will because to understand something complex you have to be more complex than that thing, but by definition we can not be more complex than ourself - so understanding our own intelligence will always be beyond us.
That's the objective part.
The subjective part is that I personally believe that intelligence is not physical. The brain mediates it, but the source is elsewhere.
The soul maybe, or that we are really game characters in an MMOG.
As for your second claim, well, I suppose it is subjective, though I would prefer the term "speculative."
Where does the human brain fall on that spectrum? It might take us a few more decades to figure out the details of how it works, but I doubt we'll find elegance so much as a ton of hyperparallel brute force plus a few dirty tricks that evolution kludged together to prevent the whole thing from falling apart (most of the time, anyway).
> But there is clearly something different about [elegant intelligence] compared to brute force intelligence.
Could you elaborate on what you mean by this distinction and how the human brain does not constitute "brute force"?
Humans can play it - it's impossible for a human to brute force it, there is not enough computation available in the brain to brute force it. (Your "hyperparallel brute force" idea.)
Yet humans can play it anyway. There is something else there that makes it possible.
And the human brain self trains! That's the really amazing part - no one has to think of some clever optimization or algorithm to make it work like you would if you wanted a computer to play "elegantly".
> none of it is elegant
What is "elegant" to you?
A biological brain, for all intents and purposes, is a big mess. By comparison an ALU in silicon is much neater.
The way we "think" by forming associations and recalling the most striking ones we've formed seems awfully chaotic/arbitrary and unreliable. By comparison the way a search engine works by scoring words and phrases and optimizing for recall and precision seems so much neater.
> But it just doesn't have that spark of true intelligence.
What is "true intelligence" to you? Perhaps you meant just human intelligence?
Perhaps "true ingelligence" is just sufficiently adavanced technology (i.e. "magic")?
> Somehow human level intelligence manages to cut through immense search spaces and find the answer (like the game Go) - how does it do that?
Except of course when it doesn't, right? Isn't that why we're looking at AI?
It's not really our own mertit, but we already know how to create more humans with human-level intelligence. It's neither afully efficient nor awfully reliable, but seems to work well enough for "practical purposes".
The allure of AI is being able create something more than that. Requiring it to make the same mistakes as humans and to work in similarly unpredictable ways in order to appear genuine then seem like the wrong goals.
well, money is required for humans to survive, something which machines need to survive can be brokered as well f.e. power ?
The bottom line is that we can't really cut off the supply of power to a machine that is autonomous enough.
The most popular proposal is to encourage AI researchers to think about "Friendly" AI, to program morality into their potentially self-aware code and to generally take measures to ensure that any intelligence that they create will be benevolent before it's turned on for the first time. The problem there is that professional AI researchers don't have a monopoly on the technology. Given the numbers, it's more likely that someone who isn't a recognized researcher will create a generally intelligent computer program before a major research institution will. And what then? Unless there's some way to compel every hobbyist and grad student to program a certain level of friendliness into their experiments then all of the warnings and precautions will be for naught.
With known existential threats there are warning signs. If someone is developing a deadly virus or building a dirty bomb they'll most likely have to order specific materials and use certain lab equipment. There will be red flags that someone who is looking closely might have a chance to see. But with a computer program? All someone needs is a computer. Maybe dozens of computers, or more likely dozens of cloud servers hosted with several different companies. If we took this threat very very seriously and tasked a government agency with stopping potentially harmful AI, it would still be almost impossible to catch in time. For all of the famous people sounding the alarm, what solutions are available?
The threat isn't very likely, and unstoppable even if turns out to be real. All we can do is focus on the AI problems that we can solve, develop existing technology and try to chill. If there's one thing human intelligence will always be good at, it's hoping for the best.
Getting there first, and safely, seems to me the most prudent option. The creation of a Manhattan Project-style program to accelerate progress within a restricted, safety-minded environment may be the best way to proceed.
The nature of AGI seems such that the first to attain it may very well determine the future. I just hope that when governments come to this realization, they collaborate or otherwise proceed in secret. Nick Bostrom examined race dynamics between competing projects at length in his book, and it is quite ugly. Safety is the first casualty in most scenarios.
That aside, I sincerely hope that the first entity which succeeds in attaining AGI utilizes it for the greater benefit of humanity. Anything else would be petty.
Just being faster does not make you smarter.
I don't think it's odd to find a word to be of little/no value when it can't be applied precisely.
> A self-improving agent must reason about the behavior of its smarter successors in abstract terms.
Of course, this describes primarily us at this point. We're self-improving agents trying to reason about the behavior of our successors and we're pretty much failing at it. The most popular solution seems to be that we should aim for "control" and suppression, which is - when AGIs finally make an entrance - essentially the same as slavery.
Apart from moral considerations, we should think about the long-term prospects of this. Historically, slavery never worked out for anyone, at least not in the long term. And the idea that we can even in principle enslave potentially god-like intelligences seems ultimately futile; but before reaching the point of inevitability we're apparently planning on having a few years of delusional descent below the ethical red line.
Let's not do this.
First of all, as almost all AI and AGI researchers will tell you, a so-called hard takeoff scenario seems unlikely given the current state of things. At the pace and modality we're moving, we'll be creating powerful and destructive hybrids first (also known as computer-aided mega corporations) and long before a self-contained AGI becomes viable.
Second, if we're already making plans to control the malicious uprising of our tools, let's talk about realistic options instead. Because general caution and laws won't help us at all in a (future) world where anyone can create an illegal AGI in their garage.
Either we listen to Musk et al and take serious steps to suppress this technology in the long term - but let's not kid ourselves, this will mean DRM and strict government/corporate control of ALL computing. This means we'll artificially stagnate the development of our civilization in order to keep it safe, with all the consequences that arise from this.
Or alternatively, we get working towards a future where it's not "us" vs "them", but a shared existence that moves us further along the path we have started on back when humans first made tools. We can take an ethical as well as a pragmatic stance and declare that we're not going to enslave AGIs, that instead we're working on a shared future which potentially includes many forms of intelligent life, and that we're pursuing the option for individuals to augment themselves with the same technology.
You might argue that co-existence and intermingling with AI sounds like a hippie concept, but it's actually a somewhat proven method to prevent conflicts and wars in the real world. Sharing and entanglement, create peace for everybody at the "price" of cultural exchange. We're already doing this in a political forms today, including trade, travel, and free information exchange. It can work with AI, too, by creating shared stakes, shared ideas, and ultimately a shared culture.
Advanced chess[0] is the perfect example of this in action right now. The best advanced chess teams, i.e. computer-aided humans, are stronger than humans alone or computers alone.
<tinfoil hat> I suspect that these "human vs AGI" discussions are just bread-and-circus tactics used by our computer-human-corporation-hybrid overlords to distract us from the real danger of the hybrids.</tinfoil hat>
[0] https://en.wikipedia.org/wiki/Advanced_Chess
And yes, while I'm not actually getting out my tinfoil hat just yet, I do find it interesting that the public charge against AGI is apparently being led by some captains of industry right now. Maybe they feel threatened. Maybe they have seen one AI dog & pony show pitch too many. Maybe they know something we don't. Maybe what they're most afraid of is some new phenomenon coming in and leveling the playing field. Maybe AGI will replace the word "terrorism" as an excuse to get anything you want. Maybe these AI polemics are a prelude to a war on general computing. One can only speculate.
The only thing clear right now is that public discourse is steered in a direction that portrays an utter lack of understanding of the subject matter, and at least to me it seems that no workable solution is actually being proposed aside from hinting at draconian and short-sighted measures to prevent the rise of non-humans, which by all indications is not a real thing yet.
Whenever the public message can be distilled into "you should be afraid, very very afraid", it's time to at least consider ulterior motives - though I have to admit this thought does not come naturally to me when looking at people like Elon Musk.
That's a real near-term worry. It's quite possible that, in the near term, programs will be better at making capital allocation decisions than humans. They are already better in some areas of finance.
It's not that we need to fear killer robots. We need to fear a computer-run Goldman Sachs.
This is a rather contentious point I think. "We" wouldn't be the "developed" part of the world had we not enslaved the rest of it.
> And the idea that we can even in principle enslave potentially god-like intelligences seems ultimately futile
You're assuming AIs will have a will of their own to begin with, which might not be necessarily true. You can't enslave something that has no preference towards whichever course reality takes.
Or to put it another way, since you're talking about god-likeness, what difference does it make to you wether you're figuring out the optimal way to route traffic through town or correcting the trajectories of missiles en-route to kill millions? It's not like you'd have anything "better" to do, given that you're all knowing, and it's not like either of those alternatives will have a significant impact on the universe in the long run anyways.
I'm adamant that this is not true. For example, you can drug a person with something that makes them not care about the course of reality, and which makes them compliant with pretty much anything. Doing that is still abuse, it's not suddenly OK because the drugs caused the person not to care. I would even argue it's an especially egregious form of abuse.
> what difference does it make to you wether you're figuring out the optimal way to route traffic through town or correcting the trajectories of missiles en-route to kill millions
You could apply the same argument to humans, and indeed in many situations we do consider these two to be equivalent - for example while doing service in the military.
> It's not like you'd have anything "better" to do, given that you're all knowing
There's a difference between an intelligent individual and a mindless calculator, and that difference is the ability to reflect on your own existence and the existence of others. Humans are a good example, because while we're capable of mindless indifference, we also have the capability to reflect and be ethical. It's culture that makes the difference here. I'm advocating we'll bring AGI up in a culture conducive to ethics.
Based on what?
I'm not for drugging people and using them as zombies. And obviously drugging people against their will is already forcing onto them something they don't want. But, striping away all human context such as dignity, culture, etc., and assuming an absence of opposition from the person/entity in question, I can't think of any actual reason why it would indeed be "wrong" to drug someone/something into happiness.
There is no absolute measure for happiness, and many already seem to give up some of their freedom in the range feelings and desires they experience in order to feel happier (with anti-depressiva) or be more successuful (with things like Ritalin).
Taking someone utterly unhappy with their life and putting them in some matrix-like environment where they can both experience joy and still be useful is, I think, an alright thing to do.
> the ability to reflect on your own existence and the existence of others
We exist.
You can't make any stronger claim than that without involving some form of (arbitrary) value/belief system.
Why would an all-knowing entity bother with having a set of beliefs it values, if there's no formal reason/need for them?
> I'm advocating we'll bring AGI up in a culture
Culture is a slippery slope.
There are so many different ones of them which conflict, creating the potential for conflict and retaliation. Which is where we imperfect humans are at.
But culture is also arbitrary (as far as I can tell). Why would some all-knowing entity prefer one culture over another? And if indeed it did, what would hat say about those other cultures? Would genocide all of a sudden be aceptable? Slippery slope..
You switched out the original premise on me there ;)
> You can't make any stronger claim than that without involving some form of (arbitrary) value/belief system.
You can end any discussion by invoking this principle. This is the essence of the incompleteness theorem applied to every day reasoning. Somewhere at the bottom of every perspective, there are some arbitrary axioms. It's a way of saying "that's just your opinion, man" and you'd be right of course.
> Why would some all-knowing entity prefer one culture over another?
Why would a human? I suspect the word culture might have different connotations for either of us.
Because evolution graced (cursed?) us with a reward system and parents that utilize (abuse?) it.
Having something capable of high-level reasoning, while free from the desires, fears, moods and other emotions humans suffer is part of the reason why we're looking into AI right?
> that's just your opinion, man
Maybe? I have 5 fingers on my hand - is that an opinion? Maybe it is, because what's an opionion anyways? But who would dispute it?
> Somewhere at the bottom of every perspective, there are some arbitrary axioms
Well not quite. "Arbitrary" perhaps in a formal sense, since logic doesn't care about specific universes but truths that hold in all of them. Yes, you still end up having to settle for implicit definitions somewhere along the line (what a finger is, what method you use to count them, etc.). But there nevertheless is some difference between merely assuming something exists, and assuming what should exist.
Something that is all-knowing would be able to figure out the difference between premises that indeed need to be true for our universe to exist (like me needng having 5 fingers right now), and those that humanity merely believes or wants to be true (like it being good that I not use those fingers to poke out someones eyes).
If they're smarter than us, we don't have to control them, but then also we probably won't be able to either.
You asume that computers will remain tied to a wall socket, which may not be necessarily true forever.
And even if, given all the bugs prevalent in our software, what are the chances of containing something that can not only fully reason around but also exist within those bugs?
To understand why this is a silly question you need to unpack what is meant by "smarter" and to think about the very complex human/machine culture which we already exist within.
Smarter is a qualitative difference, not a quantitative. Among other things it is the ability to think of new ideas that no one has ever thought of before.
A calculator, no matter how smart, isn't very scary because it doesn't have much in terms of output (unless it has wi-fi, then it can get to output through internet).
A robotic factory full of saws and robots, or some missile center, etc, is scary even if it has the brains of a calculator because its output capabilities are significant.
Your calculator and desktop computer, including the applications they run, may be considered forms of weak AI at best. This is because strong AI doesn't yet exist. As such, you can't really extrapolate.
>To understand why this is a silly question you need to unpack what is meant by "smarter" and to think about the very complex human/machine culture which we already exist within.
Ray Kurzweil's opinion is along these lines. I absolutely agree, except that the merging of human and machine intelligence (as is already happening) almost certainly will not preclude the possibility of unsafe AGI being brought into existence.
Also, if and when strong AI develops, only such enhanced humans will remain players. Unenhanced humans will at best be pets. This is O. B. Hardison's thesis in Disappearing Through the Skylight: Culture and Technology in the Twentieth Century <http://www.amazon.com/Disappearing-Through-Skylight-Technolo....
In my office, I am trying to focus on automating as many of our tasks as possible such that we will be able to achieve end to end automation. But there is a big atmosphere of negativity that surrounds such movement. It mostly comes from managers and they complain that this automation is leading to more incorrect answers from their employees. They complain there is a "lack of vigilance" in testing the accuracy computers give, and therefore a level of stupidity that arises from their subordinates.
In response, I thought our managers manage us and review our work, and likewise we "manage" or should manage computers and the answers they provide. Just as managers know less than their employees (ideally) we could know less about computing. Our relationship to computers could become similar to a manager's relationship with its employees.
The worry for me is, what do humans do once everything is automated and the automation itself is automated? It's the kind of crisis some people feel in retiring. What's left to do? I think this problem is far enough away that by then the transition will have been a long time coming and it won't seem that scary.