Nick Bostrom is a philosopher, not an engineer. If he spent his time actually making things instead of fear mongerng, he might have an appreciation for how truly difficult it is to actually get things to work, and the challenges that would still confront advanced AI. His instantaneous end-the-world outcomes are not plausible and will never be realized. Yet he argues for draconian, fascist controls on technology and society that have real negative ramifications.
I don’t think it’s necessarily a bad thing to let a philosopher muse a bit about ethics and advise caution. Maybe we won’t have the AI singularity, but some of the questions brought up (should we improve ourselves? Is technology inherently good?) are quite important and contentious.
I don't like dying of primitive diseases, and aging.
> Is technology inherently good?
It doesn't have to be inherently (intrinsicly?) good. It has good ramifications, such as the literally billions of people that have had a chance to live and be brought out of poverty that wouldn't have even existed otherwise.
Those aren't insightful questions. They only sound wise when you don't think critically about them.
You have to look at the longer term with technology, so far it's been an overwhelmingly positive force for humanity. I personally think it's the greatest thing ever, but we may one day invent the thing that ends us. Maybe that's AI, maybe we unlock a power in the physical world that we can't be trusted with. For example if anyone could build antimatter bombs in the garage, terrorists or other discontents would have used it to end civilization. There's no law of nature that says great power must always be hard to come by, and I wonder if that's not the answer to the Fermi paradox. Civilization might be a short-term blip, an unstable state on cosmic timescales.
H.G. Wells ends the Time Machine with: "for the question had been discussed among us long before the Time Machine was made - thought but cheerlessly of the Advancement of Mankind, and saw in the growing pile of civilization only a foolish heaping that must inevitably fall back upon and destroy its makers in the end. If that is so, it remains for us to live as though it were not so."
I definitely agree with you. But i cannot stop thinking about the dwarfs in the lord of the rings that went too far, dug too deep, potentially due to the rhetoric youre having. That was their doom.
Its a book, just a book. But i keep on thinking about it because i find it might also apply to humans and technology.
I think it is fair to see both sides of technological advancement. For every positive use of technology there is often some negative use. I think of nuclear power vs. nuclear weapons. No one thinks we should stop investigating chemicals and diseases just because someone could use the information to create chemical weapons or biological weapons. However, we do have social and political tools to attempt to deal with these negatives, whether they be conventions or treaties. As a more recent data-point look at all of the hubbub caused by social networks and their use by opposing countries to destabilize political process. We are going to make real changes to technology to attempt to mitigate those issues.
We need to think about the negative consequences of gene editing, robotics, super powerful AI, etc. not to overshadow the positive benefits but to help remind us to create the tools necessary to mitigate those negatives. If the only possible ramifications were good then we wouldn't need to be having this conversation.
It reminds me of interviews with Steven Pinker and his opinion on potential negative outfall from AI research. He claims that engineers are generally good-willed and they try to do the best they can, e.g. they don't design bridges that will fall. Yet bridges fall down sometimes, occasionally quite soon after they are built. I'm glad there is some external oversight on issues that are potentially world-changing, just to have a sober second opinion.
Because if he’s right we really, really want to pay attention to this problem so we don’t all die.
I’m sure Bostrom abstracts away an enormous amount of the difficulty in implementation but that’s quite defensible. Moore’s Law isn’t the only curve that works well to explain technological trends for decades. We can see in broad outlines what’s coming decades from now.
> Because if he’s right we really, really want to pay attention to this problem so we don’t all die.
You could say that about any doomsayer. It's important to use their reputation and authority, and your own context within reason, to weight the significance of their prophecies.
Of course, I'm not saying we should ignore all doomsayers, just that we should weight their prophecies with a reasonable estimate of trust. Saying "if it's right then we need to pay attention" isn't a good argument in and of itself. It would imply that we have to pay attention to thousands of additional crazy prophecies, like those of the Westboro Baptist Church or that lizard people are taking control of the earth.
Whether Boston is a genuine trusted authority on this is a much more interesting discussion.
Decades from now we'll likely fight each other for food and water as climate change reduces the amount of arable land and important ecosystems collapse
That flies in the face of the history of progress. Maltheus was wrong for the last 200 years of doom and gloom predictions. Why would you expect differently now?
In actual fact, the amount of resources (atoms, power) in the Earth are orders of magnitude beyond what humans need for thousands of years, even assuming exponential growth. And the carrying capacity of the Earth has grown as fast or faster than population, explicitly because more people means more innovation and more capital which means higher carrying capacity. What's more, the per capita quality of life grows linearly with population, meaning a quadratic increase in available resources overall. More people = more prosperity. It's not a zero-sum game.
Inoculate yourself against the Malthusian ideology of the Club of Rome. The world is not ending, as much as their anti-human agenda would want you to believe.
Humans need more than atoms and power to live a comfortable life. We're rapidly destroying major ecosystems that our way of life depends on. Sure, the human race can survive huddled around breeder reactors in underground caves. But I'd like to spend old age on a planet that is reasonably similar to what we have right now.
> Draconian, fascist use of technology on people, that's a matter of actual importance.
That's what he's arguing for. A global police state that knows what everyone is doing, everywhere, and can intervene at a moment's notice to stop certain kinds of technological work.
Well its not an impossible argument to understand from a game theory perspective of civilization.
I don't agree with it. Much as i don't agree with the Dark Forest premise set forward in the Three Body Problem series, I thought it was very interesting nontheless and worth spending quite a bit of time musing over. As with Nick Bostroms thoughts.
Just because we disagree with a concept on ethical or practical or whatever reason doesn't mean we should be discussing it or that we can gain something from discussing it.
At least he can argue his side, unlike the politicians constantly pushing for a more and more draconian state.
I used to believe this because I used to believe that toy models were exercises in the capability of our thoughts, and any such exercise is worthwhile. But we have a discipline for that -- theology. Towards the end of the 20th century philosophers that are widely considered to be a part of the modern canon (in particular William James and Peirce, Wittgenstein too) thought practice as being indispensable to theory, and that theory that is formed without practice just cannot have the experiential fodder necessary to ground theory in factual principle.
It sounds obvious when you say it, but here we are. Philosophers themselves have either (a) taken an "ethnographic turn" (c/o John Dewey) where they realize the limitations of there own expertise and figure the standard they should strive for is being stronger describers of reality, disentangling its confusions, or (b) acquired expertise in the thing they are philosophing about, which makes them a kind of theorotician or highly informed historian.
If Bostrom ain't writing code how do we know if he is grounded in anything?
As opposed to hoping for voluntary self-restraint and then washing our hands of the consequences? I agree with his response:
> Yes of course surveillance capabilities increase the states’ ability to suppress the restless populations…
> With some of the developments in nuclear technology, we now have the ability to wage wars that would have immediate planetary implications. So our technological prowess is galloping, whereas our wisdom and ability to peacefully cooperate with one another seems to develop at a slower pace. So, I think we need to spur on this cooperation horse to make sure that we can keep up with the galloping technological horse.
> If we think about this as a race between our growing powers and our ability to use those powers wisely, it seems unclear who will win that race.
What is stopping people from refraining from unethical use of technology, right now? Nothing. Yet many people still don't refrain, and think if something profitable can be done, if people can be manipulated or used in a certain way, then it's "smart" to do that. What'd be the the problem with not having growing disparities of power, and first becoming decent societies, and then getting to play with all sorts of fancy tech?
> If machines produce everything we need, the outcome will depend on how things are distributed. Everyone can enjoy a life of luxurious leisure if the machine-produced wealth is shared, or most people can end up miserably poor if the machine-owners successfully lobby against wealth redistribution. So far, the trend seems to be toward the second option, with technology driving ever-increasing inequality.
-- Stephen Hawking
And that inequality and making people superflous will lead to the possibility of atrocities that will be committed because they can be over in an instant, with nobody but perpetrators left to accuse them. Even right here on HN I had a short conversation with someone who earnestly thought we will and should be replaced with GAI, the next step in evolution so to speak, and that people should work towards that in secret, because people wouldn't accept being superseded by something much "better". I don't know what would lead to such an outlook, but it exists.
Just corporations doing mass surveillance isn't an absence of mass surveillance, the difference would be governments (a singular state is not mentioned, nor a police state) doing that too, preferably in working democracies, the achievement of which is part of the above "using our powers wisely" (and choosing wisely, whom we empower to do what on our behalf).
> Nick Bostrom is a philosopher, not an engineer. If he spent his time actually making things instead of fear mongerng, he might have an appreciation for how truly difficult it is to actually get things to work, and the challenges that would still confront advanced AI.
I don't see how this argument even works. Lets change a premise and we get:
'Nick Bostrom is a philosopher, not an nuclear physicist.
... he might have an appreciation for how truly difficult it is to actually build a-bomb.'
Therefore he should not speak about potential dangers/threats of nuclear bomb arsenals.
And as we all know we came extremely close to few false-positive MADs.
> His instantaneous end-the-world outcomes are not plausible and will never be realized.
That's unsubstantiated hearsay claim by you. Can you list a single counter argument against runaway AGI?
>Yet he argues for draconian, fascist controls on technology and society that have real negative ramifications.
At this point I am quite certain you have not read a single thing he Bostrom, and only making claims of articles/comments/2ndary sources.
I think it was in Superinteligence, where he opens with a parable how overlooking dangers in short term is beneficial, but once you open pandora box you cant put it back.
He is not against it AI, but thinks we shouldn't happily jump into something without thinking of future outcomes.
Let me give an example.
Say CERN starts studying micro black holes. Those are absolutely safe to create as they 'fizzle out' due to hawkins radiation. So no one should stop CERN from making any precautions so that they accidentally create a black hole that doesn't evaporate?
Right? It's micro black hole they are safe, we have no tech that could create bigger one, and we have total understanding of Physics. No need for regulation or safety talking.
I think most concepts of transhumanism are yesterday's crave for hoverboards.
Still, I don't think all his fears are unfounded. While I am skeptical about artificial general intelligence that suddenly surpasses our ability to reason, I am more concerned about specialized agents that quantify nearly everything of our lives and gain influence through human actors overestimating their abilities and therefore creating a data burocracy that will indeed have problems with bias.
Not arguing against the transhumanist tech lead that might have or might not have a form of cocaine addiction that at least supports people dreaming of more advanced AI.
More concerned about the people that buy into it and treat it as gospel. I mean it comes from a machine that doesn't have these aweful restrains humans are subjected to...
This doomsday futurism has an interesting characteristic - the author of such prophecies can't lose. Either they are right and humanity (or what's left of it) will look back on them as prescient and clear thinking, or they are wrong and everyone will be too relieved and happy to care. The Y2K bug was another example.
There are 2 schools of thought on the Y2K bug, and the one you left out was that quite a lot of money was spent fixing bugs prior to the year 2000, which helped ensure that it was a minor event, with minor failures, instead of a bigger problem. There are computer systems we are even just now starting to hear had major problems that were addressed. Yes, there were people hyping it like crazy, and the average personal computer was not really at risk of anything major happening. But there were serious concerns that were addressed prior to 2000.
Also, it's likely the 9/11 attacks had less of an impact because of redundancies and backups put in place for the Y2K bug.
I don’t think you read my comment. I’m not passing judgement on whether Y2K was a legitimate concern. The point is those saying the world was going to end could never really be criticised no matter what happened. This is the doomsayer’s privilege.
29 comments
[ 3.4 ms ] story [ 74.3 ms ] threadWhy do we continue to give him airtime?
I don't like dying of primitive diseases, and aging.
> Is technology inherently good?
It doesn't have to be inherently (intrinsicly?) good. It has good ramifications, such as the literally billions of people that have had a chance to live and be brought out of poverty that wouldn't have even existed otherwise.
Those aren't insightful questions. They only sound wise when you don't think critically about them.
H.G. Wells ends the Time Machine with: "for the question had been discussed among us long before the Time Machine was made - thought but cheerlessly of the Advancement of Mankind, and saw in the growing pile of civilization only a foolish heaping that must inevitably fall back upon and destroy its makers in the end. If that is so, it remains for us to live as though it were not so."
Its a book, just a book. But i keep on thinking about it because i find it might also apply to humans and technology.
We need to think about the negative consequences of gene editing, robotics, super powerful AI, etc. not to overshadow the positive benefits but to help remind us to create the tools necessary to mitigate those negatives. If the only possible ramifications were good then we wouldn't need to be having this conversation.
It reminds me of interviews with Steven Pinker and his opinion on potential negative outfall from AI research. He claims that engineers are generally good-willed and they try to do the best they can, e.g. they don't design bridges that will fall. Yet bridges fall down sometimes, occasionally quite soon after they are built. I'm glad there is some external oversight on issues that are potentially world-changing, just to have a sober second opinion.
I’m sure Bostrom abstracts away an enormous amount of the difficulty in implementation but that’s quite defensible. Moore’s Law isn’t the only curve that works well to explain technological trends for decades. We can see in broad outlines what’s coming decades from now.
You could say that about any doomsayer. It's important to use their reputation and authority, and your own context within reason, to weight the significance of their prophecies.
And we did came quite close to doomsday scenarios, cold war, but also false-positives in automated detection defense systems.
Whether Boston is a genuine trusted authority on this is a much more interesting discussion.
In actual fact, the amount of resources (atoms, power) in the Earth are orders of magnitude beyond what humans need for thousands of years, even assuming exponential growth. And the carrying capacity of the Earth has grown as fast or faster than population, explicitly because more people means more innovation and more capital which means higher carrying capacity. What's more, the per capita quality of life grows linearly with population, meaning a quadratic increase in available resources overall. More people = more prosperity. It's not a zero-sum game.
Inoculate yourself against the Malthusian ideology of the Club of Rome. The world is not ending, as much as their anti-human agenda would want you to believe.
Technology doesn't mind, it doesn't have feelings. Draconian, fascist use of technology on people, that's a matter of actual importance.
That's what he's arguing for. A global police state that knows what everyone is doing, everywhere, and can intervene at a moment's notice to stop certain kinds of technological work.
I don't agree with it. Much as i don't agree with the Dark Forest premise set forward in the Three Body Problem series, I thought it was very interesting nontheless and worth spending quite a bit of time musing over. As with Nick Bostroms thoughts.
Just because we disagree with a concept on ethical or practical or whatever reason doesn't mean we should be discussing it or that we can gain something from discussing it.
At least he can argue his side, unlike the politicians constantly pushing for a more and more draconian state.
It sounds obvious when you say it, but here we are. Philosophers themselves have either (a) taken an "ethnographic turn" (c/o John Dewey) where they realize the limitations of there own expertise and figure the standard they should strive for is being stronger describers of reality, disentangling its confusions, or (b) acquired expertise in the thing they are philosophing about, which makes them a kind of theorotician or highly informed historian.
If Bostrom ain't writing code how do we know if he is grounded in anything?
> Yes of course surveillance capabilities increase the states’ ability to suppress the restless populations…
> With some of the developments in nuclear technology, we now have the ability to wage wars that would have immediate planetary implications. So our technological prowess is galloping, whereas our wisdom and ability to peacefully cooperate with one another seems to develop at a slower pace. So, I think we need to spur on this cooperation horse to make sure that we can keep up with the galloping technological horse.
> If we think about this as a race between our growing powers and our ability to use those powers wisely, it seems unclear who will win that race.
What is stopping people from refraining from unethical use of technology, right now? Nothing. Yet many people still don't refrain, and think if something profitable can be done, if people can be manipulated or used in a certain way, then it's "smart" to do that. What'd be the the problem with not having growing disparities of power, and first becoming decent societies, and then getting to play with all sorts of fancy tech?
> If machines produce everything we need, the outcome will depend on how things are distributed. Everyone can enjoy a life of luxurious leisure if the machine-produced wealth is shared, or most people can end up miserably poor if the machine-owners successfully lobby against wealth redistribution. So far, the trend seems to be toward the second option, with technology driving ever-increasing inequality.
-- Stephen Hawking
And that inequality and making people superflous will lead to the possibility of atrocities that will be committed because they can be over in an instant, with nobody but perpetrators left to accuse them. Even right here on HN I had a short conversation with someone who earnestly thought we will and should be replaced with GAI, the next step in evolution so to speak, and that people should work towards that in secret, because people wouldn't accept being superseded by something much "better". I don't know what would lead to such an outlook, but it exists.
Just corporations doing mass surveillance isn't an absence of mass surveillance, the difference would be governments (a singular state is not mentioned, nor a police state) doing that too, preferably in working democracies, the achievement of which is part of the above "using our powers wisely" (and choosing wisely, whom we empower to do what on our behalf).
I don't see how this argument even works. Lets change a premise and we get:
'Nick Bostrom is a philosopher, not an nuclear physicist. ... he might have an appreciation for how truly difficult it is to actually build a-bomb.' Therefore he should not speak about potential dangers/threats of nuclear bomb arsenals.
And as we all know we came extremely close to few false-positive MADs.
> His instantaneous end-the-world outcomes are not plausible and will never be realized. That's unsubstantiated hearsay claim by you. Can you list a single counter argument against runaway AGI?
>Yet he argues for draconian, fascist controls on technology and society that have real negative ramifications. At this point I am quite certain you have not read a single thing he Bostrom, and only making claims of articles/comments/2ndary sources. I think it was in Superinteligence, where he opens with a parable how overlooking dangers in short term is beneficial, but once you open pandora box you cant put it back. He is not against it AI, but thinks we shouldn't happily jump into something without thinking of future outcomes.
Let me give an example. Say CERN starts studying micro black holes. Those are absolutely safe to create as they 'fizzle out' due to hawkins radiation. So no one should stop CERN from making any precautions so that they accidentally create a black hole that doesn't evaporate? Right? It's micro black hole they are safe, we have no tech that could create bigger one, and we have total understanding of Physics. No need for regulation or safety talking.
Still, I don't think all his fears are unfounded. While I am skeptical about artificial general intelligence that suddenly surpasses our ability to reason, I am more concerned about specialized agents that quantify nearly everything of our lives and gain influence through human actors overestimating their abilities and therefore creating a data burocracy that will indeed have problems with bias.
Not arguing against the transhumanist tech lead that might have or might not have a form of cocaine addiction that at least supports people dreaming of more advanced AI.
More concerned about the people that buy into it and treat it as gospel. I mean it comes from a machine that doesn't have these aweful restrains humans are subjected to...
PR campaign at its best
Also, it's likely the 9/11 attacks had less of an impact because of redundancies and backups put in place for the Y2K bug.