This is easy to answer and quite frankly a stupid question.
Why should you have to pay for someone else's mistake? If a child runs out in front of a car, it is either their fault or their guardian's fault for allowing them to run into traffic. It's tragic, but to suggest the driver should give up their life for someone else's mistake is ridiculous.
Who in their right mind would get inside a car that is programmed to commit suicide in certain "ethical" circumstances?
> Who in their right mind would get inside a car that is programmed to commit suicide in certain "ethical" circumstances?
I suspect there are various classes of people that would choose this option. Believers of certain religions, old/hill people with high morals or strictly pragmatic, aspirant suicides, risk lovers, etc...
Actually, most google cars will be void of any passengers. They'll be google trucks (or amazon trucks). The "ethical" question takes a whole new dimension when it's the life of a child vs. millions of dollar of merchandize, expected-just in-time by enterprises who need them to make billions of profit.
The point is, even if people do buy such a car, they don't often come back for a second car. The business will run out of customers very quickly. You do the math.
> The point is, even if people do buy such a car, they don't often come back for a second car. The business will run out of customers very quickly.
This seems to assume that the frequency with which drivers (automated or otherwise) are faced with "someone is going to die, you or the other person, and which one depends on your driving choices" situations is very high.
In the specific example of a child at a tunnel entrance, you've got a good point. However, I think there are plenty of scenarios which you can make up that function similarly without providing a culpable victim.
For example, the self-driving car you're in is driving by an elementary school at the end of the day, so there are a number of children on the sidewalk. Suddenly, a truck appears coming at your car going in the wrong direction. Should your car take an evasive maneuver onto the sidewalk, with a high likelihood of striking a child?
I would argue that traffic law provides the answer. If you can't avoid violating any law, there is a clear sequence defined you should use to decide what law to violate (they're all numbered, TLDR is : violate the larger numbers first).
In traffic law people actually thought about this. First you are to safeguard pedestrians, then cyclists and motorcyclists then cars, then trucks/buses/... and finally yourself. Things like right side of the road, red lights, ... are far down. Things like trains/trams are tricky.
What is not generally understood is that this sequence matters. Most people realize that in order not to block an ambulance, you are not just allowed, but required to run a red light. Less known is that ignoring road safety, to some extent, is legal and required if you're bringing a seriously injured person to the hospital. If you can avoid a collision by crossing to the wrong side of the street, you are required to do that ...
So the legal answer here is simple and well-defined : Assuming you can't move to the other side of the road reasonably safely, you should slam your car into the truck (needless to say, as slowly as possible). You are certainly NOT to take liberties with the safety of pedestrians to avoid hitting a truck.
This is complicated by other laws of course. First you are not expected to do more than a certain "standard" human would do, second in this situation obviously there is a problem of intent when things do go south (did you intend to hit those pedestrians ? What about if you made that choice to avoid the truck ? What if you and the truck both decided to avoid each other and both went for the b). Who is the "root" responsible, and how is the responsibility divided amongst the people who violated the law ? So this makes legal culpability in these sorts of cases a murky area. If you slam your car into a kindergarten to protect yourself, though, I think it's reasonable to expect prison time and a hefty financial problem.
But the answer seems to be : if there is any way to avoid the truck, ignoring all normal traffic laws (but NOT hitting anyone), that's the way to go. If there's not, you hit the truck.
> Who in their right mind would get inside a car that is programmed to commit suicide in certain "ethical" circumstances?
Maybe you'll find this version more interesting: suppose the car has the opportunity to save a dozen people (who are not themselves at fault) by sacrificing you (or failing to do everything in its power to protect you). Should it?
If no, which seems to be the answer you think is obvious, consider that you are far more likely to find yourself as one of the dozen potential people saved than as the sacrifice, especially since we're deciding on the rule for all autonomous cars, not just yours. So following your logic makes it more likely for you to get killed.
The example is illustrative. There is nothing stupid about the original question. Someone else's fault and self-preservation are just convenient escapes in this situation. Suppose instead of a child running in front of the car, it was a large rock. Going straight will kill you. Swerving left will kill a mom and two kids, going right will kill an old trucker. What does your car do?
By the 'fault' logic, it was nobody else's fault that the rock landed in front of you, why should they die so you can live?
Going right and slamming your semi into the mini-Cooper means you live without a scratch, but good luck to the couple in the car. Going left and hitting the other truck there will probably make your truck flip and give you broken bones, but nobody dies. What now?
It's easy to answer, and quite frankly, the opposite of your suggestion - you swerve away from the person in the road. There's 2000 kg of car to protect you whereas there's nothing to protect the person in the road against a 2000 kg missile coming at them.
> Who in their right mind would get inside a car that is programmed to commit suicide in certain "ethical" circumstances?
What is the practical difference between that and getting into any car operating in an environment with other cars (automated or not) that are "programmed" (whether literally, or as a description of the operator's behavior) to kill you instead of their operator if there is a choice?
If you are in a car, about the only way another car could kill you is by crashing into your car. And two cars that are both programmed to save the operator over others are not going to evaluate their surroundings and both decide that the best action to save their operator is to crash into each other... unless there is literally no other option. And if that is the case, then the crash was going to happen with or without autonomy or questionable ethics.
Maybe we'll get ethical settings in our robots so, like a human driver today, we get to decide if we'll kill the kid or kill ourselves. But the problem with letting user have too many options is that we might end up at a less than optimal global outcome due to prisoner's dilemma-type choices. There could be more people killed, worse traffic, worse energy efficiency, etc. because users choose the selfishly beneficial options.
It is a stupid question, but not even because people, even children, should be responsible of their acts, but because a driver should expect such occurences.
This is the reason why speed limits are very low in residential and school areas: because it is expected that there are people on the side walks that _could_ possibly change direction, move on the road, and even perhaps stop (fall) there.
Similarly, when driving in the mountains or in the Arizona desert, you might go fast when there are nobody around, but if you see somebody walking on the side, you must take into account his possible changes of dirrection, and therefore slow down. Around highways, there are often fences to prevent people to come close, so that cars may go fast. Also, you don't go as fast when there is fog than when the sky is clear: you have to expect more unknowns when there is fog than when it's clear.
Now, I'm not saying that human drivers apply speed limits (either legal or moral (ie. "bayesian")) often or at all, but they should.
I'm saying that a robotic car will have much less difficulties in controlling scientifically its speed, taking into account the presence of people, animals and other mobile objects surrounding it, their velocities and their possible direction changes. Therefore a "tunnel" situation as described in that articule won't occur (unless there's a bug in the software).
Unless instructed to by me, my robot driver should never leave the road unless it is to save me, not kill me. The decision should be to drive the car into the area that causes the occupants the least harm.
I've never understood this logic, especially since you're deciding on the rule for all cars, not just your car. Shouldn't you prefer that the rule be to minimize the number of people harmed, since this is the option which (when taken over all cars) will minimize the likelihood of you being harmed?
Yeah, sure, but now suppose that you're going to decide the rule that everyone's car will follow. Are you still going to choose what amounts to defecting in the prisoner's dilemma?
Put it another way: this is one of those very rare occasion where collective cooperation (a la prisoner's dilemma) is possible. Shouldn't that... be the thing we want?
I would not be deciding that rule for everyone's car. I would provide a checkbox (or what ever way there would be to customize a car's options). I'm a pretty big believer in user settings. I wish more software gave users more options on some of the stuff some random developer thought was best.
The thing is, in the absence of collective action people choose to defect in prisoner's dilemmas. Giving people the option results in everyone defecting. Having it be preset results in everyone cooperating. Everyone cooperating is a better outcome for everyone.
Will autonomous cars prevent me from choosing the radio station or the cabin temperature? No. Why should they prevent me from choosing if I live or die?
Autonomous cars are supposed to be more intelligent, and capable of making better decisions than a human driver, based on access to more information and having more accurate controls. The entire argument about autonomous cars improving safety is predicated on the assumption that human drivers, in general, are simply not capable of driving safely.
You being able to make a decision about the way your autonomous vehicle behaves which could result in a greater risk to other drivers is no different than the risk you present anyway just by having access to a steering wheel, and making the equivalent decision directly. Changing the radio isn't going to kill anybody, but letting you decide the ethics of your car just might. What happens when two cars with conflicting driver-set ethical subroutines come into contact? Do they debate? It doesn't work unless there's a single standard.
An autonomous car is still going to have a finite number of actions it can take. Using the info it gathers on the fly, it will traverse a pre-programmed decision tree to arrive at the action(s) it should take. Being a computer, of course it will do this faster and use a lot more information than a human would. This is going to result in "driving safely."
I'm not saying we should be able to rewrite the code. But a single user option like "Should I sacrifice your life to save another? [Yes/No]" just means the decision tree has another branch point that still leads to the same set of possible actions. It can still make better decisions about driving than the human driver because it still has the access to more information and more accurate controls. Deciding to crash me into a tunnel wall to save a child is not really about "driving safely."
Plus the law has made that choice for you. Sacrificing another life to save your own, intentionally, is second-degree murder (first if answering this question means you planned it in advance).
If you answer yes to this question, aside from being a scumbag, if it does actually lead to an action taken by the car, you could face execution (worst possible case, granted, but there's a minimum prison sentence too).
Plus, this would hardly be the first way for cars to kill you. Brakes can fail, the steering column can fail, the gas tank can fail. All these things are rare, but certainly not impossible, and quite likely to kill or at least injure you.
Looking at Issac Amimov's laws, I believe that the 0th law would cause the child to die because that outcome reasonably reduces the number of humans potentially harmed in the situation (as driving into the wall may cause unpredictable results including additional harm to humans).
Looking at the EPRSC / AHRC principles of robotics, I believe the child would die because robots are artifacts and should not invoke an emotional response.
It might be worth mentioning that Asimov's Three Laws were plot devices meant to be broken by robot antagonists for the sake of drama, and that actually taking them seriously as a framework for AI ethics is probably a fool's errand.
This debate has come up a few times now and I doubt it reflects the reality of what responses to such situations can be practically expressed in code using today's computers and languages anyway (in the absence of some AI that can make 'ethical' decisions).
I suspect the best collision avoidance algorithm in such scenarios will simply be decelerate as quickly as possible whilst maintaining course. Then a 'best effort' at avoiding a collision can be seen to have been made, and perhaps the pedestrian will survive their injuries. This is what a human driver would have most likely done?
What if the child/small adult (can the computer vision practically recognise the difference) is themselves quite 'evil' (again, another value judgement) and is running in front of robot cars in such situations on purpose?
It's also a bit of a red haring. While it could happen in very rare instances, the whole benefit of the automated car is that the number accidents will drop off a cliff. When nearly every car on the road is automated then there simply won't be many accidents any more.
You're assuming the best possible implementation of autonomous car will be the one which actually arrives on the roads. Given the daunting complexity necessary to even make that feasible,it would surely be the first system of its kind and scale without a considerable number of flaws due to bad programming, exploits, cost cutting, unforseen consequences, etc.
The more 'intelligent' and 'autonomous' the cars are, and the more of them there are on the road, the more likely it is systemic problems will arise. This is not a criticism of the idea of autonomous cars but just an inevitable result of complex systems interacting. Just because they're computers, doesn't mean they won't fail, or that accidents will be so rare as to be not worth considering. Airplanes almost never crash now compared to a century ago, but that doesn't make worrying about air safety, or the affect of automation on pilot competence in the event that manual control becomes necessary, a red herring.
There's only two systemic problems in the current road system that I know of: traffic jams, and the fact that increasing road capacity increases road traffic (and therefore doesn't reduce the potential for traffic jams). They're entirely due to the behavior of human drivers (given the surrounding circumstances in part too, of course).
Are other systemic problematic effects known in the current road transportation system? Notably, are there any systemic effects that lead to increasing speed and accidents, rather than traffic jams (not that traffic jams don't cause accidents, since human drivers are running too fast, not expecting jams).
It seems obvious that the two systemic problems above, won't occur with intercommunicating robotic cars.
It would be "fun" to discover new systemic epiphenomenon with robotic cars, but I would expect them to be more discreet and even less "catastrophic".
It seems obvious that the second systemic problem (increasing road capacity increases traffic) is a result of the effect of road capacity on the decision to take trips using a car, rather than the way people drive, and will therefore continue with intercommunicating robotic cars.
You also have to consider potential problems with the cars themselves, in terms of engineering and software, as well as potential issues with networking, weather, inaccurate satellite data, etc. It would be a vastly more interconnected and complex system than the one which already exists.
But you're also seemingly forgetting that cars these days are very safe, with airbags, seat belts, crumple zones, roll bars... this whole 'should it kill you' is not likely an issue these days. Look at all those idiots who crashed their cars going 80mph and lived to tell about it (tesla for example). The point being that if a car controlled the crash, your chances of survival are far better. So it's an option of killing someone or having a controlled crash, not 'kill or be killed'.
> Allowing designers to pick the outcome of tunnel-like problems treats those dilemmas as if they must have a “right” answer that can be selected and applied in all similar situations [...] it becomes clear that there are certain deeply personal moral questions that will arise with autonomous cars that ought to be answered by drivers.
The answer is not as clear-cut as the example would suggest. First, it's not zero-sum since computer control results in more safety overall. The computer has more information available (e.g. sensor readings on all sides) and can react much faster (eliminating the ~1.5 s typical reaction time [1]). The driver and the kid in the road will both survive more often this way.
Second, drivers can be pretty selfish. E.g. everyday behaviors like cutting people off or running red lights offer little benefit for the cost of endangering everyone on the road. How many would simply set it to "always drive over the kid in the tunnel"?
Yeah, drivers are selfish; humans are frequently selfish.
For example, if a truck driver has the option of hitting another car head-on, or running off the road and just wreaking the truck, they will frequently hit the car.
The reason being, if there's no evidence of a second vehicle in the accident, they will end up being responsible for the loss of the cargo, and lose their job.
Plus, the truck driver will typically survive a head-on with a car.
I'd say that in the example given in the article, the best decision would be to slam into the side of the tunnel instead of hitting the child. It makes no difference the age of the child, since if they are laying in the road, there's a high likelihood that they will be killed and moderate likelihood that you' be deflected enough to be seriously injured or killed as well by choosing the 'through the living being route' since that particular case is fraught with uncertainty, the most logical choice is a controlled crash. Since cars these days have airbags and crumple zones and such things, it's hard to hit an inanimate object and get killed in such a vehicle. And with car insurance, and unlawful death lawsuits in practice, it seems the financially responsible choice is to have a controlled crash. So to all those saying there is no 'right' answer, you are generalizing the problem too much. The question as stated in the article, not generalized to all situations, but to the situation as it was stated, seems clear.
46 comments
[ 3.2 ms ] story [ 62.8 ms ] threadWhy should you have to pay for someone else's mistake? If a child runs out in front of a car, it is either their fault or their guardian's fault for allowing them to run into traffic. It's tragic, but to suggest the driver should give up their life for someone else's mistake is ridiculous.
Who in their right mind would get inside a car that is programmed to commit suicide in certain "ethical" circumstances?
I suspect there are various classes of people that would choose this option. Believers of certain religions, old/hill people with high morals or strictly pragmatic, aspirant suicides, risk lovers, etc...
Not for me, though :-)
This seems to assume that the frequency with which drivers (automated or otherwise) are faced with "someone is going to die, you or the other person, and which one depends on your driving choices" situations is very high.
This is not consistent with the evidence.
For example, the self-driving car you're in is driving by an elementary school at the end of the day, so there are a number of children on the sidewalk. Suddenly, a truck appears coming at your car going in the wrong direction. Should your car take an evasive maneuver onto the sidewalk, with a high likelihood of striking a child?
In traffic law people actually thought about this. First you are to safeguard pedestrians, then cyclists and motorcyclists then cars, then trucks/buses/... and finally yourself. Things like right side of the road, red lights, ... are far down. Things like trains/trams are tricky.
What is not generally understood is that this sequence matters. Most people realize that in order not to block an ambulance, you are not just allowed, but required to run a red light. Less known is that ignoring road safety, to some extent, is legal and required if you're bringing a seriously injured person to the hospital. If you can avoid a collision by crossing to the wrong side of the street, you are required to do that ...
So the legal answer here is simple and well-defined : Assuming you can't move to the other side of the road reasonably safely, you should slam your car into the truck (needless to say, as slowly as possible). You are certainly NOT to take liberties with the safety of pedestrians to avoid hitting a truck.
This is complicated by other laws of course. First you are not expected to do more than a certain "standard" human would do, second in this situation obviously there is a problem of intent when things do go south (did you intend to hit those pedestrians ? What about if you made that choice to avoid the truck ? What if you and the truck both decided to avoid each other and both went for the b). Who is the "root" responsible, and how is the responsibility divided amongst the people who violated the law ? So this makes legal culpability in these sorts of cases a murky area. If you slam your car into a kindergarten to protect yourself, though, I think it's reasonable to expect prison time and a hefty financial problem.
But the answer seems to be : if there is any way to avoid the truck, ignoring all normal traffic laws (but NOT hitting anyone), that's the way to go. If there's not, you hit the truck.
Maybe you'll find this version more interesting: suppose the car has the opportunity to save a dozen people (who are not themselves at fault) by sacrificing you (or failing to do everything in its power to protect you). Should it?
If no, which seems to be the answer you think is obvious, consider that you are far more likely to find yourself as one of the dozen potential people saved than as the sacrifice, especially since we're deciding on the rule for all autonomous cars, not just yours. So following your logic makes it more likely for you to get killed.
By the 'fault' logic, it was nobody else's fault that the rock landed in front of you, why should they die so you can live?
Going right and slamming your semi into the mini-Cooper means you live without a scratch, but good luck to the couple in the car. Going left and hitting the other truck there will probably make your truck flip and give you broken bones, but nobody dies. What now?
What is the practical difference between that and getting into any car operating in an environment with other cars (automated or not) that are "programmed" (whether literally, or as a description of the operator's behavior) to kill you instead of their operator if there is a choice?
This is the reason why speed limits are very low in residential and school areas: because it is expected that there are people on the side walks that _could_ possibly change direction, move on the road, and even perhaps stop (fall) there.
Similarly, when driving in the mountains or in the Arizona desert, you might go fast when there are nobody around, but if you see somebody walking on the side, you must take into account his possible changes of dirrection, and therefore slow down. Around highways, there are often fences to prevent people to come close, so that cars may go fast. Also, you don't go as fast when there is fog than when the sky is clear: you have to expect more unknowns when there is fog than when it's clear.
Now, I'm not saying that human drivers apply speed limits (either legal or moral (ie. "bayesian")) often or at all, but they should.
I'm saying that a robotic car will have much less difficulties in controlling scientifically its speed, taking into account the presence of people, animals and other mobile objects surrounding it, their velocities and their possible direction changes. Therefore a "tunnel" situation as described in that articule won't occur (unless there's a bug in the software).
Put it another way: this is one of those very rare occasion where collective cooperation (a la prisoner's dilemma) is possible. Shouldn't that... be the thing we want?
You being able to make a decision about the way your autonomous vehicle behaves which could result in a greater risk to other drivers is no different than the risk you present anyway just by having access to a steering wheel, and making the equivalent decision directly. Changing the radio isn't going to kill anybody, but letting you decide the ethics of your car just might. What happens when two cars with conflicting driver-set ethical subroutines come into contact? Do they debate? It doesn't work unless there's a single standard.
I'm not saying we should be able to rewrite the code. But a single user option like "Should I sacrifice your life to save another? [Yes/No]" just means the decision tree has another branch point that still leads to the same set of possible actions. It can still make better decisions about driving than the human driver because it still has the access to more information and more accurate controls. Deciding to crash me into a tunnel wall to save a child is not really about "driving safely."
If you answer yes to this question, aside from being a scumbag, if it does actually lead to an action taken by the car, you could face execution (worst possible case, granted, but there's a minimum prison sentence too).
Plus, this would hardly be the first way for cars to kill you. Brakes can fail, the steering column can fail, the gas tank can fail. All these things are rare, but certainly not impossible, and quite likely to kill or at least injure you.
Looking at Issac Amimov's laws, I believe that the 0th law would cause the child to die because that outcome reasonably reduces the number of humans potentially harmed in the situation (as driving into the wall may cause unpredictable results including additional harm to humans).
Looking at the EPRSC / AHRC principles of robotics, I believe the child would die because robots are artifacts and should not invoke an emotional response.
See: http://en.m.wikipedia.org/wiki/Laws_of_robotics
I suspect the best collision avoidance algorithm in such scenarios will simply be decelerate as quickly as possible whilst maintaining course. Then a 'best effort' at avoiding a collision can be seen to have been made, and perhaps the pedestrian will survive their injuries. This is what a human driver would have most likely done?
What if the child/small adult (can the computer vision practically recognise the difference) is themselves quite 'evil' (again, another value judgement) and is running in front of robot cars in such situations on purpose?
You're assuming the best possible implementation of autonomous car will be the one which actually arrives on the roads. Given the daunting complexity necessary to even make that feasible,it would surely be the first system of its kind and scale without a considerable number of flaws due to bad programming, exploits, cost cutting, unforseen consequences, etc.
The more 'intelligent' and 'autonomous' the cars are, and the more of them there are on the road, the more likely it is systemic problems will arise. This is not a criticism of the idea of autonomous cars but just an inevitable result of complex systems interacting. Just because they're computers, doesn't mean they won't fail, or that accidents will be so rare as to be not worth considering. Airplanes almost never crash now compared to a century ago, but that doesn't make worrying about air safety, or the affect of automation on pilot competence in the event that manual control becomes necessary, a red herring.
Are other systemic problematic effects known in the current road transportation system? Notably, are there any systemic effects that lead to increasing speed and accidents, rather than traffic jams (not that traffic jams don't cause accidents, since human drivers are running too fast, not expecting jams).
It seems obvious that the two systemic problems above, won't occur with intercommunicating robotic cars.
It would be "fun" to discover new systemic epiphenomenon with robotic cars, but I would expect them to be more discreet and even less "catastrophic".
The answer is not as clear-cut as the example would suggest. First, it's not zero-sum since computer control results in more safety overall. The computer has more information available (e.g. sensor readings on all sides) and can react much faster (eliminating the ~1.5 s typical reaction time [1]). The driver and the kid in the road will both survive more often this way.
Second, drivers can be pretty selfish. E.g. everyday behaviors like cutting people off or running red lights offer little benefit for the cost of endangering everyone on the road. How many would simply set it to "always drive over the kid in the tunnel"?
[1] http://www.tandfonline.com/doi/abs/10.1207/STHF0203_1#.U9wU2...
For example, if a truck driver has the option of hitting another car head-on, or running off the road and just wreaking the truck, they will frequently hit the car.
The reason being, if there's no evidence of a second vehicle in the accident, they will end up being responsible for the loss of the cargo, and lose their job.
Plus, the truck driver will typically survive a head-on with a car.