> What we know is that the vehicle was on a divided highway with Autopilot engaged when a tractor trailer drove across the highway perpendicular to the Model S. Neither Autopilot nor the driver noticed the white side of the tractor trailer against a brightly lit sky, so the brake was not applied.
Tragic no doubt, but I'm relieved that this was not a "Autopilot did something very very wrong" story.
Autopilot has the potential to save a large number of lives (I'm sure Tesla execs are thinking about touting "estimated lives saved by autopilot" if the numbers work out, after a few billion miles), so I hope incidents like this don't hamper public perception and therefore research.
That doesn't sound like what happened. "The driver didn't notice" well, that's why they're using autopilot.
No matter how many messages people throw at users, any autopilot that doesn't detect that the driver is awake and operating the vehicle is going to be used by sleeping or otherwise unattentive drivers. It's just human nature.
So what likely happened was the driver was screwing around and the autopilot completely failed to notice a giant white truck crossing the road and blocking their way until it was too late.
edit: reply is quite right, shouldn't speak ill of the dead that way. "Screwing around" is unfair. I just mean that "driver falls asleep/paying attention to other things" is the expected outcome of any autopilot system, and so it's unreasonable of autopilots to have any expectation of drivers for attentiveness.
From the information given, we don't know if the truck pulled out right in front of the car giving the driver no time to react, or if the truck pulled out a minute beforehand and the driver was just oblivious. In the absence of any information either way, it's nasty to just assume the driver was inattentive.
I don't think it's nasty, I think it's realistic. I also think that it aligns with the research that shows humans take a significant amount of time to adjust to taking back over control from autopilot. This is one of the primary design considerations behind the Google self driving car having absolutely no steering wheel or brake. The illusion of humans being able to be the fallback when they're not physically capable of it is a dangerous situation.
I don't think it's realistic to rampantly speculate about a situation where so little information is available. And I absolutely think it is nasty to talk ill of the dead when there is zero evidence for what's being said.
For all we know the driver had half a second to react before he died. Or maybe he had five minutes. The point is, we don't know. To start from that position and say it's "likely... the driver was screwing around" is pretty crappy.
Fair, I wasn't reading it as a knock on the driver. I read it as a mistake for Tesla to hide behind the driver having ultimate responsibility when the reality is no driver is going to actually be attentive or even capable of taking over.
I'll admit that the word "screwing around" was unfair and judgemental, but I'm gonna be honest: I would never want to own a car with this sort of "autopilot with driver fallback" feature because I'd totally fall asleep at the wheel. Completely. So I didn't mean any judgment by it. I could have made the same mistake.
In fact, I'm judging the software, if anything. Any autopilot software that doesn't force the driver to be attentive is effectively taking the responsibility for driving onto themselves, and so cannot blame the driver for being inattentive. You can't say "I got this" and then at the last second say "oh shit, I need you driver!" even if you put warnings that you don't really got this, because you're dealing with human beings who do normal human things.
Yes, I shouldn't have said the driver was screwing around, you're right. But if the driver fell asleep, and a conscious driver would have avoided this accident? I still think it's Tesla's fault, because he'd be awake and alive in a manual car.
And how many times do human drivers fall asleep in cars with no automatic driving features of any kind, and get themselves killed in a situation where Tesla's Autopilot would have saved them?
I'd recommend trying it before you knock it. I had similar reservations, but after using it for nearly a year, I find that I'm more attentive and more awake with it than without it. A couple of weeks ago the system refused to engage (broken sensor somewhere) and I had to drive 80 miles home manually. It was exhausting! The mental effort of staying between the lines and maintaining a safe distance from the next car is pretty significant. Getting rid of that frees up more of your attention for the bigger picture. Now, I'm sure for some people it just frees up more of your attention for texting or sleeping, I'm sure I'm not representative of everybody, but I think it can easily be a net win for many.
I always pay attention while Autopilot is engaged, but I'm able to look farther down the road and take more care when checking out traffic to the rear and sides than I can when I'm doing it all myself.
>>This is one of the primary design considerations behind the Google self driving car having absolutely no steering wheel or brake.
Didn't know about this. If they are truly testing their software in a car without break and steering wheel I have a whole new level of respect for them.
When you start to design a self-driving car the question of a human fallback shouldn't even exist. The car either drives itself, or humans do. Mixing responsibilities in situations of life and death doesn't end well.
A car traveling 60mph is traveling 88 feet per second - double this because the truck is likely traveling the opposite direction at the same speed. Average human reaction time in best case scenarios is ~215ms (see http://www.humanbenchmark.com/tests/reactiontime).
Assuming the driver is perfectly attentive, and hits the brake immediately upon noticing the truck, they would still need at least 139.84 feet to come to a stop.
This also assumes the truck is no longer moving. If it is you'd have even less distance to stop.
I don't care whether you're waking up from sleep, or completely alert and instantly react, it's a huge assumption to think that any possible reaction would have prevented an accident or fatality. It's just as likely that even had autopilot engaged the brake immediately, the truck still would have plowed into the Tesla and caused a fatality.
tl;dr - two way traffic involved in a head on collision is a dangerous game, made doubly dangerous because of the combined speed of both vehicles working against inertia.
From the article, it seems that this autopilot checks that the driver is attentive.
> When drivers activate Autopilot, the acknowledgment box explains, among other things, that Autopilot “is an assist feature that requires you to keep your hands on the steering wheel at all times," and that "you need to maintain control and responsibility for your vehicle” while using it. Additionally, every time that Autopilot is engaged, the car reminds the driver to “Always keep your hands on the wheel. Be prepared to take over at any time.” The system also makes frequent checks to ensure that the driver's hands remain on the wheel and provides visual and audible alerts if hands-on is not detected. It then gradually slows down the car until hands-on is detected again.
It does but it's not terribly frequent and it's easy to bypass. When you give people a feature they're going to use it however they want regardless of what you tell them. Users are funny like that :)
Those "frequent checks" are pretty rare. I've gone 15-20 minutes without seeing one. For a while it was every 3-5 minutes, but they've dialed it back up in newer software updates.
And yet it really doesn't. Every so often it'll beep at you to make your presence known, but it can easily go many minutes with hands off the wheel and no checks.
I'd say most accidents are the result of the driver "not noticing", especially in the current majority of cars which lack autopilot, so even if this edge case exists in the future, having an autopilot is still an improvement.
Everything is speculation at this point so I am not sure how you can assume that's not what happened.
> "The driver didn't notice" well, that's why they're using autopilot.
If Tesla builds and designs a system to be used as an assistant, explicitly tells and warns the user of it's use case and potential danger of using it as anything other than an assistant, then the excuse that "it's human nature" is null, I don't think it's human nature to leave your safety and well being at the hands of something designed to be nothing more than an assistant.
Perhaps the name autopilot exerts to much confidence in its capabilities.
How would we ever know that the autopilot acted correctly? Should companies be required to release logs after every crash when a car is in autopilot? Seems fair.
Autopilot engaged when a tractor trailer drove across the highway perpendicular to the Model S. Neither Autopilot nor the driver noticed the white side of the tractor trailer against a brightly lit sky, so the brake was not applied.
How is this not an "Autopilot did something very very wrong" story?
Tesla autopilot drove the vehicle full-speed into a tractor-trailer, which wasn't registered by its sensor systems. It's great that Tesla already has an explanation for the failure, but the situation as described is far from an improbable edge case. What they are saying is that the autopilot can't detect white objects when a bright light is present -- that's a pretty serious limitation to overlook.
Human drivers do this kind of thing with great frequency. As long as the system is at least as good as a human driver at this stuff, I'm happy with it. Actually, scratch that: as long as the system combined with me is at least as good as a human driver, I'm happy with it.
It's serious enough to warrant its own road sign, but its not as much of a realistic issue on a divided highway, so maybe this road didn't have such a sign.
This type of danger is probably what the article is describing.
I guess the question is, was the driver able to see it? I can see how maybe the contrast was so low that maybe it was impossible to see anything except for the wheels which you might not notice. And if the part of the tractor trailer in the path is elevated... sensors sweeping forward see just road.
I guess I'd like to see an NHTSB simulation of the situation in manual driving to see how humans do unassisted.
And if the part of the tractor trailer in the path is elevated... sensors sweeping forward see just road.
It could be even worse. If there was a "road surface mirage" then it's possible that both the truck and the road underneath it were not visible at all. Since the forward radar doesn't have the range or coverage to see a truck at a distance, the only sensor that could have detected the problem was the camera. Because of the phenomena I've mentioned above and the parent post has mentioned, we know this can't possibly work.
Even worse, the circumstances in which the car's sensor could fail are precisely the ones in which the Mark 1 Eyeball would also fail.
Replace "autopilot" with "cruise control" -- would you have the same situation? Autopilot is basically advanced cruise control plus steering assist. From what I make of this story, this same accident would have happened with any other car with cruise control turned on.
A lot of modern cars have cruise control with radar for keeping distance to the car in front. Some use the same sensor to auto brake for obstacles at city speeds. I'm sure some will also alert for obstacles at highway speeds, but likely don't auto break then, for safety reasons.
I disagree. In a car with normal or even advanced cruise control, the driver would still be paying attention to the road and ostensibly would have seen the truck ahead and applied the brakes. Autopilot makes it very easy to not pay attention at all and let the car drive itself for minutes at a time.
I'm sure you'll say, "yes but that's a driver attention problem, not a technology problem", but to me they're the same thing. If you give drivers technology that encourages them to not pay attention then accidents caused as a result of that are partially your fault.
No, it's not the same at all. With cruise control you are still fully in control of your car and necessarily aware of how fast you are going and what is going on in your surroundings. With Autopilot that is simply not the case - the technology virtually begs you to set it and then play with your phone or the touchscreen instead of paying attention to the road, text warnings be damned.
If you want an actually applicable analogy, I'll pull one I've used before. Say you buy an oven that has this great auto-baking feature that lets you just put some cookie dough inside it and then come back whenever you want, even hours later, to have fresh baked cookies waiting. That's how it's marketed - the AutoBake Oven. Except the manufacturer says that you're still supposed to watch the oven at all times in case something goes wrong. So you make some cookies with it and they turn out great, and nothing goes wrong. And hey, it's the AutoBake Oven, right? It says it will bake things automatically so you don't have to worry about them for hours at a time. So you logically decide to put some dough in there and then go the grocery store for a few minutes to get some items for dinner. And you come back and your house burned down because the AutoBake Oven lit itself on fire due to having subpar sensors and beta software in it. Now tell me you wouldn't file a lawsuit against the manufacturer.
Or maybe it's just a human behavior problem, and unfortunately it may take a few accidents of this type to train ourselves that AI assist doesn't mean we can lessen our attention.
Especially after hearing about incidents like this, you won't find me going through my email while letting "autopilot" do its thing. I don't think I would before this, but there's certainly a lower probability that I would now. I'm being trained.
The autopilot doesn't drive the car, the driver does. The autopilot is a help just like cruise control. Tesla are very clear (apart from the naming of the feature) that the driver must still drive the car.
The autopilot will avoid obstacles it can detect and the driver is expected to avoid others (since he has eyes on the wheel and eyes on the road the whole time).
I think they misleading thing about Teslas Autopilot I its current state is that they call it "Autopilot" while other manufacturers call this kind of assist features e.g "Lane Assist".
A distance-sensing cruise control can't be trusted not to crash into the back of the vehicle in front either, not least because it can't be sure it can "see" it. I suspect motorcycles and other small vehicles may fool radars just like this case seems to have been about contrast and colors in a camera sensor.
>The autopilot will avoid obstacles it can detect and the driver is expected to avoid others
I'm sorry, how is that supposed to work, exactly? How does the driver know? Sounds like the driver has to avoid every obstacle - to avoid the case where the autopilot fucks you, like this one - and in that case, what is the autopilot doing?
The driver has to be aware of every obstacle and observe the autopilot avoid them, or intervene should it fail. Yes.
Have you driven a car with a distance-keeping cruise control? It's very comfortable to let it do that and just watch it and take over when it fails (it's limited in the braking and acceleration it is allowed to do). I can't take my eyes of the road or my hands off the wheel for even a second but it's still useful because it removes all the micromanagement of throttle and brake and just lets me supervise.
That is a massive assumption to be making. That the driver can see and react to every obstacle. Not to mention there are videos where the Tesla reacted before and on behalf of the driver, saving them from an accident.
The "lane assist" (what it should be and is called by other manufacturers) spots any number of obstacles that the driver failed to see - takes over - and saves the driver from an accident. Or reacts faster than the driver could react - thus saving the driver. That is what it is doing. It is not driving your vehicle. It is assisting in driving your vehicle.
It is still possible to drown with a life vest on. That doesn't mean you shouldn't wear one because "if I'm expected to stay above water - what is the life vest doing?"
That's some PR firm-level explaining. First, autopilot is not just lane assist, it's lane assist + adaptive cruise control + additional features (it can change lanes if you tell it to). Other OEMs have purposely shied away from enabling an autopilot-like feature set together, even though it's technically possible, because they are absolutely terrified of something exactly like this situation happening.
Second, you're completely ignoring that autopilot will very much entirely drive the car by itself for minutes at a time (reports are anywhere from 5 to 15 depending on software version), which will obviously encourage drivers to not pay attention to the road. I don't care about disclaimers, I don't care about nag screens or chimes, and I definitely don't care about some warranty text that flashes on the screen. If you make a car that can drive itself you encourage drivers to let it do just that and you should be prepared for that eventuality. Period.
Very new and fairly experimental technology that can kill you tends to be fairly heavily regulated. New drugs, rockets, aircraft, etc. don't generally get released via software update to the general public. They go through very regimented, restricted, contracted trials.
Tesla risks getting regulators to crack down on new car features across the entire industry if they're not careful.
Much like if you make a car that does 0-60mph in 3 seconds or does 200mph you encourage doing that. The fact that the autopilot is so good that it "almost" works as a driver is an issue for Tesla as we can see here. I realize autopilot has a nice ring to it, but it's not a chauffeur...
Tesla sure has balls to pick the name "Autopilot" when it really should be called "CoPilot" and that would have saved a lot of trouble.
The Drive article is an eerily prescient interview with Volvo's R&D chief and is only two weeks old:
Critics see a neither-nor limbo, with technology just
good enough to make drivers drop their guard, but not
good enough that driving off a cliff is an
impossibility. ... Shooting for but missing Level 3
autonomy would create his nightmare scenario: a car
that gets flummoxed by one of thousands of variables,
then asks a daydreaming co-pilot to wake up and save
the day.
> The autopilot doesn't drive the car, the driver does. The autopilot is a help just like cruise control. Tesla are very clear (apart from the naming of the feature) that the driver must still drive the car.
I'm sorry, but that's a cop-out. What's the point of an autopilot if you have to avoid every single obstacle and react to every single thing as if the auto-pilot didn't exist? You can't say "autopilot is safer than normal humans" (heavily implied by the blog post) and then turn around and say "but it's really the driver who's responsible for everything!" anytime there's an accident.
Unlike autopilot, cruise control has very limited functionality. No driver expects it to do anything other than keep their speed, so they are still actively engaged with the road to respond to anything that takes place.
> I'm sorry, but that's a cop-out. What's the point of an autopilot if you have to avoid every single obstacle and react to every single thing as if the auto-pilot didn't exist?
You don't have to react - you have to supervise it though. It lets the driver monitor a system that micromanages steering, brakes and throttle, instead of doing that micromanagement himself. It requires the same attention but is less work.
> Unlike autopilot, cruise control has very limited functionality. No driver expects it to do anything other than keep their speed,
Well, these days you can often set it with a distance to the car in front using a radar. You then let the cruise control do something more than just keep constant speed, it will slow down if the car in front does etc.
The driver has to pay full attention though for anything the cruise control can't handle.
You seem to imply "what's the point of an autopilot if I can't take a nap". - well the issue is that it's a little too good to be called a cruise control, but not good enough for a nap. We're simply not there quite yet. At least not legally.
The problem to me seems to be: at highway speeds, you have to decide very quickly whether or not it's something that the system can't handle.
If you're constantly responding to every change in conditions, what's the point of the autopilot? So you don't respond every time, and then something comes up that the autopilot doesn't recognize, and you don't know that until it's too late.
> If you're constantly responding to every change in conditions, what's the point of the autopilot?
The same as a cruise control - you are supervising rather than micromanaging because micromanagement is more work than supervision.
A distance-keeping cruisecontrol may be changing the throttle level dozens of times per minute. A driver supervising that system can keep an eye out for stalled cars on the side of the road, wildlife etc., withput using most of the attention on keeping a constant distance to the vehicle ahead. This complements the driver and increases safety because the drivers cognition can be used to what it's best at, and simple tasks like keeping a constant speed or constant distance can be done by a computer.
The problem with this "Level 2.5" autopilot is twofold:
1) It's called "AutoPilot", even though that shouldn't be the name of it until it's feature complete (level 4 autonomy) simply because people seem to assume that AutoPilot means something that drives by itself.
2) It appears good enough to allow the driver to not pay attention, while still requiring the driver to pay attention. The disclaimers that the driver can never be distracted for a short momet (i.e. use it like a pilot uses an autopilot) don't seem to help.
A lot of the comments here seem to indicate "if I can't text/sleep/whatever then what's the point of an AutoPilot" - clearly misunderstanding the current capability of the system!
I think we're in agreement. My point is that you can't supervise something you don't understand so I don't know how a driver can reasonably be expected to switch from supervision to reaction quickly enough every time that is needed.
I know what my cruise control will do (effectively nothing). I don't know how to anticipate what Tesla's sensors and "AI" can't handle.
Calling it 'autopilot' is a very bad thing, then, because 'autopilot' means 'runs automatically' to the general public.
And even though actual autopilots in planes are monitored by meatpilots, meatpilots can still take their attention away for a short time. The autopilot keeps the plane flying straight and level, and doesn't take on any navigational role (in most cases). Unless your roads are dead straight and there's no traffic, driving requires a constant navigational role.
> Calling it 'autopilot' is a very bad thing, then, because 'autopilot' means 'runs automatically' to the general public.
Indeed. Tesla wanted to use that name now even though it's the proper name for a feature not yet complete. They have to use a ton of warnings and disclaimers to do it. We'll see if authorities find that sufficient. If I were tesla I'd rename it to something with "assist" for the time being.
> And even though actual autopilots in planes are monitored by meatpilots, meatpilots can still take their attention away for a short time.
So does Teslas autopilot, but the time you can safeley take your attention away on a highway may be perhaps a second or two if you are on a clear straight with wildlife barriers. The autopilot will ensure the car runs straight while you fiddle with the radio.
The same time in an airplane may be ten minutes. THAT is the difference. In an airplane theare are controllers making sure there is no traffic one in front, and there is no risk of a deer or a child stepping in front of the plane. Basically the AutoPilot will have the meatpilot respond to anything it can't handle too - which in an airplane is mostly tecnical issues.
> Tesla autopilot drove the vehicle full-speed into a tractor-trailer
The tractor trailer drove across the divided highway without seeing an oncoming car. I've seen this accident happen in front of me with a fatality, no auto-pilot involved.
A vehicle moves in front of you when you're sufficiently close and you've no time to react. Anything (like glare) that makes such a vehicle harder to spot reduces your available reaction time.
Making it sound like there was a static obstacle in front of the car, and it drove into it at high speed is very tendentious, and not warranted by the report, I feel. It is exactly the kind of 'omg AI kills driver' fearbait that we don't need.
If self-driving systems were ten-times safer than people, then there'd still be a lot of people being killed in them. We need to get a sane perspective on their risk.
It is essentially AI kills driver. The truck was making a left turn in front of the Tesla, which was in the right lane of a 4 lane divided highway. The truck would have had to accelerate in the left turn lane, cross the divider, cross the left lane, then drive further so that the middle of the trailer was in the right lane. This takes at least 5-6 seconds for a semi, and if the semi were detected, there would have been ample time to apply the brakes to give the truck enough time to clear the intersection. If the autopilot system can't detect and respond to this situation appropriately, it shouldn't be able to be activated on a non-controlled access highway.
It's unrealistic to assume that human drivers are attentive all the time. Autopilot never gets tired, never eats, never applies makeup, never sneezes, and never gets distracted by screaming children.
You have to compare autopilot to realistic humans, not your platonic ideal of what a driver should be. I'm sick and tired of people making statements about road transit predicated on the idea that real humans pay attention to the road with 100% fidelity at all times.
I read it more closely and they do try to enforce that a driver has his hands on the wheel, etc. I am not convinced of the benefits though. I think something that enforces people to pay attention when driving is a better solution.
> It's unrealistic to assume that human drivers are attentive all the time...
Yes, this is an unfortunate fact of life that human beings are no angels.
However, most modern societies agree that reckless drivers are to be held accountable for their less than perfect behavior. If a police officer (or more recently, surveillance camera) catches you while DUI, or texting, or some other of the most blatant irresponsible behaviors, they will give you a big ticket. And if the consequences of your own idiocity happen to materialize you will suffer much sterner punishment (up to criminal charges for manslaughter, if it came down to that).
That's assuming the infractor does survive. It is a known fact that a non trivial amount of people responsible for accidents end up loosing their lives (or those of loved ones) in the very tragedies they caused themselves. Death is the ultimate judge, and a merciless one at that.
> You have to compare autopilot to realistic humans, not your platonic ideal of what a driver should be.
Why?
Neither the engineers who created those subpar systems - nor the executives who pushed for an unfeasible schedule, nor the shareholders that benefited from the whole deal, - will ever face the same consequences as described above. Corporations raison-d'etre is precisely to shield people engaging in economic activities from the unintended consequences of their actions; that's why they must be held to a higher standard than the everyman down the road.
Would you imagine Tesla's Board of Directors having approved the green light for this impressive-yet-still-limited technology, if the company was required to pay for every traffic ticket that every Model S accrued while being in autopilot mode, nationwide? What about being accountable for every accident, no disclaimer licensing or such nonsense?
> I'm sick and tired of people making statements about road transit predicated on the idea that real humans pay attention to the road with 100% fidelity at all times.
People will make statements about road transit, because it is literally a life-or-death situation. Maybe you do prefer the cattle to shut up and die quietly, so your utopic techno-fantasies are never challenged?
I prefer to talk about real utilitarian tradeoffs instead of putting I imaginary, arbitrarily virtuous angels against computers. Every day you delay automation of automobile traffic is another batch of lives lost.
Are you more interested in saving lives or in expressing the short sighted and sanctimonious outrage of a Luddite?
Since when is caring about Engineering standards became Luddite?
How many lives do you think autopilot is going to save if being as good as a sleep deprived teenager is good enough? And who benefits from that state of affairs?
Anyways, you are a true beliver or a paid shill, so have a good night. EOM.
It's also unrealistic to assume that an attentive driver would have avoided the crash.
Driving at speed, you have to make assumptions that other people will do the right thing. You pass cars all the time. If you assume that the car might side-swipe you, you wouldn't get far, and you'd be a hazard yourself.
You drive on undivided highways all the time without driving as if the oncoming car might cross in front of you. And so on. We have to assume that others are minimally competent. That's the point of road laws, not just to control you, but to let you assume things about others.
Where I'm from if you make a left turn and get side-swiped then it's your fault, period. You made an unsafe turn, plain and simple[1].
If what you describe is true, then human error is the main cause of the accident (although the AI failed to prevent it and absolutely should share the blame). I just think "AI kills driver" is a bit sensationalist don't you?
[1]Of course assuming that all other vehicles present were obeying the law (e.g. not speeding) and there were no other extenuating circumstances.
The only situation I can imagine a driver not noticing a huge truck perpendicular to them on the highway, regardless of what color it is, is that they're not paying any attention at all.
This peculiar wording is making it sound like the computer made the same mistake as the driver, but it seems much more likely that they made two different mistakes. The computer's mistake was in image recognition, and the person's mistake was trusting the autopilot fully and not paying attention to the road.
The only situation I can imagine a driver not noticing a huge truck perpendicular to them on the highway, regardless of what color it is, is that they're not paying any attention at all.
The optical distortions that cause road surface mirages could well entirely hide another vehicle. In this case, since the trailer was painted white, only the wheels and lower chassis would need to be hidden for the image to become unrecognizable. Not saying that's what happened in this case, but there are situations when your forward vision could be compromised. (There is a theory that the colder ocean may have caused a similar optical phenomenon and hidden the iceberg that sunk the Titanic.)
This seems like a pretty big oversight, large object that car is too tall to clear safely seems like it could cause a lot of problems. Would the car try to make a lane change while next to a trailer?
Tesla's Autopilot only changes lanes when commanded to by the driver. This is done by putting the turn signal on. If you put the turn signal on while next to a trailer that doesn't have stuff coming down near the ground, then yes, the car will likely change lanes right into it. But it's easy to avoid: just look before you signal.
Unless it's changed recently, drivers have to tell the car specifically to change lanes (using the turn signal) or else it won't do so. And yes, this is the same issue as that other fairly recent incident in which a Model S drove itself into a parked trailer when the owner summoned it (supposedly inadvertently) due to the height of the trailer bed above the ground. Very concerning to me given the prevalence of trucks on the road.
I'll give you that it's not a story of Autopilot driving off a cliff, but it may be a case of someone getting killed due to a problem with Autopilot. While it's factually true that the driver did not notice, what we can't know is if the driver would have noticed if they were more aware without autopilot engaged. Anyone that thinks drivers pay as much attention to the road when Autopilot is enabled is being naive.
Technologies like autopilot are going to save lives. This was one of around 90 fatal traffic accidents in the US on the same day (based on historical averages). It's going to be hard to make a foolproof system, however anything that makes it less likely for an inattentive drive to crash into something is a win.
Tragic. Good that they own it, though I'm not thrilled with this:
It is important to note that Tesla disables Autopilot by default and requires explicit acknowledgement that the system is new technology and still in a public beta phase before it can be enabled. When drivers activate Autopilot, the acknowledgment box explains, among other things, that Autopilot “is an assist feature that requires you to keep your hands on the steering wheel at all times," and that "you need to maintain control and responsibility for your vehicle” while using it. Additionally, every time that Autopilot is engaged, the car reminds the driver to “Always keep your hands on the wheel. Be prepared to take over at any time.” The system also makes frequent checks to ensure that the driver's hands remain on the wheel and provides visual and audible alerts if hands-on is not detected. It then gradually slows down the car until hands-on is detected again.
This is the first known fatality in just over 130 million miles where Autopilot was activated. Among all vehicles in the US, there is a fatality every 94 million miles. Worldwide, there is a fatality approximately every 60 million miles.
The wording implies that these numbers are directly comparable, but they are not. I would guess that Autopilot is much more likely to be activated in safe situations. But the other statistics include all situtations.
Yes, and since Tesla explicit ask driver to take over in dangerous situation, the number (fatality/miles) is certainly higher.
Still, impressive result by the autopilot.
I'm wondering, is there any advanced driving simulation? I'm guessing since they have now lots of data, they could simulate dangerous/extreme situation.
Wouldn't it be hilarious if they were being really tacit about something like this? In an exceptional scenario, the autopilot disengages and tells the human user to take over. So maybe the autopilot drives into an untenable situation, disengages, the car crashes and kills the "driver", and then the statistics say "Well, Autopilot drove X miles and it was fine. The crash occurred while under human control..."
As they say, it's not the fall that kills you. It's the sudden stop at the end.
A few weeks ago there was a case where a model S crashed because the human driver tapped the brakes, disengaging autopilot, but the driver did not know, and his car hit another car.
I'm guessing that there is not a traditional steering shaft connected to the steering wheel. Instead, opting for a drive by wire system. There's a sensor in the steering wheel that detects user input and then sends a signal to another motor connected to the steering rack which actually turns the wheels. The steering wheel also has a small motor to that accepts input from the steering rack to give feedback to the driver, think "Force Feedback" systems in video game racing wheels. So, Tesla can send steering signals to the steering rack independent of what the driver is doing with the wheel. This system is becoming more common in all vehicles, not just Tesla's cars.
This disclaimer also seems odd in the eventual case that an Autopilot accident involves someone in another vehicle being injured or killed. Although I implicitly agree to all sorts of risks when I drive on a public road, I'm still struggling with the idea that I'm agreeing to participate in a beta test for an automated driving system. Being a software engineer just makes the idea more terrifying!
"If our imperfect software kills you, too, rest assured that we will offer our empty condolences to your family while simultaneously using technical language to absolve ourselves of any possible fault"
This was entirely defensive and self-serving on behalf of Tesla. It's a general PR release, companies do these all the time. What made this so special?
You would see a similar release from any company where a technology they have been pushing could be front and center as the cause.
This isn't just another horrible car accident on US roads, the car was in charge this time and a technology they've been heralding for it's safety is going to be blamed. That's why there is a federal investigation into it. That's why there is a press release to protect the stock price.
Well yes. Total domestic US car production was two orders of magnitude greater than Tesla's was last year. Ford produced 40% more cars in 1911 than Tesla did last year. Just due to scale, no other company could manage this level of response.
On the other hand, Tesla doesn't do this for every accident. They are treating this one special because Autopilot very carefully, and in a fully controlled manner, drove the car directly into a semi-trailer, which is more than a little embarrassing for the company. And terminal for the driver.
I haven't seen anything suggesting this was "very carefully, and in a fully controlled manner..." Did you get additional news on the accident from another source?
Tesla should not allow autopilots. They have nothing to gain by the negative publicity they will get from cases like these. Regardless of how bad luck the driver had, tesla can always be blamed for relaxing the driver's reflexes by giving a false sense of safety. It doesnt matter if it was by default turned off or hidden in menus.
What other companies use this kind of driving assist system? The ones i know are only hinting to the driver , and disengage if the driver does not respond.
Got to be careful about these things. You don't put beta stuff in cars. It's like your doctor making drug trials on you. They should figure out a way that autopilot engages the driver and keeps his eyes on the road.
>Tesla should not allow autopilots. They have nothing to gain by the negative publicity they will get from cases like these.
This is literally the only way to develop such systems. And regardless of incidents like this, these systems will be developed sometime in the fairly near future and whichever company is on the leading edge of them will command a massive new market that will probably start growing incredibly rapidly right after it takes off.
This is just factually incorrect. Google is developing self-driving tech just fine without releasing it to consumers as a "beta". So is Delphi. So, presumably, is Uber now. Personally I am also of the opinion that releasing autopilot in its current state (especially with the weak sensors) was not a good move in terms of safety for drivers. Great move for PR and gaining data, yes, but we can now make a very plausible argument that it has killed at least one person.
Of course not, but it will work a lot better than autopilot, with significantly lower accident rates. No system will ever be perfect, but there is a huge difference between privately testing something to the best of your abilities before releasing it, and turning on a beta feature easily capable of killing drivers via a software update protected by a disclaimer screen.
What reason do you have to believe that? What are you basing this on?
Just because something is in development longer doesn't mean the quality is better. Furthermore, Tesla might be gaining some critical information with their early release. And whose to say Tesla didn't test this stuff to the best of their abilities?
'Significantly lower accident rates' compared to what, exactly? The highly accurate accident rates collected from the many decades of experience with Tesla accidents that never happened?
I'm basing it on what those companies are actually doing right now in the real world. Both companies have paid drivers babysitting their "driverless" cars on real roads right now, and they will continue to test them this way, out of the hands of the public, for years, gathering data in a safe way over millions of miles driven. Because that's the way to gather data when real lives are at stake. Both companies are also adamant about having LIDAR and additional sensors in their vehicles that Tesla has decided aren't needed. Neither company has or will release a version to the public until they actually think it is safe to turn control over to the car entirely.
Now compare that to Tesla - beta software and sub-par sensors that can't detect stationary objects in the road (there was a recent case where a Model S on Autopilot hit a stalled car on the freeway despite plenty of time to stop), can't detect obstacles at windshield level (there was a recent case where a Model S hit a parked trailer when summoned), and apparently have trouble distinguishing between sky and lightly colored objects in full daylight (the most recent accident). This is a company that is prioritizing technology releases ahead of human lives.
Significantly lower accident rates compared to Model S Autopilot, if that wasn't obvious. In the future, you will take accidents per miles driven in autonomous mode and compare between Tesla, Google, Apple, Uber, etc. and then see which one has the lower rate. I am saying that Tesla will be the worst.
I don't know if autopilot really "saved the day" there. Maybe for an incompetent driver (which is most, so this is a net positive for safety).
But as a human driver, I would have taken human psychology into account and guessed the turning car was going to take advantage of that gap, thus causing me to approach much more cautiously.
I don't know how fast that tesla was going, but I don't think I'd be doing more than 5-10 mph in that kind of traffic. I would also expect cars in the right lane to unexpectedly switch into the left. It seems to me the Tesla was overconfident and going about 25 mph.
edit: I feel like my tone here is too harsh (mainly because my personal reaction is that this autopilot feature is professionally irresponsible). Let me pull back my emotions a little and try again:
The only reason the autopilot feature here "saved the day" is because it was driving in an irresponsible manner in the first place. As the comment below me points out, it was driving at 45 MPH, compared to the 5-15 MPH you should be driving. For a defensive human driver, there shouldn't have been a close call in the first place.
The Tesla is driving irresponsibly because it cannot take into account human behavior. It stopped in time in this scenario, but there's obviously a point at which the stopped car could turn and the Tesla could not physically stop in time. This can happen at any speed, but the 45 MPH collision is much much worse than a 10 MPH collision.
I'd also like to point out that the autopilot was likely breaking the law. The speed limit on that road is probably 55 - 65 MPH, but the speed limit indicats the maximum speed in perfect conditions. When it's dark and raining and heavy traffic, going the speed limit is in fact speeding. If the Tesla had hit the turning car, the driver of the Tesla would probably have been cited for speeding, even at 45 MPH.
It says in the comments/description that the Tesla was driving at about 45 mph, which is entirely too fast in a situation like this. You have a lane of stopped traffic directly adjacent to an open lane. People constantly swerve out of the stopped lane in situations like this.
Further, it would have been common courtesy to stop and allow the car to turn and take advantage of the temporary gap in traffic. People allow me to turn in situations like this all the time, and I return the road karma when I can.
Overall, it seems like quite a failing despite the avoidance of the accident.
There were lots of lights and reflections, and it was night, and the turning car was black. Watching the video the first time, I didn't see the car before it was already in front of the camera. Watching the video second time and knowing what to expect, then I of course see the car in advance.
> There were lots of lights and reflections, and it was night
All those factors indicate the car was going too fast given the conditions -- autopilot or not. It appears that in general, the Tesla AP does not take road and environmental conditions into account (or weigh them heavily at least). In this case, there was no collision in spite of the conditions. But this should have been a giant red flag for Tesla engineers. Unfortunately, the story at hand tells us that they're not doing their job.
> Watching the video the first time, I didn't see the car before it was already in front of the camera.
I didn't either, not at first. But given just the first frame and knowing a collision was about to be avoided, my first reaction was the autopilot was going to prevent a rear-end due to a short stop, or it was going to stop as someone merged in from the right lane.
But you and I are not the driver of this car. The driver should have known the highway was not divided, should have known of the possibility for cross traffic, and he should have adjusted his behavior accordingly. I guarantee you there was also a sign (before the start of the video) indicating there is an opportunity for cross traffic ahead. Given that, those white headlights (they look reddish in the video, but in real life they would be bright white) should caution the driver that a car is probably looking to turn.
The autopilot did not take any of this context into account, and that's exactly why someone is dead today. It's operating with superhuman reaction speed, but with the common sense and wisdom of a 10 year old.
> Was travelling a little under 45 mph. There was some rain, but roads were pretty dry. I was watching stopped traffic to my right.
> I had an Uber passenger
inattentive faux-professional driver, chatting up a client (gotta get that 5-star rating) instead of watching the toad, driving too fast in dark, wet, complicated conditions (line of stopped cars to the right).
It's only similar in terms of "vehicle pulls in front of the Model S". The type of vehicle (car vs. truck), time of day, traffic pattern, etc. all seem to be quite different based on the sparse information in the article. I will also point out that as you have already seen, Autopilot failed in a "similar situation" as well: http://www.digitaltrends.com/cars/tesla-s-summon-under-trail...
Neither Autopilot nor the driver noticed the white side of the tractor trailer against a brightly lit sky,
I'm intrigued that the color is relevant in the car's case - wouldn't it be using some sort of radar to detect and map objects rather than vision? I appreciate I am probably missing something.
Autopilot uses a forward facing camera & radar. It sounds like the truck came in at a perpendicular angle. It may have been in the cone of sight for the camera but not the radar depending on how those are setup.
The 360 degree sensors on the Tesla are for parking only (< 20 feet range).
Personally when I worked on a self driving car we were going with 360 degree camera and doing path planning based on that, but Tesla has opted not to do that.
You could merge multiple cameras in software, it's actually pretty straightforward to stitch them together in OpenCV. Basically just have to correct for the difference between cameras & their physical locations, but that is trivial since they are statically mounted on the car.
Tesla's Autopilot uses a radar unit, a front-facing camera, and twelve ultrasonic sensors on the front and back bumper. The ultrasonic sensors are short range and are for detecting adjacent cars and obstacles while parking, the camera is used for detecting lanes, and the camera and radar work together to detect cars.
The radar is low to the ground and probably doesn't pick up a trailer that's high off the ground. The camera could, but not if contrast is too low. (And I'm not sure if the software is able to recognize the side of a trailer anyway.)
Yeah, the height seems to be the problem. There was a "similar" accident a few months ago where a Tesla was summoned out of the garage and it hit a trailer because it was to high for the radar to get it. http://s3.amazonaws.com/digitaltrends-uploads-prod/2016/05/T...
The range of most, if not all, digital camera sensors ranges into the infrared spectrum. I don't know if it's a useful part of the spectrum for this purpose, nor whether Tesla uses image processors that maintain the full sensor data. Would definitely be an interesting thing to work with, from a tech point of view.
Do you think they will/should change this? I understand the radar can't see an obstruction above x feet, but that doesn't do any good when the car is x+y feet tall.
From what I read about humans, you need +/- 30 degrees up/down vision to get a license (varies by jurisdiction). And we kill a little over a million people each year on roads. Not sure what this should imply about a car's radar though.
I mean I understand Tesla has to make a statement here and I understand they want to ensure everyone that it's not really their fault but to title a post "A Tragic Loss" and then spend the majority of the post discussing all of your car's safety features and how it wasn't your fault just seems tone deaf and distasteful to me.
Maybe they had to do it for legal reasons I don't know (I'm certainly not a lawyer) and I'd love to own a Tesla but couldn't they have worded this a little more sympathetic and a little less lawyer?
I agree. I'm starting to get really weirded out by Tesla's immediate "not our fault!" blog posts every time a Tesla is involved in any kind of accident. I could see the necessity the first few times back when it was new technology to a lot of people (and Top Gear had it in for them), but with it happening over and over it seems whiny and a bit callous. How long to they plan to keep doing this?
Yeah no kidding, thousands of people die every year in Fords and no one gives a shit, 1 guy dies in a Tesla and "these cars are unsafe, run for your lives, hide yo kids and yo wife, sell your stock, return your car!". I fucking hate modern media.
Ford sells 6-7 million cars each year, so there's going to be close to 100 million Fords driving around globally. There's a little over 100 000 Teslas. So since there is 1000x as many Fords, and assuming every Ford is just as safe as a Tesla, you quite literally get thousands of fatalities a year.
The Tesla Model S will now never be one of them, and in fact, is now among average for fatality rates that includes motorcycles, drunks, and other dangerous factors. It's now not even among the class of luxury cars that have very low driver deaths.
Right now self-driving cars are far too dangerous for public use.
Which version? There is 60, 60D, 70, 70D, 75, 75D, 80, 80D, 85, 85D, 90 and 90D so which one are you talking about? Tesla calls their cars Model S the same way Apple calls their laptops Macbook.
It was the other way around - the truck drove into the side of Tesla, which didn't react because it didn't saw the truck with its cameras because of combination of brightly lit sky and truck white color.
Look at a comparable example with software related car malfunctions. Toyota's unintended acceleration problem made headlines and made it before a judge.
It is because, if Tesla allows it to happen, the media will attack them incessantly over these things because they smell a story (and because a lot of vested interests are willing to pay for PR). If you don't defend yourself in that kind of situation, you die.
As long as media continues to blame their new cars for clickbait articles and until they are treated the same as any other car manufacturer that's not blamed for every incident involving their vehicles.
Yeah the thing that rubbed me the wrong way was how they responded to the NYT review about cold weather performance. I've been highly skeptical about Elon and Tesla ever since.
I feel 'A Tragic Loss' sets the tone for the remaining highly technical overview, instead of the technical breakdown seeming tone-deaf, it added tone until getting to the concluding paragraph.
Unfortunately, fortunately, Tesla is having to educate people and being very clear to not allow room for panic and unwarranted fear mongering.
I would say it is as to the point as it can be, and that it is heartfelt.
In addition it had important information for how other Tesla drivers can use the auto-pilot feature more safely. The article combines condolences and helpful safety information to prevent this kind of thing from happening again. Also, you don't see GM making a blog post every time someone passes away in a car they make.
And regardless this was primarily an after-the-fact technical analysis with relevant information to other Tesla owners, the primary purpose was not a formal announcement of a death. The news typically handles that role.
Notice how quick they are to bring up "this is the first fatality in 130 million miles", but AFAICT, traffic fatalities in the US occur at a frequency of around 10 per billion miles [1], which is the same.
So even though this is just one data point, it's spot on the current average for fatality frequency in normal cars.
Furthermore, to gather enough statistics to be able to say with confidence "semi-autonomous Teslas are/aren't safer than normal cars", Tesla either needs to increase their sales volume by several orders of magnitude, or we'll have to wait for ten years.
Is there a statistical difference in accident rate between high and low income populations? If so, do we know what the fatality/miles rate is in the group that would be able to afford a tesla? When I read the blog post, I couldn't help but think that while tesla's rate might be smaller than the national average, it is also not a fair comparison to draw conclusions from.
IDK about statistical difference, but the mechanical difference between a brand new high-end full-size car and a twenty year old Honda Civic is going to be massive. Just look at the downward slope of that curve of fatalities per billion miles in each year. Also remember that those numbers are averages over the entire car population in each year, so most cars in the 2015 data are quite a bit older than 2015.
If we had data for the "high end modern full-size car" category, it would most definitely be lower than 10 fatalities per billion miles.
Having the same fatality statistic was not enough for Google to bring autonomous driving into production. Many people think that it should be at least an order of magnitude safer than manual driving
Sure. The problem with this is statistics: you need to have your autonomous cars travel at least 100 billion miles before you can confidently say they're an order of magnitude safer.
A normal car drives about 100 000 miles in 10 years. You then need one million autonomous cars driving around for a decade before you even know whether they're safer!
That's very true, but if you know your product well enough, and you perform well conducted experiments, you can come up with a good estimate for this without actually having your cars travel 100 billion miles.
Sure. The main issue with that is the unknown unknowns. In hypothetical cars an order of magnitude safer than today, we're talking about events so rare that a single (hypothetical) accident which kills all 7 passengers due to the Tesla misinterpreting an obscure Hungarian road sign meaning "bridge out ahead" will significantly influence your safety statistics.
There are enough youtube videos with people falling asleep in Tesla on the highway and a Tesla switching lanes and almost causing a frontal crash. These are known problems.
Maybe a billion miles are needed for a fatal crash, but for small mistakes it's much less.
You don't need an equal sample size to determine if autonomous cars are safer. But if you're comparing to all the cars on the road today, you need quality data that matches the distribution of cars today. Google isn't going to prove that autonomous cars are safer if they only test in Palo Alto in ideal weather conditions.
Close is not the same and neither is the same as 1/134 million. Of course I have no idea if the differences are statistically significant, and it's likely impossible to tell given the lack of data on Tesla autopilot at the moment (new).
Well of course it's not statistically significant, 1/134 is based on a single data point. At this level of confidence, all these numbers are equal. If this was the first fatality in 1000 million miles, it might be an indication of higher safety. Might.
If you want to do the numbers to compare the Tesla to other cars, where they measure driver fatalities per million registered years, do this:
130,000,000 miles / 12,000 average miles per year per car = 10,833 years.
That's 10,833 registered years for the Model S.
The IIHS site normalizes to deaths per million registered years, so multiply that number by 92.3, and since we have one death so far, the Tesla Model S is now at 92.3 driver deaths per million registered car years.
Let's compare to other car models:
Ford Taurus 2WD: 20 deaths
Hyundai AccentL: 86 deaths
Lexus LS 460 2WD: 18 deaths
Acura TL 2WD: 5 deaths
Audi A4 4WD: 0 deaths (yes, a bunch of car models had zero driver deaths)
And average for ALL 2011 vehicles: 28 deaths.
So, right now, it looks like the Tesla Model S is about 4x deadlier than the average car when using Autopilot, and about 10-20x deadlier than the safest cars.
>So, right now, it looks like the Tesla Model S is about 4x deadlier than the average car when using Autopilot, and about 10-20x deadlier than the safest cars.
How you can draw such a conclusion from a singular data point? That's beyond ridiculous.
Seems as valid (and invalid) as the implied conclusion here:
> This is the first known fatality in just over 130 million miles where Autopilot was activated. Among all vehicles in the US, there is a fatality every 94 million miles. Worldwide, there is a fatality approximately every 60 million miles.
Fair enough. I think the framing matters a bit though.
"4 Times Deadlier!" certainly comes across as sensationalist based on such flimsy data (never mind that the real figure is actually 3.2 and the parent chose to round up rather than down to serve his/her purpose).
Whereas Tesla's statement comes across more as a statement of fact (albiet quite self-serving.. no question about that).
When these automated cars can complete a journey __without__ handing back to the human, only then can those journey miles be added to the miles driven under autopilot claim. Until such time, consider all automated cars a considerable risk to other road users.
How are you defining automated? No truly autonomous cars are available yet, so they're all some form of driver assist. How are these a risk to other road users, and at which point do they become a risk and not a safety benefit?
Auto-pilot is semi-autonomous... I gather you consider that a risk...
What about Subaru EyeSight? It can autonomously brake. I can drive for hours on the freeway without ever hitting the brake nor gas, even in stop and go traffic. It has warned me multiple times of a sudden slowdown just as I noticed it. It hasn't yet saved me from anything, but if any of those circumstances I'd been distracted (even for a legit reason) it could have. It once warned me of a motorcycle in my path on a very dark night when the dim tail-light of the motorcycle was overwhelmed by the much brighter tail-lights of the cars to its left and right. I would have seen it soon enough, but the car saw it first. Is that a risk to other road-users? Yet, it's automated by some definition.
What about various other manufacturer's lane keeping features? That automation to some degree.
What about ABS? Stability control? Again... automation to some degree.
All of these have their detractors, yet I think statistics clearly show an improvement with each level of driver assist.
To the people downvoting the parent of this comment: yeah, I should have put something like [deleted] in there instead of just hacking in random characters. You can stop now. It's no longer useful.
The 130 million miles is not Model S overall, but Model S with autopilot engaged. It's also only the first 130 million miles, so the sample size is currently 1. This will be much more interesting to look at in 5 years, and at the moment it tells us nothing about Model S.
I had posted the same comment before I saw this one (I deleted mine).
I would assume that the higher-end GM cars do have an automatic braking collision avoidance system by now, since that's a pretty standard feature above a certain price point. And I would assume it's not foolproof either, but at least it doesn't lead the driver to think he can take his hands off the wheel and not pay attention to what's happening on the road.
> at least it doesn't lead the driver to think he can take his hands off the wheel and not pay attention
Tesla reiterates every time and makes you confirm you understand that this is not the case. If the driver did it, that's completely on them being irresponsible.
The autopilot in planes works the same way: pilot is strongly recommended to keep hands on the controls, autopilot will release control if any conditions fall outside the autopilot's functional envelope.
there is no such feature, there is a MobilEye _lane following_ system Tesla repackaged under Autopilot(TM) brand.
Say it out loud - lane following. Mercedes sold this same tech almost 10 years ago, difference is Tesla felt pressure and started believing their own marketing lies about autonomous driving.
Yeah, but that's not how people's emotions usually work.
Although US foreign policy and intelligence decisions were more or less directly responsible for 9/11, you don't lecture people at Ground Zero about the dangers of blowback and not being aware of your government's activities.
There's no such thing as "condolences, but...", and Elon Musk has shown a pattern of this kind of tone-deafness before. Not to say he's not brilliant and not to say autonomous cars won't ultimately save many lives, just that he should either listen to his PR department if he overruled them, or fire his PR department if they did this without him.
"There's no such thing as "condolences, but...", and Elon Musk has shown a pattern of this kind of tone-deafness before. Not to say he's not brilliant and not to say autonomous cars won't ultimately save many lives, just that he should either listen to his PR department if he overruled them, or fire his PR department if they did this without him."
Excellent, Doc. Further to the point about these statements and Musk's leadership: not too long ago Tesla would include the fact that nobody has been killed in a Tesla in their public statements. I found it completely inappropriate since we all knew it was just a matter of time, sadly. Again they play with fire, as another death with autopilot will really wreck their statistics.
There's a warning that makes it clear that you need to maintain your full attention on the road, when you enable Autopilot and every single time you take your hands off the wheel. If anything, giving users specific scenarios where Autopilot may fail would take away from that fact, and make it seem like there are other cases where it's ok to take your eyes off the road.
All that matters is, did the technology fail. I don't care about crash rates in other cars, if the impact had been different, the only fact that matters in this case is, did the tech fail. If so then is it safe to leave on or should it be disabled across the board until it cannot fail this scenario again.
one failure and they will take a minor publicity and money hit, two and its going to devastating
Whether it performed as designed or not really isn't relevant. The salient point here is Tesla's promotion of an "auto-pilot" system that encourages users to take their attention off of the road and put too much faith in a glorified lane-following system.
Now, this is pure speculation, but I really can't imagine running into the side of an 18-wheeler crossing the highway unless A) I was traveling at a reckless speed (unlikely given that "auto-pilot" was on) or B) I wasn't paying attention to the road.
For one thing, it lookslike the car kept driving after the crash, hit 2 fences and a pole, then tried to get back on the road before it shut down. Sounds like pretty big screwup.
They could have titled it "Tesla Safety" -- which is what the article is about, and opened with an acknowledgement of the tragedy, instead of the tacky bait-and-switch headline.
Yes this is in really poor taste. I'm sure it's very comforting to the deceased's family to know that safety features could have saved their life, had the angle of impact been different.
Press Releases like this are for investors and media outlets, not for general consumers. Of course they are going to focus on damage control, because if they didn't focus on the safety features of autopilot and the hands-on requirement some media outlet would contort the situation and say "What safety features does this car to prevent an accident like this in the future? Is this technology actually ready to be used on our roads?".
I think they did a fine job of handling this, and with the high visibility of this incident due to the use of autopilot they really had no other option.
I would not call a sentence in a manual a "requirement"
A requirement would be when the autopilot didn't work unless one has at least one hand on the steering wheel. (Just like other manufactures do that)
Tesla knows people are not going to use autopilot hands-on (especially after the novelty wears off)
That's exactly the requirement the Model S has. It starts beeping at you and gradually brings the vehicle to a stop if you take your hands off the wheel.
I was even left with the sense that the tragedy actually felt by the writer was not so much the death of "a friend," as the fact of a blemish on the near-perfect safety record that Tesla has made part of their brand's cachet.
They have to defend autopilot not only to protect the brand but to protect the public's perception of autonomous vehicles in general.
Self driving tech is poised to save many many lives. So from a utilitarian perspective, it's probably justified to take extraordinary measures to make sure reactionary media and public whim doesn't kill it off, however uncomfortable that might seem in the short term.
Whilst this case is incredibly sad (and I don't want to downplay that in any way), if you're trying to minimise the overall amount of fatal crashes, exonerating the tech is the priority (if it is truly not at fault).
They could defend the public's perception of autonomous vehicles in general by not rushing to market a beta that is less technically capable then other autonomous systems being worked on by their competitors.
Are you saying that just based on the fact that they've been developing it for less time than Google, or is there a more in depth case for that somewhere? I would be very interested to read about that if there is.
> They have to defend autopilot not only to protect the brand but to protect the public's perception of autonomous vehicles in general.
Why? Why not just build the cars people want? (including cars that people want but don't realize they want yet)
There shouldn't be a political agenda associated with engineering. Build what is needed. Build what people want. Build what people will need. But never "defend my reputation and the reputation of this device that I'm making"
EDIT: I mean, I get why Public Relations are important and so forth. So Tesla is certainly free to do what they want here. But lets not pretend that this carefully crafted "condolence" piece that has come out roughly one and a half months late is anything but damage control for this company.
> Self driving tech is poised to save many many lives.
The computer would have to be 99.99999% reliable to do that.
The accident rate is around 74 per 100 million miles (and fatalities is 1.13).
It's unclear exactly how to turn that into a percentage, but no matter how you do it it's quite high.
Say an accident takes 5 minutes, and people drive 30 miles/hour. Then that works out to 99.999% for humans. If you use the numbers for fatalities then it's 99.99999%.
I.e. 99.99999% of the time, as whole across all [US] humans, people drive in a way that does not cause a fatality.
That's the bar computers have to cross in order to save any lives at all.
This is the first fatality collision after 130 million miles of tesla's autopilot, so it's already above that bar.
Additionally, 92 people are killed in fatal car accidents in the US every day. So it's not as though this is some uncommon occurrence that autonomous vehicles would be unlikely to improve.
> This is the first fatality collision after 130 million miles of tesla's autopilot, so it's already above that bar.
It most definitely is not above that bar! The autopilot is combined with a human operating the car.
How many accidents were avoided by the human rather than the autopilot?
I suspect a ton of accidents (i.e. the autopilot has a high error rate).
How many did the autopilot avoid that the human would not have? Probably not that many, if any.
> Additionally, 92 people are killed in fatal car accidents in the US every day. So it's not as though this is some uncommon occurrence that autonomous vehicles would be unlikely to improve.
Do the math.
I did.
0.00001% seems pretty uncommon to me. There is a lot of driving in the US, so even that low of a number is visible, but it's still a low number.
Remember every single computer must be all but perfect to reduce the error rate.
Have you ever seen a computer to be that good? At even something as simple as not crashing? Never mind driving a car.
Except I am not guessing. That's the point of math.
The autopilot plus a human, only did slightly better than a human alone. That means the autopilot did nothing, since slightly better is well within the range of normal for a human alone.
> So please keep your math out of this.
What a strange reply. Why keep math out of this?
Are you hoping that this will be real if you ignore all evidence to the contrary?
Right now the evidence is in: Computer assisted cars don't do anything helpful to the accident rate. This bodes poorly for self driving cars, and since the error rate they have to hit is so low, it's really not looking good.
I personally don't expect self driving cars to ever be used on regular streets. Only on computer-exclusive roads, specially marked for them.
92 people per day isn't "uncommon". It might be statistically improbable, but that's not what uncommon means. It's a regular, daily occurrence for 92 real human lives every day.
Uncommon for the computer (or human) driving the car.
You have to understand the magnitude of the perfection needed to have any hope of implementing it.
Tell me: Before you read my message, would you have assumed at a computer that is 99% perfect, or 99.9% perfect, would be better than a human?
I can tell you, that until I did the math, I thought so myself.
But 99.99999% is 10,000 better! If the computer was 99.9% perfect you would have almost 1 million fatalities per year (assuming things scale linearly, which I'm sure they wouldn't, probably most of the time driving is easy and 99.9% would still not get into an accident).
> It's a regular, daily occurrence for 92 real human lives every day.
I know. (Although you said that badly: it's not a daily occurrence for those people. But I get your emotion.)
But a computer will not solve this problem, not for a long time. We simply are not able to make a computer that is that good.
Let's see if we can make a web browser that is that good before we try to make a driving computer that good.
It also puts human drivers in a new light. I was of the camp that people are terrible, horrible drivers that kill all the time.
But actually humans are nearly perfect at driving, it's just there are so very very many people driving, so even a tiny cumulative error shows up.
There are already videos of assisted-driving cars avoiding accidents which likely would have been fatal. We already know they can potentially save lives. It's something we already accept for flying. Autopilot in planes has resulted in deaths of crew and passengers. So we have precedent in at least one mode of transportation.
Drivers get T-boned or rear end other people all the time. Those are two scenarios where driver-assist could have applied the brakes and actively saving lives for drivers inattentiveness. You'll also find no shortage of Liveleak clips of people being killed by drivers who failed to see them and brake in time. Many being scenarios that Google's self-driving cars have the technology to avoid.
This is a ridiculous statement. Are you seriously saying that birds regularly fly at over 30000 ft? Or are you trying to imply that autopilot during landing/takeoff actively tracks birds and tries to avoid them?
In other words, the plane autopilot doesn't "deal" with wildlife at all unless you count running into it. Which is apparently what Tesla autopilot does as well.
>> There are already videos of assisted-driving cars avoiding accidents
I'm not concerend about assisted driving. I'm concerned about fully autonomous cars being handed complete control in real-world situations.
I'm not just "concerned" about them- they scare the hell out of me.
>> Many being scenarios that Google's self-driving cars have the technology to avoid.
What Google says it programs its cars to do, what it really programs its cars to do and what its cars can really do are all separate things.
The problem is that the current technology level is nowhere near advanced enough to allow fully autonomous vehicles to operate safely. There is a huge number of situations that those cars aren't programmed for, that they can't be programmed for, because those situations are completely unforseeable.
Machine learning has a huge problem with data sparsness. You may train a learner in petabytes and petabytes of data collected from the real world and still miss the vast majority of events that may occur.
That is why, like I say elsewhere, machine learning-based AI makes utterly ludicrous mistakes that humans would never do, even in difficult situations were they can't be expected to perform with 0% error. I've used a few metaphors- here's another one: a human would never mistake a truck for a cucumber. A machine learning algorithm, might.
And what's worse, there's no way to prevent this sort of mistake, or even correct it, because most of the time the models built by such algorithms are simply too complex to be processed by humans in the way a hand-crafted system would be (and goddess knows how hard those can be to process).
The number of deaths – and deaths relative to the total population – have declined over the last two decades. From 1979 to 2005, the number of deaths per year decreased 14.97% while the number of deaths per capita decreased by 35.46%.
So at least the rate of fatalities is decreasing. I won't claim I know whether that's because humans get better at driving, carse get safer, traffic regulations get safer, or any other factor, but the point is roads are getting safer already and without any autonomous vehicles on them.
[Original bit:]
Again- based on what? Why do you say that? What is even your measure of humans getting "better"? Better at what? Better drivers? Better cognitive systems? Better what?
> You say autonomous cars are "poised" to save many lives
I said no such thing, you are putting words in my mouth.
What I said is that the baseline for traffic deaths is quite high, and if (or when) autonomous cars are even a little better than humans, then less lives will be lost.
>> I said no such thing, you are putting words in my mouth.
I was replying to someone else and you jumped in. You can hold the accusations of putting stuff in places, thank you.
>> What I said is that the baseline for traffic deaths is quite high, and if (or when) autonomous cars are even a little better than humans, then less lives will be lost.
And I said that autonomous cars being better than humans is just a hypothetical.
I'll add that we have no evidence that it's the case. Computers are better at humans at tasks that require fast and accurate retrieval, or computation, but they'll not just magically become better than humans at driving. Somebody has to program them to do that.
So who is going to do that? Do you know how to do that? Do you know anyone who knows how to do that?
The theory is that we'll put enough (real or virtual) cars in enough training situations that they'll learn to drive on their own, but that approach has serious limitations, not least the fact that in the real world there may be an infinite number of situations that a learner will never encounter during training. In any case, with any other cognitive task that we train computers to carry out they end up making completely ridiculous mistakes, which in the case of driving will cost lives.
Until we have a way to develop systems that understand their surroundings, autonomous cars being better drivers than humans is just a dangerous fantasy.
Human programmers are even worse we make 15-50 bugs per 1000 lines of code.
Also don't forget that while each of us is somewhat unique in terms of driving, the cars are not, so a bug in a car's software can amplify issues even further.
There are issues also how cars will behave when they will have to interact with other self driving cars. Those problems are already difficult when writing distributed applications and with cars we don't have luxury of just letting things crash together until things improve.
> But how much work the software does is not what makes it remarkable. What makes it remarkable is how well the software works. This software never crashes. It never needs to be re-booted. This software is bug-free. It is perfect, as perfect as human beings have achieved. Consider these stats : the last three versions of the program — each 420,000 lines long-had just one error each. The last 11 versions of this software had a total of 17 errors. Commercial programs of equivalent complexity would have 5,000 errors.
Yes, self-driving cars will save lives if we assume that the self-driving technology is superior to humans in every way. That's almost a tautology though.
Self-driving cars don't get bored, but they don't seem to see white tractor trailers either. Technology has different shortcomings than people do. Whether tech shortcomings result in fewer lives lost than human shortcomings remains to be seen.
It's astounding to me that HN stories about self-driving cars sit right next to stories about iPhones and anti-virus software, and tone of the comments are diametrically opposed. If you read the comments on self-driving car stories, you'd think that technology can do everything better than people. If you read the comments on stories about iPhones and anti-virus, you'd think that everything has terrible bugs, vulnerabilities, and shortcomings.
> but they don't seem to see white tractor trailers either
Isn't Tesla using pure vision? Google also uses lidar, which fixes the problem. EDIT: Looks like I'm wrong and Tesla uses radar.
> Technology has different shortcomings than people do. Whether tech shortcomings result in fewer lives lost than human shortcomings remains to be seen.
Technology also gets better. Ten years ago, people were as bad at driving as they are today. Ten years from now, they'll be as bad as they are today.
Now compare self driving cars ten years ago to self driving cars today. And ten years from now.
> If you read the comments on self-driving car stories, you'd think that technology can do everything better than people. If you read the comments on stories about iPhones and anti-virus, you'd think that everything has terrible bugs, vulnerabilities, and shortcomings.
I assume you just pulled that out of thin air but even if you hadn't, it is completely meaningless. We have nothing to compare human performance against. Humans have been driving for more than a century. What are you going to compare that to?
As to how preventable accidents are- sure, if you remove factors like the ones you say, you'll reduce accidents, but humans suffer from those cognitive impairments on top of our extremely well-tuned cognitive apparatus. Computers have nothing of the sort. They may not get distracted, but they're rubbish at doing the thinking that keeps humans safe on the road the vast majority of the time (given that the vast majority of drivers don't get killed in accidents).
To put it plainly, if you put a brick on the gas pedal and lock the wheel, you'll reduce the amount of accidents caused by cogntivie impairments like inattention and poor judgement- but you won't reduce the overall rate of accidents.
In other words, if you hand control to a system that doesn't sleep on the wheel but can also not tell a puddle from a boulder, you will increase risks rather than reducing them.
Reminds me of their complaint blog post why the German government initiative is capped at 60k where it seems like half the point was to advertise a new lease deal.
Yeah, I had the same issue with the writing. I think it would have gone much better if the meat of the concluding paragraph came first, possibly with a matching paragraph at the bottom.
Personally, I would have used a more neutral title, led with sympathies for the family, and then gone into the technical detail.
I half expected the final paragraph offering condolences to end with "but remember, this really wasn't Tesla's fault!" The third paragraph reads like Tesla is trying to wash their hands as much as possible. "Remember, he clicked all these buttons. He knew what he was getting into!"
Yes. Last time, their excuse was "but the driver should have known that in firmware revision 6.2 and later, tapping the brake disables automatic braking".
This time, there's an interesting question. Did Tesla remotely access the crash data after the crash? Did they alter any data? Is that verifiable? The NTSB will probably explore that issue. The crash data record in an airbag controller becomes read-only when the airbag fires.
It really makes you consider why you would even want such an autopilot. From the description in this eulogy, it sounds like you must work even harder with autopilot on than if you were driving manually. Not only do you need to have the same alertness as manual driving, you also have to be ready to take over at any moment and try to detect if the autopilot has made any mistakes that might guillotine you, or worse.
IMO, they should've had their explanatory intro, then the paragraph they closed with, then a gap, and then the technical explanation. As it was, it read like hand-washing to me. Especially when you openly call it AutoPilot but require hands on the wheel at all time.
If their detectors don't see a white car against a bright background, that's obviously a serious problem.
>couldn't they have worded this a little more sympathetic and a little less lawyer?
Sure, if all the lawyers will promise not to take some statement out of context and sue them over that.. As long as such lawyers exist and that's the way the legal system works this is what can be expected out of statements from companies..
Main issue is that once you take away 90% of the requirement for a human to focus on the task at hand (driving) they are going to ignore the remaining 10% where an exception occurs.
> Neither Autopilot nor the driver noticed the white side of the tractor trailer against a brightly lit sky,
So this got me wondering: are there areas where retinae still do better than cameras at certain kinds of discrimination problems? Nature still has human technology beat on hardware in a few categories (particularly joints), but I didn't expect eyes to be one of them. When I look at or near a bright light, I can see objects close to the light much better than I can in a photograph, but I always assumed this was only because I had a cheap camera.
The corresponding question is: how expensive is the camera that prevents this accident?
Cameras are obviously worse in situations where there is very high contrast (e.g. when you are facing the sun), because the digital representation of color/luminosity has a far more narrow range than that of the eye. Uncertain on whether this has any practical consequences, but it seems likely.
Is this the new motto for autonomous vehicles? Move fast and break bones?
With these minor bugs, Tesla seems to be doing a solid job of poisoning the well for self-driving cars. I'd like to see them explain how their competitors should not be tarred with the same brush, once the political backlash hits.
It doesn't matter how many disclaimers you give before you turn on autopilot - a driver who focused on driving the car (As opposed to letting autopilot cruise) would have probably noticed a tractor driving across the road on a bright, sunny day.
It's a dangerous system. Instead of arguing about the trolley problem, I'd first like to see a car be good at making decisions that save its passengers.
I'm personally very worried about the poisoning the well angle. Tesla is playing fast and loose with this compared to Google and the results could strangle the whole field.
The thing about self driving cars is that every accident will have a wealth of information in regards to how it occurred and the decision making that went into it. Once we understand and correct the problem every subsequent car will be safer. This is not the case with humans whom regularly fail to learn from consequences to other humans.
Imagine if the first time someone fell asleep at the wheel and crashed, you could just tell everyone "hey don't fall asleep at the wheel". And it just never happened again.
I do hope they are able to learn more from this. This is just a bad coincidence of environment meets chance and someone died because of it.
I do think computer vision still has a but further to go. A white trailer and a white sky shouldn't be a problem. But everyday I'm impressed with how human sight can find and lock into the most obscure details. Like I've been sunblinded before and yet I knew something was in front of me because I noticed a grey shadow on grey asphalt and immediately hit the brakes.
> I do hope they are able to learn more from this. This is just a bad coincidence of environment meets chance and someone died because of it.
Nope, this is completely expected when unproven technology is packaged as a push-button feature and shipped on consumer technology. I don't work for Tesla, but I am a robotics engineer who has worked on exactly this technology, and I saw this coming from a mile away. Tesla should have as well.
Other premium automakers have automatic braking and pre-collision systems that take effect regardless of whether the (enhanced) cruise control is engaged.
We have seen from other incidents that Tesla's autobraking will not engage when autopilot is disabled (potentially by tapping the brakes), which is contrary to widespread user expectation -- disclaimers notwithstanding.
However, in this particular case, it was the obstruction detection that failed. Based on their blog post disclaimer, it appears to be a visual sensing system; eye-height radar-based system used by most other manufacturers would have correctly detected the obstruction.
Is there an industry standard (yet) for the efficacy and behavior that these systems must meet?
I stand corrected about the term 'eye-height' -- apparently no one I could credibly find mounts their radar at eye-height.
Mercedes-Benz uses two different radar sensors:
"the DISTRONIC radar is configured to monitor three lanes of a motorway to a range of up to 150 metres with a spread of nine degrees, the new [DISTRONIC PLUS] 24-Gigahertz radar registers the situation immediately ahead of the vehicle with aspread of 80 degrees and a range of 30 metres." [1]
Volvo has a combination of grill-mounted 15° FOV radar and a 48° FOV greyscale camera behind the top of the windshield [2].
Volkswagen uses radar [3], but not sure where it's mounted. Honda's CMBS is a radar in the grill [4]. Subaru uses two cameras mounted behind the top of the windshield [5], and no radar, albeit radar is used with some of their other safety systems.
> It is important to note that Autopilot is...still in a public beta phase...
No, it's not important to note that. You should not be able to hide behind the word "Beta" for systems that could kill people. Either you're willing to let people risk their lives on your software or you're not, and you were.
The accident probably wouldn't have happened if the driver was focused on driving the car... As opposed on letting the car drive itself.
When the autopilot acts correctly 99.9% of the time, you'd get incredibly complacent and inattentive. When you don't have an autopilot, you are paying far more attention to the road.
In this press release, they argue that it's important to note that their feature is in beta. If the accident would have happened either way, then it was not important to mention the beta status.
They already do in the back of the trailer where most collisions are likely to take place (it's typically a welded set of cross-beams that go down to roughly bumper height). I think that very few collisions are of this side-on type that would make it worthwhile to add side panels as well. It's kind of one of those "how safe is safe enough?" situations where you have to compromise at some point.
Very sad. I once designed a robot that used infra-red collision detection to avoid obstacles, my home built, and painted flat black, stereo cabinet registered as open space and the robot drove right into it at full speed. It was a quick lesson on sensor fallibility for me.
I've also observed human pilots on busy roads nearly colliding with obstacles when driving toward the setting sun, the visor pulled down and still trying to shade their eyes.
But robots aren't humans, and they don't have to rely only on vision, there are so many ways to "look" it seems like we should have several different ways of identifying obstacles. Different spectra at least.
I have had similar experiences working in robotics and I would imagine anyone else who has worked with designing navigation, mapping, or obstacle avoidance for robots would also be well aware of the shortcomings of each type of sensor.
Even in the confines of indoor environments with simple problems like transparent surfaces, specific problematic fabrics, and acoustic panels it is necessary to fuse several types of sensor data to provide reasonably reliable detection of the environment.
I would hope that as sensors and output analysis techniques continue to be developed and costs decrease, creators of autonomous systems would incorporate a wider variety of sensors into their products.
This is why driving AI is 'all or nothing' for me.
Assisted systems will lead to drivers paying less attention as the systems get better.
The figures quoted by Tesla seem impressive but you have to assume the majority of the drivers is still paying attention all the time. As auto-pilots get better you'll see them paying attention less and then the accident rate will go up, not down for a while at least until the bugs are ironed out.
Note that this could have happened to a non-electric car just as easily, it's a human-computer hybrid issue related to having to pay attention to some instrument for a long time without anything interesting happening. The longer the interval that you don't need to act the bigger the chance that when you do need to act you will not be in time.
It's pretty well established that humans (e.g. [1]) that humans have a lot of trouble paying when they don't need to be actively engaged most of the time and also have trouble taking back control. In a consumer driving context, I have zero doubt that, as systems like these develop, people will start watching videos and reading absent draconian monitoring systems to ensure they keep their eyes on the road. I'm not sure how we get past that "uncanny valley."
Its worth pointing out that Prof. Missy Cummings, who authored the paper, is a former F/A-18 pilot who specializes in human-machine interaction research.
One option is the Tesla autopilot should have an indication when it approaches "low confidence" areas without disengaging, so the driver is not startled if they have to take back manual control.
The Mercedes E-Class Drive Pilot seems to do this to some extent. For example even going from demanding the driver to place one hand on the steering wheel to requiring both hands.
Yep. I saw her speak at an event a couple years ago. She's also done a fair bit of research around humans splitting their attention across multiple vehicles--mostly drones.
I agree there can be handoffs but they mostly need to be in the vein of: I'm slowing down and turning over control in 30 seconds because there's something I don't understand coming up
Apart from the software part of it. I wonder how they handle issues like sensor malfunction.
If your eyes aren't at their best, you know well to go to a doctor and not be driving in the meanwhile. Will the car with autopilot refuse to start or go on autopilot if the camera/sensor/radar has an issue?
So yes you are right, its either full AI or nothing.
In the aviation community, there is the major concerns over pilots becoming over-reliant cockpit automation instead of flying the jet.
Asiana 214 [0] is a classic example of crashing a perfectly good airliner into a seawall on landing.
In the Boeing 777, one example of the (auto)pilot interface showing safety critical information is the stall speed indication on the cockpit display [1], warning the pilot if they are are approaching that stall speed.
Hopefully Tesla will optimize the autopilot interface to minimize driver inattention, without becoming annoying.
My understanding of the Asiana crash was that the autopilot would have landed the plane fine, and that it was the humans turning it off that caused the problem.
Your point is still valid, but perhaps we approach a time when over-reliance is better than all but the best human pilots (Sully, perhaps).
The Air France 447 accident is a better fitting example of pitfalls that may obtain in complex "humans-with-automation" types of systems.
There, automation lowered both the standard for situational awareness and fundamental stick and rudder skills. Then, when a quirky corner case happened, the pilots did all manner of wrong on the problem: so much so, they amplified a condition from "mostly harmless" to fatal for all.
Vanity Fair has a nice piece on this accident that's easy to dig up. Good read.
I heard it was the Airbus weirdness of steering setup that noticeably added to the problems (Separate, disjointed joysticks)
One pilot pulled up as hard as he could while the other one thought he was pushing down, making the confusion this much worse
That's true, but was well known (and trained on), so I'd categorize that domain as "How the machine responds when you're hands are on the controls," which is nearly a synonym for "stick and rudder skills" category I cited.
Sure, to nearly every pilot that behavior is wacky, but it shouldn't have been a surprise for more than an instant to pilots who were "operating as designed."
It seems there's no free lunch: when skills atrophy as a natural response to helpful automation it requires advancement in some other skills, should the goal of an ever improving error (accident) rate be achieved.
The Asiana pilots were not able to fly a coupled (automatic) landing due to the ILS glideslope being out of service.
The pilots were under the misguided impression that the aircraft would automatically spool-up the engines if the aircraft became to slow. This was a safety feature that didn't engage for a obscure technical reason. Even with a manual visual approach the pilot can still use the autothrust for landing.
A more rigorously trained pilot (eg. Capt. Sully) would have aborted the approach and performed an immediate go-around if he got below the glidepath (or too slow) below a certain altitude (eg. 400ft Above Ground Level).
The rules requiring a go-around (or missed approach) apply for a fully automated approach and landing, just as much as manually flown approach and landing.
In aviation, autopilots became successful because the human-machine handoff latency required is relatively large --- despite how fast planes fly, the separation between them and other objects is large and there is usually still time (seconds) to react when the autopilot decides it can't do the job ( https://www.youtube.com/watch?v=8XxEFFX586k )
On the road, where relative separation is much less (and there's even been talk of how self-driving cars can reduce following distances significantly, which just scares me more), the driver might not have even a second to react when he/she needs to take over from the autopilot.
Wow, that is some very damning criticism: "The distinction is that a Level 3 [Tesla] autonomous system still relinquishes the controls back to the driver in the event of extreme conditions the computer can no longer manage, which Victor (and Volvo) finds extremely dangerous."
Philip Greenspun's impressions after trying out the Model X for a weekend:
"You need to keep your hands on the steering wheel at all times during autosteering, yet not crank the wheel hard enough to generate what the car thinks is an actual steering input (thereby disconnecting autosteer). I found this to be about the same amount of effort as simply driving."
This is what I've now said 3 or so times in various autopilot threads. It has to be an all or nothing thing. Part of responsible engineering is engineering out the many and varied ways that humans can fuck it all up. Look at how UX works in software. Good engineering eliminates users being able to do the wrong thing as much as possible.
You don't design a feature that invites misuse and then use instructions to try to prevent that misuse. That's irresponsible, bad engineering.
The heirachy of hazard control [1] in fact puts administrative controls at the 2nd-to-bottom, just above personal protective equipment. Elimination, substitution and engineering controls all fall above it.
Guards on the trucks to stop cars going under are an engineering control and also perhaps a substituion - you go from decapitation to driving into a wall instead. It's better than no guards and just expecting drivers to be alert - that's administration - but it's worse than elimination which is what you need if you provide a system where the driver is encouraged to be inattentive.
User alertness is a very fucking difficult problem to solve and an extremely unreliable hazard control. Never rely on it, ever. That's what they're doing here and it was only a matter of time that this happened. It's irresponsible engineering.
edit: My source for the above: I work in rail. We battle with driver inattention constantly because like autopilot, you don't steer but you do have to be in control. I could write novels on the battles we've gone through just to keep drivers paying attention.
> I could write novels on the battles we've gone through just to keep drivers paying attention.
Please do, and link them here. I'd be very interested in reading about your battles and I figure many others would too. This is where the cutting edge is today and likely will be for years to come so your experience is extremely valuable and has wide applicability.
Here's a comment from 5 months ago about one example - not me personally but it's one of the major case studies in the AU rail industry - that covers exactly this topic. It also sort of morphs into general discussion about alertness tools in physical design.
I understand your point that it has to be all-or-nothing, but if you were asked to redesign the UX to make autopilot (as it currently stands) safer, how would you change it?
Precisely. This half-way 'self driving but if something goes wrong it's the human's fault' is a terrible idea, because the more reliable it gets (while still being a 'driver assist' rather than committing to actually being a fully autonomous control system) the more likely the human is to not be paying attention when the shit hits the fan.
Toyota is taking the opposite approach to Tesla: they are introducing automated features as a backstop against human error, rather than a substitute for human attention. Your Toyota (or Lexus) won't drive itself, but it might slam on the brake or swerve to avoid an obstacle you couldn't see.
As someone very excited about this space, I unfortunately have to agree that Tesla is playing fast and loose with autonomous safety (and more importantly, public opinion!) to be first to market. You can't be half in and half out, which is what these "assist" features are.
They're adding new features to inadequate/improperly configured hardware for what they're asking the car to do, and waiving away all liability for stupid peoples' actions with disclaimers (always be ready to take over).
Whether that's right or wrong is really subjective, especially when you take natural selection into account.
Tesla's (only!) radar sensor is located at the bottom of the bumper, if I'm not mistaken. Compare this with Google's, which is located in the arguably correct position, the roof. Also compare other manufacturers' solutions that are utilizing 2-3 radar sensors, as well as sonar.
510 comments
[ 3.0 ms ] story [ 342 ms ] threadTragic no doubt, but I'm relieved that this was not a "Autopilot did something very very wrong" story.
Autopilot has the potential to save a large number of lives (I'm sure Tesla execs are thinking about touting "estimated lives saved by autopilot" if the numbers work out, after a few billion miles), so I hope incidents like this don't hamper public perception and therefore research.
No matter how many messages people throw at users, any autopilot that doesn't detect that the driver is awake and operating the vehicle is going to be used by sleeping or otherwise unattentive drivers. It's just human nature.
So what likely happened was the driver was screwing around and the autopilot completely failed to notice a giant white truck crossing the road and blocking their way until it was too late.
edit: reply is quite right, shouldn't speak ill of the dead that way. "Screwing around" is unfair. I just mean that "driver falls asleep/paying attention to other things" is the expected outcome of any autopilot system, and so it's unreasonable of autopilots to have any expectation of drivers for attentiveness.
For all we know the driver had half a second to react before he died. Or maybe he had five minutes. The point is, we don't know. To start from that position and say it's "likely... the driver was screwing around" is pretty crappy.
In fact, I'm judging the software, if anything. Any autopilot software that doesn't force the driver to be attentive is effectively taking the responsibility for driving onto themselves, and so cannot blame the driver for being inattentive. You can't say "I got this" and then at the last second say "oh shit, I need you driver!" even if you put warnings that you don't really got this, because you're dealing with human beings who do normal human things.
Yes, I shouldn't have said the driver was screwing around, you're right. But if the driver fell asleep, and a conscious driver would have avoided this accident? I still think it's Tesla's fault, because he'd be awake and alive in a manual car.
I'd recommend trying it before you knock it. I had similar reservations, but after using it for nearly a year, I find that I'm more attentive and more awake with it than without it. A couple of weeks ago the system refused to engage (broken sensor somewhere) and I had to drive 80 miles home manually. It was exhausting! The mental effort of staying between the lines and maintaining a safe distance from the next car is pretty significant. Getting rid of that frees up more of your attention for the bigger picture. Now, I'm sure for some people it just frees up more of your attention for texting or sleeping, I'm sure I'm not representative of everybody, but I think it can easily be a net win for many.
I always pay attention while Autopilot is engaged, but I'm able to look farther down the road and take more care when checking out traffic to the rear and sides than I can when I'm doing it all myself.
Didn't know about this. If they are truly testing their software in a car without break and steering wheel I have a whole new level of respect for them.
When you start to design a self-driving car the question of a human fallback shouldn't even exist. The car either drives itself, or humans do. Mixing responsibilities in situations of life and death doesn't end well.
A Tesla Model S P85D can brake from 60-0 in 102 ft. (see http://www.edmunds.com/tesla/model-s/2015/road-test-specs/)
Assuming the driver is perfectly attentive, and hits the brake immediately upon noticing the truck, they would still need at least 139.84 feet to come to a stop.
This also assumes the truck is no longer moving. If it is you'd have even less distance to stop.
I don't care whether you're waking up from sleep, or completely alert and instantly react, it's a huge assumption to think that any possible reaction would have prevented an accident or fatality. It's just as likely that even had autopilot engaged the brake immediately, the truck still would have plowed into the Tesla and caused a fatality.
tl;dr - two way traffic involved in a head on collision is a dangerous game, made doubly dangerous because of the combined speed of both vehicles working against inertia.
> When drivers activate Autopilot, the acknowledgment box explains, among other things, that Autopilot “is an assist feature that requires you to keep your hands on the steering wheel at all times," and that "you need to maintain control and responsibility for your vehicle” while using it. Additionally, every time that Autopilot is engaged, the car reminds the driver to “Always keep your hands on the wheel. Be prepared to take over at any time.” The system also makes frequent checks to ensure that the driver's hands remain on the wheel and provides visual and audible alerts if hands-on is not detected. It then gradually slows down the car until hands-on is detected again.
RTFA. It mentions explicitly that autopilot checks for hands on the wheel.
Perhaps the name autopilot exerts to much confidence in its capabilities.
How is this not an "Autopilot did something very very wrong" story?
Tesla autopilot drove the vehicle full-speed into a tractor-trailer, which wasn't registered by its sensor systems. It's great that Tesla already has an explanation for the failure, but the situation as described is far from an improbable edge case. What they are saying is that the autopilot can't detect white objects when a bright light is present -- that's a pretty serious limitation to overlook.
Still, I agree it is strange that a white/glossy object cannot be detected with radar. Is it a stealth truck?
> CAUTION: SOLAR GLARE IN THE A.M.
It's serious enough to warrant its own road sign, but its not as much of a realistic issue on a divided highway, so maybe this road didn't have such a sign.
This type of danger is probably what the article is describing.
I guess I'd like to see an NHTSB simulation of the situation in manual driving to see how humans do unassisted.
It could be even worse. If there was a "road surface mirage" then it's possible that both the truck and the road underneath it were not visible at all. Since the forward radar doesn't have the range or coverage to see a truck at a distance, the only sensor that could have detected the problem was the camera. Because of the phenomena I've mentioned above and the parent post has mentioned, we know this can't possibly work.
Even worse, the circumstances in which the car's sensor could fail are precisely the ones in which the Mark 1 Eyeball would also fail.
I'm sure you'll say, "yes but that's a driver attention problem, not a technology problem", but to me they're the same thing. If you give drivers technology that encourages them to not pay attention then accidents caused as a result of that are partially your fault.
If you want an actually applicable analogy, I'll pull one I've used before. Say you buy an oven that has this great auto-baking feature that lets you just put some cookie dough inside it and then come back whenever you want, even hours later, to have fresh baked cookies waiting. That's how it's marketed - the AutoBake Oven. Except the manufacturer says that you're still supposed to watch the oven at all times in case something goes wrong. So you make some cookies with it and they turn out great, and nothing goes wrong. And hey, it's the AutoBake Oven, right? It says it will bake things automatically so you don't have to worry about them for hours at a time. So you logically decide to put some dough in there and then go the grocery store for a few minutes to get some items for dinner. And you come back and your house burned down because the AutoBake Oven lit itself on fire due to having subpar sensors and beta software in it. Now tell me you wouldn't file a lawsuit against the manufacturer.
Especially after hearing about incidents like this, you won't find me going through my email while letting "autopilot" do its thing. I don't think I would before this, but there's certainly a lower probability that I would now. I'm being trained.
The autopilot will avoid obstacles it can detect and the driver is expected to avoid others (since he has eyes on the wheel and eyes on the road the whole time).
I think they misleading thing about Teslas Autopilot I its current state is that they call it "Autopilot" while other manufacturers call this kind of assist features e.g "Lane Assist".
A distance-sensing cruise control can't be trusted not to crash into the back of the vehicle in front either, not least because it can't be sure it can "see" it. I suspect motorcycles and other small vehicles may fool radars just like this case seems to have been about contrast and colors in a camera sensor.
I'm sorry, how is that supposed to work, exactly? How does the driver know? Sounds like the driver has to avoid every obstacle - to avoid the case where the autopilot fucks you, like this one - and in that case, what is the autopilot doing?
Have you driven a car with a distance-keeping cruise control? It's very comfortable to let it do that and just watch it and take over when it fails (it's limited in the braking and acceleration it is allowed to do). I can't take my eyes of the road or my hands off the wheel for even a second but it's still useful because it removes all the micromanagement of throttle and brake and just lets me supervise.
The "lane assist" (what it should be and is called by other manufacturers) spots any number of obstacles that the driver failed to see - takes over - and saves the driver from an accident. Or reacts faster than the driver could react - thus saving the driver. That is what it is doing. It is not driving your vehicle. It is assisting in driving your vehicle.
It is still possible to drown with a life vest on. That doesn't mean you shouldn't wear one because "if I'm expected to stay above water - what is the life vest doing?"
Second, you're completely ignoring that autopilot will very much entirely drive the car by itself for minutes at a time (reports are anywhere from 5 to 15 depending on software version), which will obviously encourage drivers to not pay attention to the road. I don't care about disclaimers, I don't care about nag screens or chimes, and I definitely don't care about some warranty text that flashes on the screen. If you make a car that can drive itself you encourage drivers to let it do just that and you should be prepared for that eventuality. Period.
Tesla risks getting regulators to crack down on new car features across the entire industry if they're not careful.
Tesla sure has balls to pick the name "Autopilot" when it really should be called "CoPilot" and that would have saved a lot of trouble.
If only someone could have foreseen these issues...
http://www.theverge.com/2016/4/27/11518826/volvo-tesla-autop... http://www.thedrive.com/news/3753/former-google-exec-and-ai-... http://www.thedrive.com/tech/3909/volvo-rd-chief-teslas-auto...
I'm sorry, but that's a cop-out. What's the point of an autopilot if you have to avoid every single obstacle and react to every single thing as if the auto-pilot didn't exist? You can't say "autopilot is safer than normal humans" (heavily implied by the blog post) and then turn around and say "but it's really the driver who's responsible for everything!" anytime there's an accident.
Unlike autopilot, cruise control has very limited functionality. No driver expects it to do anything other than keep their speed, so they are still actively engaged with the road to respond to anything that takes place.
You don't have to react - you have to supervise it though. It lets the driver monitor a system that micromanages steering, brakes and throttle, instead of doing that micromanagement himself. It requires the same attention but is less work.
> Unlike autopilot, cruise control has very limited functionality. No driver expects it to do anything other than keep their speed,
Well, these days you can often set it with a distance to the car in front using a radar. You then let the cruise control do something more than just keep constant speed, it will slow down if the car in front does etc. The driver has to pay full attention though for anything the cruise control can't handle.
You seem to imply "what's the point of an autopilot if I can't take a nap". - well the issue is that it's a little too good to be called a cruise control, but not good enough for a nap. We're simply not there quite yet. At least not legally.
If you're constantly responding to every change in conditions, what's the point of the autopilot? So you don't respond every time, and then something comes up that the autopilot doesn't recognize, and you don't know that until it's too late.
The same as a cruise control - you are supervising rather than micromanaging because micromanagement is more work than supervision. A distance-keeping cruisecontrol may be changing the throttle level dozens of times per minute. A driver supervising that system can keep an eye out for stalled cars on the side of the road, wildlife etc., withput using most of the attention on keeping a constant distance to the vehicle ahead. This complements the driver and increases safety because the drivers cognition can be used to what it's best at, and simple tasks like keeping a constant speed or constant distance can be done by a computer.
The problem with this "Level 2.5" autopilot is twofold:
1) It's called "AutoPilot", even though that shouldn't be the name of it until it's feature complete (level 4 autonomy) simply because people seem to assume that AutoPilot means something that drives by itself.
2) It appears good enough to allow the driver to not pay attention, while still requiring the driver to pay attention. The disclaimers that the driver can never be distracted for a short momet (i.e. use it like a pilot uses an autopilot) don't seem to help.
A lot of the comments here seem to indicate "if I can't text/sleep/whatever then what's the point of an AutoPilot" - clearly misunderstanding the current capability of the system!
I know what my cruise control will do (effectively nothing). I don't know how to anticipate what Tesla's sensors and "AI" can't handle.
And even though actual autopilots in planes are monitored by meatpilots, meatpilots can still take their attention away for a short time. The autopilot keeps the plane flying straight and level, and doesn't take on any navigational role (in most cases). Unless your roads are dead straight and there's no traffic, driving requires a constant navigational role.
Indeed. Tesla wanted to use that name now even though it's the proper name for a feature not yet complete. They have to use a ton of warnings and disclaimers to do it. We'll see if authorities find that sufficient. If I were tesla I'd rename it to something with "assist" for the time being.
> And even though actual autopilots in planes are monitored by meatpilots, meatpilots can still take their attention away for a short time.
So does Teslas autopilot, but the time you can safeley take your attention away on a highway may be perhaps a second or two if you are on a clear straight with wildlife barriers. The autopilot will ensure the car runs straight while you fiddle with the radio.
The same time in an airplane may be ten minutes. THAT is the difference. In an airplane theare are controllers making sure there is no traffic one in front, and there is no risk of a deer or a child stepping in front of the plane. Basically the AutoPilot will have the meatpilot respond to anything it can't handle too - which in an airplane is mostly tecnical issues.
The tractor trailer drove across the divided highway without seeing an oncoming car. I've seen this accident happen in front of me with a fatality, no auto-pilot involved.
A vehicle moves in front of you when you're sufficiently close and you've no time to react. Anything (like glare) that makes such a vehicle harder to spot reduces your available reaction time.
Making it sound like there was a static obstacle in front of the car, and it drove into it at high speed is very tendentious, and not warranted by the report, I feel. It is exactly the kind of 'omg AI kills driver' fearbait that we don't need.
If self-driving systems were ten-times safer than people, then there'd still be a lot of people being killed in them. We need to get a sane perspective on their risk.
It's unrealistic to assume that human drivers are attentive all the time. Autopilot never gets tired, never eats, never applies makeup, never sneezes, and never gets distracted by screaming children.
You have to compare autopilot to realistic humans, not your platonic ideal of what a driver should be. I'm sick and tired of people making statements about road transit predicated on the idea that real humans pay attention to the road with 100% fidelity at all times.
Yes, this is an unfortunate fact of life that human beings are no angels.
However, most modern societies agree that reckless drivers are to be held accountable for their less than perfect behavior. If a police officer (or more recently, surveillance camera) catches you while DUI, or texting, or some other of the most blatant irresponsible behaviors, they will give you a big ticket. And if the consequences of your own idiocity happen to materialize you will suffer much sterner punishment (up to criminal charges for manslaughter, if it came down to that).
That's assuming the infractor does survive. It is a known fact that a non trivial amount of people responsible for accidents end up loosing their lives (or those of loved ones) in the very tragedies they caused themselves. Death is the ultimate judge, and a merciless one at that.
> You have to compare autopilot to realistic humans, not your platonic ideal of what a driver should be.
Why?
Neither the engineers who created those subpar systems - nor the executives who pushed for an unfeasible schedule, nor the shareholders that benefited from the whole deal, - will ever face the same consequences as described above. Corporations raison-d'etre is precisely to shield people engaging in economic activities from the unintended consequences of their actions; that's why they must be held to a higher standard than the everyman down the road.
Would you imagine Tesla's Board of Directors having approved the green light for this impressive-yet-still-limited technology, if the company was required to pay for every traffic ticket that every Model S accrued while being in autopilot mode, nationwide? What about being accountable for every accident, no disclaimer licensing or such nonsense?
> I'm sick and tired of people making statements about road transit predicated on the idea that real humans pay attention to the road with 100% fidelity at all times.
People will make statements about road transit, because it is literally a life-or-death situation. Maybe you do prefer the cattle to shut up and die quietly, so your utopic techno-fantasies are never challenged?
Are you more interested in saving lives or in expressing the short sighted and sanctimonious outrage of a Luddite?
How many lives do you think autopilot is going to save if being as good as a sleep deprived teenager is good enough? And who benefits from that state of affairs?
Anyways, you are a true beliver or a paid shill, so have a good night. EOM.
Since you impose unrealistic standards on progress in an attempt to bolster your own moral position
Driving at speed, you have to make assumptions that other people will do the right thing. You pass cars all the time. If you assume that the car might side-swipe you, you wouldn't get far, and you'd be a hazard yourself.
You drive on undivided highways all the time without driving as if the oncoming car might cross in front of you. And so on. We have to assume that others are minimally competent. That's the point of road laws, not just to control you, but to let you assume things about others.
Where I'm from if you make a left turn and get side-swiped then it's your fault, period. You made an unsafe turn, plain and simple[1].
If what you describe is true, then human error is the main cause of the accident (although the AI failed to prevent it and absolutely should share the blame). I just think "AI kills driver" is a bit sensationalist don't you?
[1]Of course assuming that all other vehicles present were obeying the law (e.g. not speeding) and there were no other extenuating circumstances.
On average, humans are rather unsafe car drivers.
This peculiar wording is making it sound like the computer made the same mistake as the driver, but it seems much more likely that they made two different mistakes. The computer's mistake was in image recognition, and the person's mistake was trusting the autopilot fully and not paying attention to the road.
The optical distortions that cause road surface mirages could well entirely hide another vehicle. In this case, since the trailer was painted white, only the wheels and lower chassis would need to be hidden for the image to become unrecognizable. Not saying that's what happened in this case, but there are situations when your forward vision could be compromised. (There is a theory that the colder ocean may have caused a similar optical phenomenon and hidden the iceberg that sunk the Titanic.)
It is important to note that Tesla disables Autopilot by default and requires explicit acknowledgement that the system is new technology and still in a public beta phase before it can be enabled. When drivers activate Autopilot, the acknowledgment box explains, among other things, that Autopilot “is an assist feature that requires you to keep your hands on the steering wheel at all times," and that "you need to maintain control and responsibility for your vehicle” while using it. Additionally, every time that Autopilot is engaged, the car reminds the driver to “Always keep your hands on the wheel. Be prepared to take over at any time.” The system also makes frequent checks to ensure that the driver's hands remain on the wheel and provides visual and audible alerts if hands-on is not detected. It then gradually slows down the car until hands-on is detected again.
This is the first known fatality in just over 130 million miles where Autopilot was activated. Among all vehicles in the US, there is a fatality every 94 million miles. Worldwide, there is a fatality approximately every 60 million miles.
The wording implies that these numbers are directly comparable, but they are not. I would guess that Autopilot is much more likely to be activated in safe situations. But the other statistics include all situtations.
Not only that, you'd expect drivers in the higher-income demographic of US auto customers to have less fatal accidents.
The real comparison should between a model S and cars like the Mercedes S-Class, since this is all about car marketing anyways.
What? Why?
They also might be more sheltered from the consequences of their actions and likely to be reckless while driving.
Your assumption may even be correct, but I don't think it's obvious.
In the end, the accident rate comparison to the global population is completely wrong.
Which makes it a better comparison to Teslas, really.
http://jalopnik.com/the-tesla-model-s-just-got-upgraded-to-l...
Telsa isn't one of them...
As they say, it's not the fall that kills you. It's the sudden stop at the end.
Also, if I were the target of the PR shenanigans pulled on Tesla, I would want to use precise language, too.
Do you think that reassures Tesla owners?
It's also a reminder to engineers to have an extremely high bar of excellence.
EDIT: thanks for telling me I'm wrong. This was a self-serving blog post on their part to damage control.
Imagine how many press releases there'd be per day if another auto company felt compelled to do one for each accident their cars were involved in.
This isn't just another horrible car accident on US roads, the car was in charge this time and a technology they've been heralding for it's safety is going to be blamed. That's why there is a federal investigation into it. That's why there is a press release to protect the stock price.
Other auto accidents are non-interesting to the public, but this one involved automated driving.
The Tesla suspension issue with non-disclosure agreements also made front pages, just a week or two ago.
http://www.cnbc.com/2016/06/11/tesla-revises-nondisclosure-c...
Tesla's blog is their primary marketing / damage control arm. Anything they publish there becomes instant news.
This is as bad as it sounds.
On the other hand, Tesla doesn't do this for every accident. They are treating this one special because Autopilot very carefully, and in a fully controlled manner, drove the car directly into a semi-trailer, which is more than a little embarrassing for the company. And terminal for the driver.
What other companies use this kind of driving assist system? The ones i know are only hinting to the driver , and disengage if the driver does not respond.
Got to be careful about these things. You don't put beta stuff in cars. It's like your doctor making drug trials on you. They should figure out a way that autopilot engages the driver and keeps his eyes on the road.
They've also, FWIW, gained quite a bit of positive press. https://www.youtube.com/results?search_query=tesla+autopilot...
This is literally the only way to develop such systems. And regardless of incidents like this, these systems will be developed sometime in the fairly near future and whichever company is on the leading edge of them will command a massive new market that will probably start growing incredibly rapidly right after it takes off.
Just because something is in development longer doesn't mean the quality is better. Furthermore, Tesla might be gaining some critical information with their early release. And whose to say Tesla didn't test this stuff to the best of their abilities?
'Significantly lower accident rates' compared to what, exactly? The highly accurate accident rates collected from the many decades of experience with Tesla accidents that never happened?
Now compare that to Tesla - beta software and sub-par sensors that can't detect stationary objects in the road (there was a recent case where a Model S on Autopilot hit a stalled car on the freeway despite plenty of time to stop), can't detect obstacles at windshield level (there was a recent case where a Model S hit a parked trailer when summoned), and apparently have trouble distinguishing between sky and lightly colored objects in full daylight (the most recent accident). This is a company that is prioritizing technology releases ahead of human lives.
Significantly lower accident rates compared to Model S Autopilot, if that wasn't obvious. In the future, you will take accidents per miles driven in autonomous mode and compare between Tesla, Google, Apple, Uber, etc. and then see which one has the lower rate. I am saying that Tesla will be the worst.
But as a human driver, I would have taken human psychology into account and guessed the turning car was going to take advantage of that gap, thus causing me to approach much more cautiously.
I don't know how fast that tesla was going, but I don't think I'd be doing more than 5-10 mph in that kind of traffic. I would also expect cars in the right lane to unexpectedly switch into the left. It seems to me the Tesla was overconfident and going about 25 mph.
edit: I feel like my tone here is too harsh (mainly because my personal reaction is that this autopilot feature is professionally irresponsible). Let me pull back my emotions a little and try again:
The only reason the autopilot feature here "saved the day" is because it was driving in an irresponsible manner in the first place. As the comment below me points out, it was driving at 45 MPH, compared to the 5-15 MPH you should be driving. For a defensive human driver, there shouldn't have been a close call in the first place.
The Tesla is driving irresponsibly because it cannot take into account human behavior. It stopped in time in this scenario, but there's obviously a point at which the stopped car could turn and the Tesla could not physically stop in time. This can happen at any speed, but the 45 MPH collision is much much worse than a 10 MPH collision.
I'd also like to point out that the autopilot was likely breaking the law. The speed limit on that road is probably 55 - 65 MPH, but the speed limit indicats the maximum speed in perfect conditions. When it's dark and raining and heavy traffic, going the speed limit is in fact speeding. If the Tesla had hit the turning car, the driver of the Tesla would probably have been cited for speeding, even at 45 MPH.
Further, it would have been common courtesy to stop and allow the car to turn and take advantage of the temporary gap in traffic. People allow me to turn in situations like this all the time, and I return the road karma when I can.
Overall, it seems like quite a failing despite the avoidance of the accident.
All those factors indicate the car was going too fast given the conditions -- autopilot or not. It appears that in general, the Tesla AP does not take road and environmental conditions into account (or weigh them heavily at least). In this case, there was no collision in spite of the conditions. But this should have been a giant red flag for Tesla engineers. Unfortunately, the story at hand tells us that they're not doing their job.
> Watching the video the first time, I didn't see the car before it was already in front of the camera.
I didn't either, not at first. But given just the first frame and knowing a collision was about to be avoided, my first reaction was the autopilot was going to prevent a rear-end due to a short stop, or it was going to stop as someone merged in from the right lane.
But you and I are not the driver of this car. The driver should have known the highway was not divided, should have known of the possibility for cross traffic, and he should have adjusted his behavior accordingly. I guarantee you there was also a sign (before the start of the video) indicating there is an opportunity for cross traffic ahead. Given that, those white headlights (they look reddish in the video, but in real life they would be bright white) should caution the driver that a car is probably looking to turn.
The autopilot did not take any of this context into account, and that's exactly why someone is dead today. It's operating with superhuman reaction speed, but with the common sense and wisdom of a 10 year old.
inattentive faux-professional driver, chatting up a client (gotta get that 5-star rating) instead of watching the toad, driving too fast in dark, wet, complicated conditions (line of stopped cars to the right).
I'm intrigued that the color is relevant in the car's case - wouldn't it be using some sort of radar to detect and map objects rather than vision? I appreciate I am probably missing something.
The 360 degree sensors on the Tesla are for parking only (< 20 feet range).
Personally when I worked on a self driving car we were going with 360 degree camera and doing path planning based on that, but Tesla has opted not to do that.
The radar is low to the ground and probably doesn't pick up a trailer that's high off the ground. The camera could, but not if contrast is too low. (And I'm not sure if the software is able to recognize the side of a trailer anyway.)
The only way to see it would be lidar (it might have issues too) or radar. Ultrasonics don't have the range.
They should have the autopilot refuse to operate when the sun is within X degrees of the road in the direction of travel.
From what I read about humans, you need +/- 30 degrees up/down vision to get a license (varies by jurisdiction). And we kill a little over a million people each year on roads. Not sure what this should imply about a car's radar though.
It's going to get better, but it's crazy to assume it's currently much more than a highly-assistive cruise control.
Maybe they had to do it for legal reasons I don't know (I'm certainly not a lawyer) and I'd love to own a Tesla but couldn't they have worded this a little more sympathetic and a little less lawyer?
Already down $6/share in after hours trading.
So long as every Tesla accident makes the homepage of CNBC, unlike the millions that occur in other makers' vehicles.
Tesla puts out an immediate PR, and the breaking news that went up on all the financial sites included quotes from it nearly as fast.
http://www.cnbc.com/2016/06/30/us-regulators-investigating-t...
Which is about right.
Table 28 of http://www.cdc.gov/nchs/data/hus/hus15.pdf#019 is relevant to this discussion
The Tesla Model S will now never be one of them, and in fact, is now among average for fatality rates that includes motorcycles, drunks, and other dangerous factors. It's now not even among the class of luxury cars that have very low driver deaths.
Right now self-driving cars are far too dangerous for public use.
Which version? There is 60, 60D, 70, 70D, 75, 75D, 80, 80D, 85, 85D, 90 and 90D so which one are you talking about? Tesla calls their cars Model S the same way Apple calls their laptops Macbook.
It seems fair to me.
Unfortunately, fortunately, Tesla is having to educate people and being very clear to not allow room for panic and unwarranted fear mongering.
I would say it is as to the point as it can be, and that it is heartfelt.
Notice how quick they are to bring up "this is the first fatality in 130 million miles", but AFAICT, traffic fatalities in the US occur at a frequency of around 10 per billion miles [1], which is the same.
So even though this is just one data point, it's spot on the current average for fatality frequency in normal cars.
Furthermore, to gather enough statistics to be able to say with confidence "semi-autonomous Teslas are/aren't safer than normal cars", Tesla either needs to increase their sales volume by several orders of magnitude, or we'll have to wait for ten years.
[1] https://en.wikipedia.org/wiki/List_of_motor_vehicle_deaths_i...
If we had data for the "high end modern full-size car" category, it would most definitely be lower than 10 fatalities per billion miles.
A normal car drives about 100 000 miles in 10 years. You then need one million autonomous cars driving around for a decade before you even know whether they're safer!
Maybe a billion miles are needed for a fatal crash, but for small mistakes it's much less.
Can you give a source for that number as it seems a bit low given warranty numbers.
"Among all vehicles in the US, there is a fatality every 94 million miles. Worldwide, there is a fatality approximately every 60 million miles."
If you want to do the numbers to compare the Tesla to other cars, where they measure driver fatalities per million registered years, do this:
130,000,000 miles / 12,000 average miles per year per car = 10,833 years.
That's 10,833 registered years for the Model S.
The IIHS site normalizes to deaths per million registered years, so multiply that number by 92.3, and since we have one death so far, the Tesla Model S is now at 92.3 driver deaths per million registered car years.
Let's compare to other car models:
So, right now, it looks like the Tesla Model S is about 4x deadlier than the average car when using Autopilot, and about 10-20x deadlier than the safest cars.How you can draw such a conclusion from a singular data point? That's beyond ridiculous.
> This is the first known fatality in just over 130 million miles where Autopilot was activated. Among all vehicles in the US, there is a fatality every 94 million miles. Worldwide, there is a fatality approximately every 60 million miles.
"4 Times Deadlier!" certainly comes across as sensationalist based on such flimsy data (never mind that the real figure is actually 3.2 and the parent chose to round up rather than down to serve his/her purpose).
Whereas Tesla's statement comes across more as a statement of fact (albiet quite self-serving.. no question about that).
Auto-pilot is semi-autonomous... I gather you consider that a risk...
What about Subaru EyeSight? It can autonomously brake. I can drive for hours on the freeway without ever hitting the brake nor gas, even in stop and go traffic. It has warned me multiple times of a sudden slowdown just as I noticed it. It hasn't yet saved me from anything, but if any of those circumstances I'd been distracted (even for a legit reason) it could have. It once warned me of a motorcycle in my path on a very dark night when the dim tail-light of the motorcycle was overwhelmed by the much brighter tail-lights of the cars to its left and right. I would have seen it soon enough, but the car saw it first. Is that a risk to other road-users? Yet, it's automated by some definition.
What about various other manufacturer's lane keeping features? That automation to some degree.
What about ABS? Stability control? Again... automation to some degree.
All of these have their detractors, yet I think statistics clearly show an improvement with each level of driver assist.
I would assume that the higher-end GM cars do have an automatic braking collision avoidance system by now, since that's a pretty standard feature above a certain price point. And I would assume it's not foolproof either, but at least it doesn't lead the driver to think he can take his hands off the wheel and not pay attention to what's happening on the road.
Tesla reiterates every time and makes you confirm you understand that this is not the case. If the driver did it, that's completely on them being irresponsible.
And pilots still make mistakes when adjusting to a context where the autopilot has unexpectedly handed control back to the humans.
there is no such feature, there is a MobilEye _lane following_ system Tesla repackaged under Autopilot(TM) brand.
Say it out loud - lane following. Mercedes sold this same tech almost 10 years ago, difference is Tesla felt pressure and started believing their own marketing lies about autonomous driving.
Although US foreign policy and intelligence decisions were more or less directly responsible for 9/11, you don't lecture people at Ground Zero about the dangers of blowback and not being aware of your government's activities.
There's no such thing as "condolences, but...", and Elon Musk has shown a pattern of this kind of tone-deafness before. Not to say he's not brilliant and not to say autonomous cars won't ultimately save many lives, just that he should either listen to his PR department if he overruled them, or fire his PR department if they did this without him.
Excellent, Doc. Further to the point about these statements and Musk's leadership: not too long ago Tesla would include the fact that nobody has been killed in a Tesla in their public statements. I found it completely inappropriate since we all knew it was just a matter of time, sadly. Again they play with fire, as another death with autopilot will really wreck their statistics.
one failure and they will take a minor publicity and money hit, two and its going to devastating
Now, this is pure speculation, but I really can't imagine running into the side of an 18-wheeler crossing the highway unless A) I was traveling at a reckless speed (unlikely given that "auto-pilot" was on) or B) I wasn't paying attention to the road.
https://img.washingtonpost.com/blogs/the-switch/files/2016/0...
IMO, human (customers') expectations require more than a mere autopilot.
I think they did a fine job of handling this, and with the high visibility of this incident due to the use of autopilot they really had no other option.
Edit: Given the down votes, I guess people really want the mental picture. Sad.
They have to defend autopilot not only to protect the brand but to protect the public's perception of autonomous vehicles in general.
Self driving tech is poised to save many many lives. So from a utilitarian perspective, it's probably justified to take extraordinary measures to make sure reactionary media and public whim doesn't kill it off, however uncomfortable that might seem in the short term.
Whilst this case is incredibly sad (and I don't want to downplay that in any way), if you're trying to minimise the overall amount of fatal crashes, exonerating the tech is the priority (if it is truly not at fault).
2. From the rest of this thread, it sounds like their long-range cameras/radar has very limited FOV.
I personally can't wait for fully self-driving cars, but it's going to be a big PR battle in the US
> They have to defend autopilot not only to protect the brand but to protect the public's perception of autonomous vehicles in general.
They could have done that in two separate posts. This was tone deaf at best.
Why? Why not just build the cars people want? (including cars that people want but don't realize they want yet)
There shouldn't be a political agenda associated with engineering. Build what is needed. Build what people want. Build what people will need. But never "defend my reputation and the reputation of this device that I'm making"
EDIT: I mean, I get why Public Relations are important and so forth. So Tesla is certainly free to do what they want here. But lets not pretend that this carefully crafted "condolence" piece that has come out roughly one and a half months late is anything but damage control for this company.
The computer would have to be 99.99999% reliable to do that.
The accident rate is around 74 per 100 million miles (and fatalities is 1.13).
It's unclear exactly how to turn that into a percentage, but no matter how you do it it's quite high.
Say an accident takes 5 minutes, and people drive 30 miles/hour. Then that works out to 99.999% for humans. If you use the numbers for fatalities then it's 99.99999%.
I.e. 99.99999% of the time, as whole across all [US] humans, people drive in a way that does not cause a fatality.
That's the bar computers have to cross in order to save any lives at all.
Additionally, 92 people are killed in fatal car accidents in the US every day. So it's not as though this is some uncommon occurrence that autonomous vehicles would be unlikely to improve.
It most definitely is not above that bar! The autopilot is combined with a human operating the car.
How many accidents were avoided by the human rather than the autopilot?
I suspect a ton of accidents (i.e. the autopilot has a high error rate).
How many did the autopilot avoid that the human would not have? Probably not that many, if any.
> Additionally, 92 people are killed in fatal car accidents in the US every day. So it's not as though this is some uncommon occurrence that autonomous vehicles would be unlikely to improve.
Do the math.
I did.
0.00001% seems pretty uncommon to me. There is a lot of driving in the US, so even that low of a number is visible, but it's still a low number.
Remember every single computer must be all but perfect to reduce the error rate.
Have you ever seen a computer to be that good? At even something as simple as not crashing? Never mind driving a car.
Or probably it avoided 100,000 accidents!
Unfortunately, we are both just guessing.
So please keep your math out of this.
Except I am not guessing. That's the point of math.
The autopilot plus a human, only did slightly better than a human alone. That means the autopilot did nothing, since slightly better is well within the range of normal for a human alone.
> So please keep your math out of this.
What a strange reply. Why keep math out of this?
Are you hoping that this will be real if you ignore all evidence to the contrary?
Right now the evidence is in: Computer assisted cars don't do anything helpful to the accident rate. This bodes poorly for self driving cars, and since the error rate they have to hit is so low, it's really not looking good.
I personally don't expect self driving cars to ever be used on regular streets. Only on computer-exclusive roads, specially marked for them.
You have to understand the magnitude of the perfection needed to have any hope of implementing it.
Tell me: Before you read my message, would you have assumed at a computer that is 99% perfect, or 99.9% perfect, would be better than a human?
I can tell you, that until I did the math, I thought so myself.
But 99.99999% is 10,000 better! If the computer was 99.9% perfect you would have almost 1 million fatalities per year (assuming things scale linearly, which I'm sure they wouldn't, probably most of the time driving is easy and 99.9% would still not get into an accident).
> It's a regular, daily occurrence for 92 real human lives every day.
I know. (Although you said that badly: it's not a daily occurrence for those people. But I get your emotion.)
But a computer will not solve this problem, not for a long time. We simply are not able to make a computer that is that good.
Let's see if we can make a web browser that is that good before we try to make a driving computer that good.
It also puts human drivers in a new light. I was of the camp that people are terrible, horrible drivers that kill all the time.
But actually humans are nearly perfect at driving, it's just there are so very very many people driving, so even a tiny cumulative error shows up.
Based on what?
Human drivers are very bad. We kill over 32,000 people a year (and that's a 60 year low, just a few years ago it was over 40,000).
So if self driving cars are just a little better than humans, they will save many lives. If they are a lot better than us, they will save many more.
Bad compared to pretty much every other form of transportation that humans employ.
That's a hypothetical. You say autonomous cars are "poised" to save many lives -not even avoid many deaths, actively "save".
That's very strong language to use based on an "if" that nobody yet knows the answer to with any certainty.
That is, unless you are privy to some information that the rest of us are not? Are you?
What is it that you base your assertion on? Do you base it on anything or is it just a pretty turn of phrase you thought sounded cool?
There are already videos of assisted-driving cars avoiding accidents which likely would have been fatal. We already know they can potentially save lives. It's something we already accept for flying. Autopilot in planes has resulted in deaths of crew and passengers. So we have precedent in at least one mode of transportation.
Drivers get T-boned or rear end other people all the time. Those are two scenarios where driver-assist could have applied the brakes and actively saving lives for drivers inattentiveness. You'll also find no shortage of Liveleak clips of people being killed by drivers who failed to see them and brake in time. Many being scenarios that Google's self-driving cars have the technology to avoid.
Autopilot in a plane doesn't have to deal with obstacles besides the odd mountain here and there.
So I'd say that in many (most?) ways it's easier and safer to autopilot a plane than a car, and as you say, even planes with autopilot have failed.
Planes most certainly do have to deal with wildlife (e.g. birds) appearing suddenly in their path regularly.
In other words, the plane autopilot doesn't "deal" with wildlife at all unless you count running into it. Which is apparently what Tesla autopilot does as well.
I'm not concerend about assisted driving. I'm concerned about fully autonomous cars being handed complete control in real-world situations.
I'm not just "concerned" about them- they scare the hell out of me.
>> Many being scenarios that Google's self-driving cars have the technology to avoid.
What Google says it programs its cars to do, what it really programs its cars to do and what its cars can really do are all separate things.
The problem is that the current technology level is nowhere near advanced enough to allow fully autonomous vehicles to operate safely. There is a huge number of situations that those cars aren't programmed for, that they can't be programmed for, because those situations are completely unforseeable.
Machine learning has a huge problem with data sparsness. You may train a learner in petabytes and petabytes of data collected from the real world and still miss the vast majority of events that may occur.
That is why, like I say elsewhere, machine learning-based AI makes utterly ludicrous mistakes that humans would never do, even in difficult situations were they can't be expected to perform with 0% error. I've used a few metaphors- here's another one: a human would never mistake a truck for a cucumber. A machine learning algorithm, might.
And what's worse, there's no way to prevent this sort of mistake, or even correct it, because most of the time the models built by such algorithms are simply too complex to be processed by humans in the way a hand-crafted system would be (and goddess knows how hard those can be to process).
https://en.wikipedia.org/wiki/List_of_motor_vehicle_deaths_i...
From which I quote:
The number of deaths – and deaths relative to the total population – have declined over the last two decades. From 1979 to 2005, the number of deaths per year decreased 14.97% while the number of deaths per capita decreased by 35.46%.
So at least the rate of fatalities is decreasing. I won't claim I know whether that's because humans get better at driving, carse get safer, traffic regulations get safer, or any other factor, but the point is roads are getting safer already and without any autonomous vehicles on them.
[Original bit:] Again- based on what? Why do you say that? What is even your measure of humans getting "better"? Better at what? Better drivers? Better cognitive systems? Better what?
I said no such thing, you are putting words in my mouth.
What I said is that the baseline for traffic deaths is quite high, and if (or when) autonomous cars are even a little better than humans, then less lives will be lost.
I was replying to someone else and you jumped in. You can hold the accusations of putting stuff in places, thank you.
>> What I said is that the baseline for traffic deaths is quite high, and if (or when) autonomous cars are even a little better than humans, then less lives will be lost.
And I said that autonomous cars being better than humans is just a hypothetical.
I'll add that we have no evidence that it's the case. Computers are better at humans at tasks that require fast and accurate retrieval, or computation, but they'll not just magically become better than humans at driving. Somebody has to program them to do that.
So who is going to do that? Do you know how to do that? Do you know anyone who knows how to do that?
The theory is that we'll put enough (real or virtual) cars in enough training situations that they'll learn to drive on their own, but that approach has serious limitations, not least the fact that in the real world there may be an infinite number of situations that a learner will never encounter during training. In any case, with any other cognitive task that we train computers to carry out they end up making completely ridiculous mistakes, which in the case of driving will cost lives.
Until we have a way to develop systems that understand their surroundings, autonomous cars being better drivers than humans is just a dangerous fantasy.
Also don't forget that while each of us is somewhat unique in terms of driving, the cars are not, so a bug in a car's software can amplify issues even further.
There are issues also how cars will behave when they will have to interact with other self driving cars. Those problems are already difficult when writing distributed applications and with cars we don't have luxury of just letting things crash together until things improve.
Only when that's deemed acceptable.
http://www.fastcompany.com/28121/they-write-right-stuff
> But how much work the software does is not what makes it remarkable. What makes it remarkable is how well the software works. This software never crashes. It never needs to be re-booted. This software is bug-free. It is perfect, as perfect as human beings have achieved. Consider these stats : the last three versions of the program — each 420,000 lines long-had just one error each. The last 11 versions of this software had a total of 17 errors. Commercial programs of equivalent complexity would have 5,000 errors.
You say that so glibly. Have you ever done the math? I did, I posted it here:
https://news.ycombinator.com/item?id=12012166
Summary: The computer would have to be better than 99.99999% perfect to save any lives.
Yes, self-driving cars will save lives if we assume that the self-driving technology is superior to humans in every way. That's almost a tautology though.
Self-driving cars don't get bored, but they don't seem to see white tractor trailers either. Technology has different shortcomings than people do. Whether tech shortcomings result in fewer lives lost than human shortcomings remains to be seen.
It's astounding to me that HN stories about self-driving cars sit right next to stories about iPhones and anti-virus software, and tone of the comments are diametrically opposed. If you read the comments on self-driving car stories, you'd think that technology can do everything better than people. If you read the comments on stories about iPhones and anti-virus, you'd think that everything has terrible bugs, vulnerabilities, and shortcomings.
Isn't Tesla using pure vision? Google also uses lidar, which fixes the problem. EDIT: Looks like I'm wrong and Tesla uses radar.
> Technology has different shortcomings than people do. Whether tech shortcomings result in fewer lives lost than human shortcomings remains to be seen.
Technology also gets better. Ten years ago, people were as bad at driving as they are today. Ten years from now, they'll be as bad as they are today.
Now compare self driving cars ten years ago to self driving cars today. And ten years from now.
> If you read the comments on self-driving car stories, you'd think that technology can do everything better than people. If you read the comments on stories about iPhones and anti-virus, you'd think that everything has terrible bugs, vulnerabilities, and shortcomings.
Those aren't contradictory.
I assume you just pulled that out of thin air but even if you hadn't, it is completely meaningless. We have nothing to compare human performance against. Humans have been driving for more than a century. What are you going to compare that to?
As to how preventable accidents are- sure, if you remove factors like the ones you say, you'll reduce accidents, but humans suffer from those cognitive impairments on top of our extremely well-tuned cognitive apparatus. Computers have nothing of the sort. They may not get distracted, but they're rubbish at doing the thinking that keeps humans safe on the road the vast majority of the time (given that the vast majority of drivers don't get killed in accidents).
To put it plainly, if you put a brick on the gas pedal and lock the wheel, you'll reduce the amount of accidents caused by cogntivie impairments like inattention and poor judgement- but you won't reduce the overall rate of accidents.
In other words, if you hand control to a system that doesn't sleep on the wheel but can also not tell a puddle from a boulder, you will increase risks rather than reducing them.
https://www.teslamotors.com/de_DE/blog/teslas-antwort-auf-da...
Personally, I would have used a more neutral title, led with sympathies for the family, and then gone into the technical detail.
This time, there's an interesting question. Did Tesla remotely access the crash data after the crash? Did they alter any data? Is that verifiable? The NTSB will probably explore that issue. The crash data record in an airbag controller becomes read-only when the airbag fires.
If their detectors don't see a white car against a bright background, that's obviously a serious problem.
Sure, if all the lawyers will promise not to take some statement out of context and sue them over that.. As long as such lawyers exist and that's the way the legal system works this is what can be expected out of statements from companies..
So this got me wondering: are there areas where retinae still do better than cameras at certain kinds of discrimination problems? Nature still has human technology beat on hardware in a few categories (particularly joints), but I didn't expect eyes to be one of them. When I look at or near a bright light, I can see objects close to the light much better than I can in a photograph, but I always assumed this was only because I had a cheap camera.
The corresponding question is: how expensive is the camera that prevents this accident?
With these minor bugs, Tesla seems to be doing a solid job of poisoning the well for self-driving cars. I'd like to see them explain how their competitors should not be tarred with the same brush, once the political backlash hits.
It doesn't matter how many disclaimers you give before you turn on autopilot - a driver who focused on driving the car (As opposed to letting autopilot cruise) would have probably noticed a tractor driving across the road on a bright, sunny day.
It's a dangerous system. Instead of arguing about the trolley problem, I'd first like to see a car be good at making decisions that save its passengers.
Imagine if the first time someone fell asleep at the wheel and crashed, you could just tell everyone "hey don't fall asleep at the wheel". And it just never happened again.
I do hope they are able to learn more from this. This is just a bad coincidence of environment meets chance and someone died because of it.
I do think computer vision still has a but further to go. A white trailer and a white sky shouldn't be a problem. But everyday I'm impressed with how human sight can find and lock into the most obscure details. Like I've been sunblinded before and yet I knew something was in front of me because I noticed a grey shadow on grey asphalt and immediately hit the brakes.
We're almost there but not quite.
Nope, this is completely expected when unproven technology is packaged as a push-button feature and shipped on consumer technology. I don't work for Tesla, but I am a robotics engineer who has worked on exactly this technology, and I saw this coming from a mile away. Tesla should have as well.
We have seen from other incidents that Tesla's autobraking will not engage when autopilot is disabled (potentially by tapping the brakes), which is contrary to widespread user expectation -- disclaimers notwithstanding.
However, in this particular case, it was the obstruction detection that failed. Based on their blog post disclaimer, it appears to be a visual sensing system; eye-height radar-based system used by most other manufacturers would have correctly detected the obstruction.
Is there an industry standard (yet) for the efficacy and behavior that these systems must meet?
Mercedes-Benz uses two different radar sensors: "the DISTRONIC radar is configured to monitor three lanes of a motorway to a range of up to 150 metres with a spread of nine degrees, the new [DISTRONIC PLUS] 24-Gigahertz radar registers the situation immediately ahead of the vehicle with aspread of 80 degrees and a range of 30 metres." [1]
Volvo has a combination of grill-mounted 15° FOV radar and a 48° FOV greyscale camera behind the top of the windshield [2].
Volkswagen uses radar [3], but not sure where it's mounted. Honda's CMBS is a radar in the grill [4]. Subaru uses two cameras mounted behind the top of the windshield [5], and no radar, albeit radar is used with some of their other safety systems.
[1] http://www-nrd.nhtsa.dot.gov/pdf/esv/esv20/07-0103-O.pdf [2] http://www-nrd.nhtsa.dot.gov/pdf/esv/esv20/07-0450-o.pdf [3] http://en.volkswagen.com/en/innovation-and-technology/techni... [4] https://techinfo.honda.com/rjanisis/pubs/om/JA0606/JA0606O00... [5] http://www.roadandtrack.com/new-cars/news/a6852/subaru-camer...
No, it's not important to note that. You should not be able to hide behind the word "Beta" for systems that could kill people. Either you're willing to let people risk their lives on your software or you're not, and you were.
When the autopilot acts correctly 99.9% of the time, you'd get incredibly complacent and inattentive. When you don't have an autopilot, you are paying far more attention to the road.
Is non-standard ride-height a safety failure? Should trailers have crash panels to avoid drive-unders?
I've also observed human pilots on busy roads nearly colliding with obstacles when driving toward the setting sun, the visor pulled down and still trying to shade their eyes.
But robots aren't humans, and they don't have to rely only on vision, there are so many ways to "look" it seems like we should have several different ways of identifying obstacles. Different spectra at least.
Even in the confines of indoor environments with simple problems like transparent surfaces, specific problematic fabrics, and acoustic panels it is necessary to fuse several types of sensor data to provide reasonably reliable detection of the environment.
I would hope that as sensors and output analysis techniques continue to be developed and costs decrease, creators of autonomous systems would incorporate a wider variety of sensors into their products.
Assisted systems will lead to drivers paying less attention as the systems get better.
The figures quoted by Tesla seem impressive but you have to assume the majority of the drivers is still paying attention all the time. As auto-pilots get better you'll see them paying attention less and then the accident rate will go up, not down for a while at least until the bugs are ironed out.
Note that this could have happened to a non-electric car just as easily, it's a human-computer hybrid issue related to having to pay attention to some instrument for a long time without anything interesting happening. The longer the interval that you don't need to act the bigger the chance that when you do need to act you will not be in time.
[1] http://hal.pratt.duke.edu/sites/hal.pratt.duke.edu/files/u7/...
One option is the Tesla autopilot should have an indication when it approaches "low confidence" areas without disengaging, so the driver is not startled if they have to take back manual control.
https://youtu.be/Pq5LDi3-ChU?t=40m5s
I agree there can be handoffs but they mostly need to be in the vein of: I'm slowing down and turning over control in 30 seconds because there's something I don't understand coming up
If your eyes aren't at their best, you know well to go to a doctor and not be driving in the meanwhile. Will the car with autopilot refuse to start or go on autopilot if the camera/sensor/radar has an issue?
So yes you are right, its either full AI or nothing.
Asiana 214 [0] is a classic example of crashing a perfectly good airliner into a seawall on landing.
In the Boeing 777, one example of the (auto)pilot interface showing safety critical information is the stall speed indication on the cockpit display [1], warning the pilot if they are are approaching that stall speed.
Hopefully Tesla will optimize the autopilot interface to minimize driver inattention, without becoming annoying.
[0] https://en.wikipedia.org/wiki/Asiana_Airlines_Flight_214
[1] http://imgur.com/bGsFTCG
Your point is still valid, but perhaps we approach a time when over-reliance is better than all but the best human pilots (Sully, perhaps).
There, automation lowered both the standard for situational awareness and fundamental stick and rudder skills. Then, when a quirky corner case happened, the pilots did all manner of wrong on the problem: so much so, they amplified a condition from "mostly harmless" to fatal for all.
Vanity Fair has a nice piece on this accident that's easy to dig up. Good read.
Sure, to nearly every pilot that behavior is wacky, but it shouldn't have been a surprise for more than an instant to pilots who were "operating as designed."
It seems there's no free lunch: when skills atrophy as a natural response to helpful automation it requires advancement in some other skills, should the goal of an ever improving error (accident) rate be achieved.
The pilots were under the misguided impression that the aircraft would automatically spool-up the engines if the aircraft became to slow. This was a safety feature that didn't engage for a obscure technical reason. Even with a manual visual approach the pilot can still use the autothrust for landing.
A more rigorously trained pilot (eg. Capt. Sully) would have aborted the approach and performed an immediate go-around if he got below the glidepath (or too slow) below a certain altitude (eg. 400ft Above Ground Level).
The rules requiring a go-around (or missed approach) apply for a fully automated approach and landing, just as much as manually flown approach and landing.
On the road, where relative separation is much less (and there's even been talk of how self-driving cars can reduce following distances significantly, which just scares me more), the driver might not have even a second to react when he/she needs to take over from the autopilot.
The driver might have needed to react before the auto-pilot realized it needed to react (let along could humanly respond).
There are two things that I take away from this.
* Auto-pilot should probably just keep going (or bring to a as controlled stop as possible).
* It should also collect more data to hopefully warn the driver more in advance.
http://jalopnik.com/volvo-engineer-calls-out-tesla-for-dange...
"You need to keep your hands on the steering wheel at all times during autosteering, yet not crank the wheel hard enough to generate what the car thinks is an actual steering input (thereby disconnecting autosteer). I found this to be about the same amount of effort as simply driving."
I thought that was an interesting observation.
http://blogs.harvard.edu/philg/2016/06/27/smug-rich-bastard-...
You don't design a feature that invites misuse and then use instructions to try to prevent that misuse. That's irresponsible, bad engineering.
The heirachy of hazard control [1] in fact puts administrative controls at the 2nd-to-bottom, just above personal protective equipment. Elimination, substitution and engineering controls all fall above it.
Guards on the trucks to stop cars going under are an engineering control and also perhaps a substituion - you go from decapitation to driving into a wall instead. It's better than no guards and just expecting drivers to be alert - that's administration - but it's worse than elimination which is what you need if you provide a system where the driver is encouraged to be inattentive.
User alertness is a very fucking difficult problem to solve and an extremely unreliable hazard control. Never rely on it, ever. That's what they're doing here and it was only a matter of time that this happened. It's irresponsible engineering.
edit: My source for the above: I work in rail. We battle with driver inattention constantly because like autopilot, you don't steer but you do have to be in control. I could write novels on the battles we've gone through just to keep drivers paying attention.
[1]: https://en.wikipedia.org/wiki/Hierarchy_of_hazard_control
Please do, and link them here. I'd be very interested in reading about your battles and I figure many others would too. This is where the cutting edge is today and likely will be for years to come so your experience is extremely valuable and has wide applicability.
https://news.ycombinator.com/item?id=11017034
http://www.autonews.com/article/20160321/OEM11/160329991/toy...
They're adding new features to inadequate/improperly configured hardware for what they're asking the car to do, and waiving away all liability for stupid peoples' actions with disclaimers (always be ready to take over).
Whether that's right or wrong is really subjective, especially when you take natural selection into account.
Tesla's (only!) radar sensor is located at the bottom of the bumper, if I'm not mistaken. Compare this with Google's, which is located in the arguably correct position, the roof. Also compare other manufacturers' solutions that are utilizing 2-3 radar sensors, as well as sonar.
https://forums.teslamotors.com/forum/forums/model-s-will-be-...