This entire article is based on the premise that two very different situations are directly comparable.
In the former, people sometimes accidentally stomp the gas instead of the brakes, then report the opposite.
In the latter, Tesla markets a feature as "Autopilot". It pilots a car into another vehicle. The feature had apparently been demonstrated at events with key safety features disabled: attendees are able to use it without their hands on the wheel. "Autopilot" fails to account for a common real-world traffic occurrence and its driver has an inadequate mental model of the feature, which is made up of a complicated mesh of sensors and software.
There may be something to the argument that self-driving car features will reduce traffic fatalities even as they become responsible for a distinctive (but hopefully small) subset of those fatalities. But this article doesn't make that point.
The problem people have with features like Autopilot is that it may not be reasonable --- at least, as those features are currently designed --- to expect human drivers to model their behavior accurately enough to use them safely. This is an HCI problem that people in the computer industry should be very familiar with. I'm surprised to see it dismissed so easily.
The author is comparing them in the context of the risk of uproar that they create. There's no comparison with the exact circumstances of the two problems.
The article is describing how a past issue with Audi caused a huge sales problem for the company, and is suggesting that 'autopilot' style technologies could cause the same style of event.
That's kind of a circular argument, right? The uproar is a function of the perceived cause of the event. In the Audi case, the perceived cause is incorrect. But we don't know that about the Tesla case.
While I greatly appreciate Tesla pushing this technology out in a reasonable time frame, I do feel that they let their marketing speak trump safety by referring to this technology as Autopilot. I don't doubt that they intend to eventually push this tech to the point that Autopilot is an appropriate name, but calling it such at this stage is disenguous at best and ultimately could be hindered by people treating it as its named and getting hurt or killed. It's not reasonable to expect all drivers that may get behind the wheel of a Tesla to get proper training and definitions for its technology package.
The term is certainly no exaggeration if you look at what an autopilot does in a plane. Keep attitude, maybe turn to next waypoint. All in all super primitive to what a Tesla does.
I'm sure Tesla has stacks of pre-prepared legal defense around the term, and they'll certainly make the same airplane argument.
My bet is it will quickly fall apart. Ask your average person what "autopilot" implies and they're going to give you the sci-fi definition of a completely autonomous driving system, not many people know about the mechanics and limitations of autopilots in airplanes.
That's totally sufficient for airplanes since crossing semi-trailers are fairly rare on airways. Jokes aside, all obstacles for airplanes are radar tracked, so keeping attitude is all that it needs doing. If human intervention is needed, the system usually has time to warn. Still, even those super-simple systems fail.
In short: as automation is better able to handle easy-to-predict scenarios, that increasingly leaves only rare, complex scenarios for the humans to deal with which, by definition, they are far worse at handling.
Part of my criticism of Tesla, though, comes from the fact that a large truck crossing in front of you is actually not a particularly complex situation for humans (just brake), even for a sleepy driver.
>It appears to have been something of a freak accident -- white trailer riding high against the bright sky, so that the autopilot didn’t detect the truck in its path.
It wasn't a freak accident. The AI failed. The human driver was distracted and failed to respond in time. Two failures Tesla has to answer for. IMO the second one is the gravest. They shouldn't have deployed the technology, primitive as it is.
It depends on what kind of failure. If there is a red button next to an slightly darker red button and you get the instruction from your boss to press the red one, you have a 50% of failing. But maybe the designer of said buttons should have thought about giving the buttons different colors instead of similar ones.
"The Design of Everyday Things" talks about this, that sometimes it's the designer of something's fault, rather than the user of the something. Also called "Human Error" by statements.
Point being, unless Tesla makes it safe to be used as autopilot, you probably shouldn't say it's autopilot. Otherwise it's not good enough and you would fault Tesla for the design rather than the users of the autopilot.
I agree that the naming is poor - that doesn't make them liable for human error though, the driver has to opt in and is made well aware of the operation modes.
Incidentally I'm carrying that exact book in my backpack right now. I think the problem here is much deeper, and even opposite to what Norman discusses in the book. An autopilot is the epitome of "don't make me think" - I want to go from A to B, no need to worry about the driving. The problem at hand is that the technology is not mature enough to fully take over from humans, and we are not good at staying alert in this half-focused state - I don't own a Tesla but it must be incredibly boring. It's an interesting conflict of tech vs human nature.
Yes, "freak accident" generally implies some wildly unlikely confluence of events. White trailers are really common. The sun shining in your face is not uncommon.
As far as I can tell, the only rare circumstance here is that Tesla's autopilot would be enabled on a road where there's cross traffic.
I agree that it's not much of a freak accident, and you got me thinking that there might be some additional aspects of this case that might be worth considering.
For most of my life I have lived somewhere with lots of snow, and where the sun hangs out close to the horizon during the entire day during the winter, and I have never understood why white cars and tractor trailers are legal on public roads. In some conditions you are lucky if you see them at 1/2 the range of a vehicle of just about any other color. Even compared to ligt grays/silver metallic it's a huge difference.
Maybe something with the faint silhouette not looking very vehicle-like making the brain ignore it, I don't know?
While white vehicles can be harder to spot only with the sun being close to the horizon, add some drifting snow and they essentially becomes ghosts.
So while the accident clearly show Teslas autopilot is not good enough for hands free driving - of which it has made no claim to be - it might also be telling us a very simple fact: That white is a particularly bad colour for tractor trailers regardless if it is a human or a computer interpreting the visual field ?
the point of this tech is that it can be better than human, specifically wider viewfield at wider wavelength band. Humans have only narrow wavelength band ("visible light") only in reflected/passive mode in a 120 by 60 viewfield. The tech can do 360 by 360 viewfield in human visible - cameras(passive) and lidars(active), infrared - cameras and lidars, centimeter - radar, ultrasound radar. Most of this tech (except may be non-stereo low quality camera) would see the trailer. Tesla's main failure here is that they did viewfield smaller than even the human's one.
Agreed. This article seems like Tesla propaganda designed to protect their multi-billion dollar investment.
I know I'm not going to be one of Tesla's beta users. Fuck that.
Any system that controls the fate of human lives should have several fail-safes. i.e They shouldn't only have one way of detecting objects. Maybe video + sonar + laser. I don't know the details of their architecture but even a Microsoft Kinect could have detected a white trailer against the bright sky using it's infrared + video camera. Gravely disappointed in Tesla and even more for articles like this who don't hold them accountable for inferior technology.
The author of this piece is a market libertarian generally and an admirer of tech companies specifically. McArdle is not a vector for Tesla propaganda. I disagree with the piece --- as I disagree with almost everything McArdle writes --- but it's useful to get an intellectually coherent perspective from a different worldview than my own.
We're very rarely well served by phrases like "this article seems like propaganda".
To the point, here's a previous post by the same author about the topic, and which touches on Level 2 systems like Tesla:
"Self-Driving Cars Will Thrive With More Regulation"
Level 2 automation is not really a self-driving car, and from the driver’s perspective, sitting there staring at the road and waiting for something to happen is probably worse than just driving the car. Moreover, since people don’t actually pay as much attention as they’re supposed to when using these features, these systems will probably also cause some accidents, even as they prevent others.
And another
Getting to level-three automation, where the car mostly drives itself most of the time, seems imminent. But the actual advantages of level-three automation over cruise control and early warning systems seem dubious. If people are distracted when the car needs them to take over, then level-three automation could end up being more dangerous than the older systems. And if people aren’t distracted, which is to say they are grimly staring at the road with nothing to do for hours at a stretch, then it seems worse than actually driving the car.
The analogy in aircraft circles is "lining up the holes in the swiss cheese"
Between the systems, the SOPs, and the training, there are layers of redundancy (slices of cheese) in almost everything. All have weaknesses, but not the same weaknesses. Still, occasionally the holes in the cheese line up and you have a crash. Usually after the investigation another layer of cheese is added.
In this case the Tesla had only two layers: the autopilot, and the driver. The driver wasn't paying attention so when the autopilot made a mistake there was nothing to save the situation.
I used to work on the radar systems in these vehicles. It's a problem that the feature is called "autopilot," yes, but the larger issue is that the car does not enforce what the company often says in press releases, which is that you need to remain alert and with your hands on the wheel. There have been similar systems in other vehicles for years - I'm most familiar with those in Mercedes vehicles that are not as advanced but perform similar actions. Those systems will automatically turn off after just a few seconds of removing your hands from the wheel. That is what the Tesla should do. The autopilot feature is a toy; it's not to be used in place of a human driver, nor would a human driver be able to step in quickly and take over for the system were it to suddenly fail had the person not been paying attention. It should not be possible to watch a Youtube video of someone driving completely hands free for minutes at a time.
Yes, Tesla should not call it "autopilot," but if you want to really prevent people from using the system in a way in which it's not intended, you need to enforce it in hardware.
Is Tesla's autopilot using radar? How is radar fooled by a white truck against a bright background? I was surprised by this point of failure, which suggests to me that they aren't even using radar.
Maybe I'm a little clueless on how radar works, but how would something that is lower than a car's height be mistaken for something as distant as a overhead sign?
As of July 2016, the autonomous driving system in the Model S (& X, presumably) is not using radar, only cameras. The vehicles do have radar which is used for other things (various crash avoidance programs), but not this, and these things presumably did not fire correctly. I do not know if Tesla's software update capabilities would allow it to push a future autonomous driving update in the future that could interface with the radar sensors; I'd assume so. When I was working on these things the software build that went to the factory was what was on the car for 15 years, obviously this way is superior.
As a side note, radar in cars and semi trucks are a difficult match. The height of tractor trailer trucks makes it difficult to figure out if they are actually obstacles or if you're really seeing a bridge or something. Normally there's enough hanging down in back that it's not an issue, but in this case the truck was perpendicular to the car. It would not surprise me if the radar went right underneath the truck and either didn't see it, or saw it but thought it was something like a bridge that the car could go under. These are difficult and complex systems to engineer.
I am driving an '14 Audi A6 and I can tell you, the radars + software on it are the most advanced I ever saw. They even catch a small cat when was trying to cross the street. The car actioned a emergency break before I even realized. The only time when this wasn't working, was during a snow blizzard, when the radar sensors got iced and they couldn't see anything. Otherwise, they are pretty wide range, at least 100m.
It also keeps the car in between the lanes - this is done with a camera, but if you take your hands of the wheel, you get a max of about 30 seconds of free driving to maybe open a bottle of water, before it get's really loud.
> :o) Tesla is not using radars?! I am driving an '14 Audi A6 and I can tell you, the radars + software on it are the most advanced I ever saw.
TIL everyone on Hacker News drives a much nicer car than I do. The crash avoidance measures run in tandem with the autonomous driving features, but the system (which is mainly licensed technology from Mobileye) just doesn't get any depth intelligence from it. Presumably if autonomous driving pointed you into a brick wall, the radar stuff would still trigger the AEB features (autonomous braking).
It uses radar, but combines that with the cameras to discard "obstacles" that the cameras tell it to, like road signs (big flat surfaces are very reflective to radar). The cameras did not see the truck and so told the radars to ignore it.
The system does ping and then yell at you if you don't have your hand on the wheel. There's no way to prevent someone from keeping a hand on the wheel and watching a movie while driving.
It's really tragic what happened but as a driver you should be aware of what's happening. I've put ~15k miles of Autopilot on our Tesla and while it helps with the mental fatigue I was still watching the road 100% of the time exactly for reasons like this.
Also I think the killer app for Autopilot is rush hour traffic. Things happen much slower and this is where I found the larger percentage of mental fatigue happens.
There are two situations where it pings:
1) After a set amount of time (around 5 minutes or so). This is to ensure you are awake.
2) If it hits a turn or situation where it is less confident.
3) Major escalation in situations where a crash may be imminent (this ping is much louder and urgent).
Agree that on straightaways, the pings are pretty infrequent.
Sure, but the unstated assumption there is that stage won't also force-feed you the side of a tractor trailer through your windshield.
Currently Tesla's cars act as though they've satisfied that assumption even though they haven't, and someone died over it. Tesla absolutely should not be acting like they've created a self-driving car with safety features that let you ignore the road when in fact they are driving assists and you still have to maintain awareness.
The driver is compelled to keep their hands on the wheel by the relevant laws/regulations, not by the car, same as every other car manufactured in the last 100 years.
> The system does ping and then yell at you if you don't have your hand on the wheel. There's no way to prevent someone from keeping a hand on the wheel and watching a movie while driving.
Daimler actually shipped something intended to do this and beep/yell at you if it thought you were "distracted" or tired, but I honestly never knew what it was measuring. If you could point a camera at the driver it would be trivial, but people would definitely have an issue with that!
> It's really tragic what happened but as a driver you should be aware of what's happening. I've put ~15k miles of Autopilot on our Tesla and while it helps with the mental fatigue I was still watching the road 100% of the time exactly for reasons like this.
Yes, even just adaptive cruise control is a big help and wonderful in traffic. But the ACC I'm familiar with will turn off as soon as you let go of the wheel. I'm happy to hear that the features have worked well for you, and it's nice that the Tesla yells at you, but it just needs to turn off.
> It's a problem that the feature is called "autopilot,"…
Why is that a problem? An autopilot system is designed to assist, not replace, human operators. And that's exactly what Tesla's Autopilot does.
> … but the larger issue is that the car does not enforce what the company often says in press releases, which is that you need to remain alert and with your hands on the wheel.
Cars without Autopilot (i.e. every car ever manufactured in the past 100 years) also do not enforce that the driver must hold the steering wheel. I don't see anyone complaining about that. It's the driver's responsibility to place their hands on the wheel.
Most cars manufactured in the last hundred years has a tendency to, you know, _crash within seconds_ when the driver lets go of the steering wheel. In practice, this tends to train users to keep a hand on the wheel.
> It's the driver's responsibility to place their hands on the wheel
I'd argue it's the manufacturer's responsibility that they ensure a driver using their "autopilot" feature keeps their hands on the wheel and maintains attention so I am not hit by one of their broken vehicles.
Other cars that do automated driving tasks alert drivers loudly when they take their hands off the wheel, there is no reason that shouldn't be standard.
Pushing the responsibility on to consumers when it's a solved problem in the industry is negligent and irresponsible on the manufacturer's part.
>> It's a problem that the feature is called "autopilot,"…
> Why is that a problem?
The US is not in the business of allowing companies to sell products that pose a safety hazard to users or people around them.
Tesla's version of autopilot is a new feature that government agencies take pretty seriously. NTSB is now investigating the Florida crash.
There are several Youtube videos of people abusing the tech. Other drivers do not want those people driving around them. To get people to use the system safely, the government can (a) ask police to check that people are paying attention while driving, or (b) require stricter hardware controls such as hands must be on the wheel. (a) is impractical
> Cars without Autopilot (i.e. every car ever manufactured in the past 100 years) also do not enforce that the driver must hold the steering wheel. I don't see anyone complaining about that.
Other car systems are irrelevant. This is new tech that had to be approved for use in the US. It was approved last October.
- Tesla has collected 130 million miles of autopilot data in 8 months
- Google has collected 2 million miles of data in 5 years
- others have collect 0 million miles of data
One might assume that each 10x of data doubles safety. So, a billion miles of data is perhaps 8x better than a million. It was bold of Tesla to ship autopilot, but it gives them a significant data lead while Google's caution means they may be squandering their once-thought-unassailable technology lead.
My thinking is that first you eliminate the 1% problems, then the 0.1% problems, then 0.01% and so on. So each 10x of scenarios tested is roughly one step of improvement.
Maybe, or maybe Tesla wisely shipped an MVP and Google allowed the perfect to be the enemy of thr shippable.
Project those data numbers forward, do you really think Google can maintain a technology lead with such a disparity? Maybe they can, but it's a reasonable question.
Google could build a car tomorrow that does what Tesla does. They are far from revolutionary in what they are doing... Things like auto braking and lane maintaining are pretty much standard at the price point of the Model S, they are by no means unique to Tesla.
Tesla is implementing existing technology. Google is working on solving the next problem.
EDIT: No, they haven't... They have shipped industry standard driver assist features. By your logic, I assume you think my Kia Rondo has 'outshipped' Google, since it has cruise control.
A much better question is whether Tesla will be able to keep up with the new autonomous features that everyone else are doing. The new Mercedes E-class is the first to be approved as an autonomous car in Nevada, so they do have a technical edge on Tesla already. And with next years S-class, they'll start putting LIDAR on their cars. This isn't even on Tesla's roadmap.
It's beginning to look like Tesla saw there was a niche where existing hardware they could license would quickly give them a more "advanced" "Autopilot" system simply by being less conservative wrt. safety than other carmakers. Will this come back to bite them? Time will tell.
"Wisely shipped an MVP that accidentally crashes in corner cases," is that what you're saying? Because, unlike most software, people actually die in this kind of crash (Remember Therac-25? Of course not: way before agile development and MVP, ooold, booooring). Still insisting that it's wise? I would consider that position repulsive: never mind the corpses, we have a product to ship.
Depends on the middle part: is "move fast and kill people" viable? We'll see. (Which is to say, viable from a PR perspective: people are still scared of flying compared to manual car driving, despite the odds; I'm willing to take a wild guess and say that non-assisted driving is more dangerous than assisted driving, yet considered safer-because-familiar)
Google's self-driving cars is collecting a lot more data than Tesla is --- it has many more sensors that Tesla's cars. So it's not just how many miles you have driven, but what is the quality of the data. More importantly, it's not just having more data, it's how you use it, and that's what Google has been doing for the past five years.
The other reason for the caution is that people's lives are at stake. Google's self-driving cars are significantly more advanced in terms of the sorts of situations which they can handle ocmpared to Tesla's. But one of the things which Google has found is that there is an uncanny valley in terms of driver safety as self-driving cars get more capable. People will tend to trust them more, even shouldn't, and that paradoxically, makes them less safe.
The first generation self-driving cars were lent out to Google employees, where they would be told that the driver would be video'ed for research purposes. And people were found reading e-mail, newspapers, etc., even though they were told to be ready to take over at any time. After all, what do you do if you lose a tie rod? Or you suddenly have a catastrophic tire blow out at highway speeds?
That's why the second generation car has no steering wheel, because it's been shown that it's hopeless to depend on the human being. Instead, you need to have dual redundancy for everything (much like in aircraft design), so that even if you have a hardware problem, that the car can always bring itself safely to a stop at the side of the road, even if the driver is asleep or totally not paying attention because they are too busy watching Harry Potter or some other movie.
Caution is good, when human lives are at stake. Even if they are doing something that they were emphatically being told Not to do, if they do it, maybe the designer of the Autopilot has at least some moral responsibility, even if the lawyers have carefully protected Tesla from any legal liability?
You have to define what data they're collecting that's so valuable. Their system is so terrible at sensing the world around it that it will run right into construction barricades if you don't take over:
It's a fancy cruise control, not self-driving. It doesn't understand anything but how to follow lanes (sorta, it doesn't work everywhere) and it has some ability to recognize obstructions in front of it and stop (again, sorta).
Google is teaching their car how to navigate city streets, deal with human drivers and pedestrians, navigate intersections, handle construction zones, deal with emergency vehicles and cops directing traffic, etc. All with a much better sensor package than in a Tesla Model S. Their system doesn't even need lane markings on the road, as they compare LIDAR data with a pre-marked reference map. Tesla's system, and every other system that doesn't do this, breaks down when lane markings are faded or non-existent.
Google's system would not have been prone to this type of accident five years ago, much less today.
> Google is teaching their car how to navigate city streets, deal with human drivers and pedestrians, navigate intersections, handle construction zones, deal with emergency vehicles and cops directing traffic, etc. All with a much better sensor package than in a Tesla Model S.
The interesting thing is that no matter how much one anticipates situations a priori and builds them into the AI/model, there will likely exist situations not anticipated by the engineers.
For example, it is not clear to me how exactly a situation like https://en.wikipedia.org/wiki/Driving_direction#Sweden will be handled. Will this be "patchable" (never mind the logistics and security issues of providing updates to a nation's fleet), or will it require a full retraining of the AI? Or more generally, how robust/modular will the designed AI be to these kinds of situations?
The reason I like this example is because it is a classic instance where humans have inertia and thus have difficulty in consistently applying the "switch driving side" rule - I have seen international travellers requiring at least a few minutes at the wheel to reorient themselves.
In principle one would hope that a modular software solution can handle things more consistently.
Tesla could collect the data without allowing the car to drive autonomously. I think this case also raises the question of whether Tesla has the right sensors on the car. They may not have adequate dynamic range in their cameras and seems like their radar system isn't a true 3d mapping radar. Being able to detect a moving target that crosses the car's path seems like a fairly fundamental requirement of any driving system.
Google has a lot more data than their autonomously driven miles (e.g. their street view data and presumably more) and they're still not comfortable enough with what they've got. Their initial results looked very promising but it seems they've become more cautious over time.
I don't think Google is squandering their lead. I think they've just gotten to that part of the curve where they realize this is a harder problem than what they initially projected. The others haven't gotten there yet.
What really struck me when I was seeing a presentation about self-driving cars was that Google might have over-engineered it.
If Tesla gets its system right, Google will have a really hard time. Their desired lidar data monopoly would become a big pile of useless data if simpler systems could do the job at a similar safety level.
The tesla incident highlights the exact technical/ethical problems of using such systems. I was just at a conference speaking with the Google automated car team and the handoff problem is yet to be solved conceptually. The tesla accident is just a first example of it. Obviously the utility of an autopilot system becomes questionable if you can't guarantee safety. It's very similar to medical advances in this way. 1% errors is not a good error rate if it leads to death or serious injury even though such error rates for autonomous systems are remarkable.
That said , if we look at the statistical likelihood of death in a car autonomous systems may very well be much better than humans, but whether we as a society are ethically okay with coming to terms with a piece of software that kills 1 / 100M is another question. A drunk driver that kills someone or themselves is obviously to blame. In this case who's fault is it? The drivers , tesla or the safety board for allowing autopilots to be sold?
"In this case who's fault is it? The drivers , tesla or the safety board for allowing autopilots to be sold?"
Obviously the insurance. In Europe you need to have insurance to pay whenever is your fault.
With autopilot the only difference will be that the event of having an accident will be less probable. This means that paying the normal driver fee the insurance company will be making money if you autodrive.
But most of the time the fault will be on the other side, because the autopilot is not going to drive like a drunk or slept person. It will be deer or dogs or children on the road when they should not be there.
I wouldn't be so quick to say insurance companies would pick up the tab. I don't know which country you're from but in the us getting insurance companies to pay for a claim is very hard and until a set of laws are established I would expect dramatic pushback from them. Furthermore in most accidents insurance companies try to seek out the party at fault if it's not crystal clear to avoid paying.
But an interesting situation would occur if the incentives from the insurance companies would be directed towards automated driving because of reduced risks and costs to the driver. That would be an interesting scenario
Tesla has two basic problems with their "autopilot". One is hype, and one is bad technical design.
Tesla's "autopilot" is just automatic lane keeping and automatic cruise control. Mercedes and BMW have been shipping that for years. The other car companies have hardware to check if the driver's hands are on the wheel. Tesla not only didn't do that, they hyped those basic functions as being an "autopilot", as if they had something comparable to what Google has. Tesla has significantly overhyped their technology, to a dangerous level.
Here's another Tesla crash, from yesterday.[1] This is a Tesla rear-ending the corner of a stalled van on a highway. This may be an "automation in wrong mode" error, and complicated excuses from Tesla will probably be forthcoming. That's not good enough. Airline pilots get into "wrong mode" errors, and they get extensive training, including simulator time. Drivers don't get that. This driver presumably thought the automation would do something sensible in that situation. It didn't.
The other big Tesla problem is a pure technical design error - the bumper-mounted radar is blind at windshield height. This is the immediate cause of the fatal accident involving under-running the semitrailer. It's also the cause of the parking accident with the truck with overhanging load sticking out the back.[2] Tesla's implementation has deadly blind spots. They need a radar at windshield-top height. It's quite likely, now that the NHTSA is looking into this crash, that Tesla will have to do a recall and retrofit one. (Along with hands-on-wheel sensors.)
Tesla was relying on stereo cameras too much. Depth from stereo is always iffy; there needs to be some detail on the target to get range. You can't range a uniform surface with stereo vision. (Human flyers have the same problem flying over ice.) A big white truck under some lighting conditions has that property. Cameras are useful, but backed only by today's algorithms, not enough. Teslas come with a sensor suite inadequate for the job.
From a software perspective, Tesla's software seems to be overoptimistic that there's no problem ahead. 3D vision systems know when they don't have range info for a large area. Tesla's software was willing to drive into that. It didn't note the absence of range info for the road surface; it may not even be profiling the road ahead. (We all had to do that in the DARPA Grand Challenge because we were operating off-road. That's a solved problem. Google does it with their roof-mounted LIDAR.) Tesla owners have asked for "pothole detection" as a feature. Lack of this may eventually take someone off a cliff.
The press is gradually picking up on this. Until recently, Tesla's autopilot was treated as a magic black box in the press. Now some automotive writers are starting to ask the right questions.
This is not a setback for automatic driving. This is just a Tesla problem.
The Tesla owner says in the description "Yes, I could have reacted sooner, but when the car slows down correctly 1,000 times, you trust it to do it the next time to. My bad.."
And this is the same problem mentioned elsewhere in this thread. The adaptive cruise being 99.99% effective at stopping trains the driver that it will always work. Especially because Tesla doesn't force the driver to touch the steering wheel, they seem especially susceptible to this issue - and drivers do things like this: https://www.youtube.com/watch?v=5fDbfy0409U&t=0s
It seems like the solution is to have the software train the driver to take control in unusual situations.
One way to do that might be that if the car successfully avoids an obstacle on its own without the driver reacting at all, after the obstacle is passed the car pulls over and stops.
So for the next five years there won't be another fatal accident in which automatic driving is at least partially responsible and will generate significant press coverage?
Maybe in the short term, but in the long term, if automatic driving works, it works. There's just too much money to be saved by implementing it for people to not work on it.
> The other big Tesla problem is a pure technical design error - the bumper-mounted radar is blind at windshield height. This is the immediate cause of the fatal accident involving under-running the semitrailer. It's also the cause of the parking accident with the truck with overhanging load sticking out the back.[2] Tesla's implementation has deadly blind spots. They need a radar at windshield-top height. It's quite likely, now that the NHTSA is looking into this crash, that Tesla will have to do a recall and retrofit one. (Along with hands-on-wheel sensors.)
This is just completely false. We've been putting them at bumper height for 10 years because that's where the system ends up working best with the greatest number of targets to see. NHTSA has been regulating these sensors for years and designing the tests for them; they are not new to this. Tesla is not going to have to do a recall on these, and spare me the idea that they are going to "retrofit a sensor at windshield height." This is just mindless speculation you made up.
This is just completely false. We've been putting them at bumper height for 10 years because that's where the system ends up working best with the greatest number of targets to see
> because [the bumper is] where the system ends up working best with the greatest number of targets to see
Do you have a source for this? It was my understanding Google went with their roof-mounted approach because of the blind-spot issue. (They also went with Lidar over cameras because of the white-truck-in-bright-daylight problem.)
I worked on the radar systems personally and we tested them with the Lidar hardware that Google uses. When the systems are raised they become more difficult to engineer (Caterpillar uses similar sensors on its large trucks, which are raised). Ultimately we solve this with more capable radar hardware in the bumper, not multiple mounting points, though Volvo's innovation in this area is significant and commendable.
Volvo has placed radars at the top center of the windshield. They had a blind spot at bumper level, so later Volvo systems added a second radar down there. And a LIDAR. [1] One radar at bumper level is just enough to keep from rear-ending the car in front. That's not enough for a hands-off "autopilot". You need at least two, or beam steering in elevation, like the Fujitsu unit.[2] (Although that has only ±9° of elevation, not enough at bumper level to eliminate the need for a higher sensor.)
I had not seen some of that Volvo tech, thanks for the link. Ultimately the radar portion is going to be more three dimensional tech still in the bumper. It's a struggle financially to get a single one of these sensors on a commodity vehicle, so the tech is focused on how to do more with that. Still, what Volvo is doing with the technology available is very interesting transitional technology.
This is not a setback for automatic driving. This is just a Tesla problem.
Let's all hope that's how this is remembered 10 years from now (or whenever) when automatic driving is actually ready.
It's not a given that people will remember this as a Tesla-only problem. Instead it might be remembered as that time when autopilot got people killed.
And when it's actually ready... what Tesla is doing today, might delay or slow the adoption of real autopilot. And that would be a real setback for the real thing.
there will be deaths and accidents from the computer controlled cars. that is pretty much inevitable. the question is where on the x of y miles driven per accident/fatality does it make sense to accept that computers and algorithms are fallible just like humans but in aggregate the system works better than just humans doing all the manual control inputs. it's the first of its kind and we only have one datapoint w.r.t. the deadliness of autopilot used without oversight but it seems to indicate that the autopilot is safer overall in aggregate.
No, it's not clear at all that autopilot is safer. Tesla has stated that it's safer by comparing the miles autopilot has driven vs the overall average for a human.
But that's an unfair comparison.. because humans drive in all conditions, while autopilot is more likely to be used on highways (and in good weather). Even human drivers are 3x safer on highways. So until Tesla is willing to compare like with like, we do not know how it compares.
It's also too early to waive off the damage that Tesla may cause. It's been 6 months since they rolled this thing out.
Over the next few years, they're going to give 300,000 cars to consumers with this enabled.
The few incidents that have been reported may just be the tip of the iceberg. We'll see how bad it gets when you have a few hundred thousand people driving around with what they think is autonomous driving.
I think a more apt comparison is miles driven while distracted. the guy was watching a movie and completely not paying attention. How many accidents were and are caused by people texting and driving per x number of miles? There were apparently enough that they passed laws against distracted driving in many states in america.
This is a beta product that explicitly says not to use it without paying attention. there are alarms that are supposed to go off and the vehicle is supposed to slow down to a halt if you do not have your hands on the steering wheel.
I see it like the beginning of the airline industry. At first they weren't as comfortable and safe as they are now. any time they have an airline disaster they do a thorough investigation and if it is warranted they will issue new mandates that airlines need to follow. over time, every accident makes air travel safer from those mistakes/edge cases/etc.
In ten years, when there is much more reliable autonomous driving, I think this will simply be remembered as a failed and misguided earlier attempt just like people trying to fly by strapping wings to their backs and flapping.
> It's quite likely, now that the NHTSA is looking into this crash, that Tesla will have to do a recall and retrofit one. (Along with hands-on-wheel sensors.)
Will they? Considering that those are driver-assistance systems it's always the human in charge, no?
The driver would also have died if he had just enabled "dumb" cruise control and noticed the white trailer.
Of course more advanced automation systems should do better and tesla should improve, but from a technical perspective it's still just an assistance system.
Although I have no idea what the NHTSA will do, it's clear that people are treating Tesla's autopilot as more than a driver-assistance system.
Hopefully in the future Tesla will add some additional checks to make sure the driver is actually using it as an assistance system instead of an auto-pilot. And when their auto-pilot actually becomes an auto-pilot they can remove those checks.
>And when their auto-pilot actually becomes an auto-pilot they can remove those checks.
And then they can call it autopilot again...
I wouldn't be surprised if Tesla get slapped for marketing it as an autonomous driving system. Musk has publicly said their autopilot is twice as safe as a human driver. That is misleading at best considering it should never be used without a human driver.
> This is the immediate cause of the fatal accident involving under-running the semitrailer.
Rear ending a semi is one of the most fatal types of car crashes. People should definitely not trust automatic cruise control when driving behind a trailer. And regular drivers should give themselves some extra space when driving behind a semi.
The trucking industry is powerful and will work to dodge anything that increases their costs like this. To be fair, the same is true of other industry representation.
AFAIK, Tesla doesn't even have stereo cameras. It's a single, B&W camera behind the rear view mirror. The camera is used for object recognition, but the radar does all distance stuff.
It's rumored that the next version will have three cameras looking forward, so camera can be used for distance calculations.
Tesla doesn't seem to actually sense hands on wheel. They sense torque on the steering wheel.[1] Mercedes seems to have capacitative touch sensing, although customers complaining about it aren't quite sure.[2]
Tesla doesn't take the absence of hands too seriously. "I'm driving down the highway at 55, and I'm not looking, and I have no hands on the wheel".[3] Hands off driving in New York City traffic.[4] "I managed to go 50 miles without a single nag"[5]
Apparently Tesla does have only one camera on currently shipping products. They must be trying to do stereo from motion. There are rumors of a revision to the Model S with multiple cameras.[5]
I think that's what Mobileye sell to Tesla. Per comments on Twitter[1] (and the fact that they fit a radar) the car's combining output from the Mobileye unit with the other sensors when decision making.
> Tesla's "autopilot" is just automatic lane keeping and automatic cruise control.
And lane changing. But yes, they never claimed the current system could do any more than that.
> Mercedes and BMW have been shipping that for years.
Yet, their systems on their 2017 models still isn't as good as Tesla's current system. [1]
> The other car companies have hardware to check if the driver's hands are on the wheel. Tesla not only didn't do that…
Tesla's system will indeed periodically prompt you to hold the steering wheel, and will disengage if the driver does not react to several warnings.
> … they hyped those basic functions as being an "autopilot", as if they had something comparable to what Google has
No. What Google has is called autonomous driving. You've completely misunderstood the meaning of the word "autopilot". From Wikipedia:
> An autopilot is a system used to control the trajectory of a vehicle without constant 'hands-on' control by a human operator being required. Autopilots do not replace a human operator, but assist them in controlling the vehicle, allowing them to focus on broader aspects of operation, such as monitoring the trajectory, weather and systems.
And that's exactly what Tesla's Autopilot does: assist the driver in controlling the vehicle, but does not replace a human operator.
Unlike the shape of the Earth, there is no truth of the matter regarding what a word means except common usage. In certain cases a word might have a technical definition beyond the usual consensus but 'autopilot' isn't one of those words and communications to the public in, say, an advertisement have to use the common meaning of words.
>Just because people mistakenly believe something doesn't make it right.
Tesla promotes it as autopilot. Words matter.
If Tesla itself selectively denies that the common understanding is the meaning they intend, then they appear to be trying to have it both ways: cashing in on the PR and cachet that the common meaning confers, while disclaiming the responsibility that goes along with owners actually believing it.
You're right, Tesla mistakenly believed they could cash in on calling their system Autopilot™ and somehow sidestep the connotations with the name and the damage pushing a broken implementation on consumers is causing.
The unfortunate truth is, if I gave my mom a car with the autopilot feature, she'd end up eventually trusting it and not care about the implementation details that make Tesla's system not an automated driving feature. That's scary and dangerous to market to consumers.
On a sailboat "autopilot" generally means "keep this heading" or "keep this point of sail." I'm pretty sure in aviation it means "take me to this way point and elevation" or a sequence of such maneuvers. A plane flies itself or a boat navigates itself only so long as nothing unexpected occurs. In either case, a pilot must be standing by, maintaining awareness, ready to assume control and handle anything that may come up. I'm not sure what people think this means in terms of driving a car but I think it's early enough that Tesla can help define what this means.
Autopilot systems in aviation have become so complex and diverse that there is no single catch-all sentence to describe them. I guess you could come close with the phrase "a system designed to keep and guide the state of the aircraft within and through certain set constrains".
Those constrains could be constant like a certain heading or a direction (not the same thing by the way), a series of vectors or a single vector (like a flight-path or an approach to a runway) or constraints that depend on dynamic variables like controlling the altitude and speed according to the weight of the aircraft or automatically avoiding traffic.
Of course the pilot must be standing by because a) autopilots do not keep track of everything (and aviation control interfaces are quite complex) and b) sensors can and do malfunction.
I notice you comment on self-driving cars a lot and seem knowledgeable on the subject. Is this just something you follow closely or do you have some background in this area? I ask solely out of curiosity.
Humans "overreacting" is the reason why airplanes are so safe. This reaction (IMO "fair" not "over") is important for the development of the driverless cars tech. It will make them much, much safer and won't kill any innovation.
"Underreaction"[1] is much more dangerous for society than overreaction. I would cite digital privacy as an example of this.
[1] i'm not sure underreaction is a word or if i invemted it. Is underreaction so underreacted to that it is not even a word?
From a utilitarian perspective one can invest too many resources in making specific things more secure. That happens when other causes of death dwarf the probability of incidence to cost of fixing ratio of a particular failure mode.
If you're accounting for micromorts then we might be at the point where TSA checks destroy more human life-hours than terrorism and normal plane crashes combined. Simply reverting airport security to pre-9/11 levels could save more lives than infinite spending increases for airplane crash safety.
So yes, at some point it becomes an overreaction when resources would be better spent on other ways of saving lives.
I'm not saying that we're there yet with self-driving cars. We probably are with planes.
The things we could do to improve air safety at this point are not technology related. If we want to spend money on air safety, paying pilots more, spending more on their training, and reducing their work hours would make the biggest difference.
This is how I feel about car insurance in Ontario (think $5000 a year for a teenage male).
On one hand, it's great that if I get in a crash I get crap loads of money for physical therapy and what not. On the other hand I wasted 2h every day of high school waiting and sitting on the bus and so did everyone else. Hours I'll never get back. Would rather have taken my chances :/
Eventually, the National Highway Traffic Safety Administration got involved, and wrote up a report which found that … yup, these drivers were stepping on the gas instead of the brakes, often with horrific results. That didn’t save Audi, for which sales collapsed and which almost pulled out of the U.S. market.
Despite thousands of hard working engineers, QA, Marketing, Sales, Legal and design people earnestly working to put out a safe, quality vehicle, Audi was almost undone by histrionic consumers. Really depressing when you think about it as a founder.
Far as I remember, that Audi had pedals that were narrower than usual and closer than usual. So it's not about the customers being histrionic, more like somebody in engineering not really understanding what he was doing.
I'm saddened, though not surprised, with how many people are so quick to come to Tesla's defense here.
Autonomous cars will have issues. But we should hold the software makers accountable for those issues the same way we hold airplane makers / maintainers responsible in the rare event their planes crash.
As other commenters pointed out, Tesla likely released a half-baked technical solution too early, and created misleading (and in this case dangerous) marketing to go along with it: calling it "autopilot" and not requiring hands on the steering wheel. Tesla screwed up here, and shame on them.
This is not a binary world: I'm not saying that they should be bankrupted or the autonomous car industry shut down. When a plane crashes due to poor maintenance - I want that company to suffer market+legal consequences, I don't want the entire aviation industry to be shut down.
White trucks on a sunny sky are not rare enough to call this an anomaly. If Tesla had installed sensors on their cars to collect data but not enabled autopilot yet - would this man still be alive today?
The MVP model is great, but not for things where life is on the line. Google seems to be doing this right.
The product Tesla has built is a perfectly fine assistance mechanism. The problem is that it's not an autopilot and nowhere near autonomous driving, nevertheless Tesla is naming it and marketing it as such.
The problem is that Tesla's marketing here is (borderline) fraudulent. There should be and need to be legal consequences for that.
A plane can depending on the airport start, fly and land on autopilot, you just need to tell it where to go. There is pretty much no unplanned intervention necessary. This is what the term means to people, autonomous driving. I don't think most people would draw any distinction between the two.
In a Tesla it would be irresponsible to take your hands of the wheel or not to pay attention to the road for even a minute. Tesla's "Autopilot" is nothing like what people imagine an autopilot to be like.
I don't get the argument that using the name Autopilot implies that the driver need not supervise the car. In aircraft autopilots require pilot supervision who can take over in the event of autopilot failure. In marine applications autopilots are even simpler. No-one would think that an autopilot in either of these fields where the word has been traditionally applied can function completely independently.
There were 1.07 fatalities every 100M automobile miles last year. Compare that to 1 fatality in 130M autopilot miles. This is not going to be the last fatality we see but it's not clear that autopilot is any worse than a human and statistically this still looks to be the safer option.
For comparison, that is 1.391 fatalities per 130M automobile miles. So 0.391 people were saved by Tesla, but that doesn't account for the fact that this is a new, eerie technology and that people are possibly using it in the safest of conditions until they get comfortable with it. Also is the average Tesla driver really representative of the overall population, i.e. would Tesla drivers have had that many crashes without Autopilot? How much of that 0.391 is due to the safety features of the car itself, and not the success of the autopilot system?
At any rate, there isn't much room for error in those numbers, and considering the above this almost looks like evidence that Autopilot is less safe than normal driving.
>but it's not clear that autopilot is any worse than a human
Well no, that would constitute an immediate and utter failure. More importantly though it's not significantly better than a human, which is disappointing, especially since they'd already limited the system to the simplest possible use case.
The biggest elephant in my view is that Tesla has known about this accident and has been hiding it from investors since May. The only reason why this came out is the NHTSA investigation, and if it were up to Musk, this crucial information would have stayed secret until he had completed the dumping of SolarCity onto Tesla shareholders.
Best outcome of NHTSA investigation would be financial penalty and order prohibiting use of AutoPilot brand on NON autonomous system. It was clearly picked to mislead customers into thinking they are buying more than there really is.
A bad joke, but this somehow reminded me about: Darwin Awards movie, Autopilot Cruise Control scene. Afaik, Tesla Autopilot is an assistant, not an fully autonomous system. Which inherently means that it requires constant supervision. Using boats or planes autopilot won't either relieve you from monitoring where you're going to end up. Afaik, some of the news articles were titled with pure lies. 'Autonomous car', hmm, nope. Tesla isn't such. It's always important to acknowledge the true capability s of a system and not blindly trust it. This applies to any more or less automated system. People start blindly trusting it even if it really shouldn't be trusted. Let's say robot or AI stock investment bots. Luckily those are't going to kill anyone.
I know, quite rude. It's sad that what happened happened. But in technological sense, we all knew it was going to happen sooner or later. We're going to see more and more things like this in future. - I've blogged several times about this same phenomenon, even before this case.
I think Tesla screwed up by hyping it like they have an autonomous driving system. Mercedes, BMW, etc. did not make that mistake with similar technology. Just google "tesla autopilot" and you'll see the phenomena: people sleeping, playing games, recording videos of themselves.
From that search, here's an interesting article from WIRED warning about the dangers (and lawsuit proneness) of the technology:
164 comments
[ 4.6 ms ] story [ 213 ms ] threadIn the former, people sometimes accidentally stomp the gas instead of the brakes, then report the opposite.
In the latter, Tesla markets a feature as "Autopilot". It pilots a car into another vehicle. The feature had apparently been demonstrated at events with key safety features disabled: attendees are able to use it without their hands on the wheel. "Autopilot" fails to account for a common real-world traffic occurrence and its driver has an inadequate mental model of the feature, which is made up of a complicated mesh of sensors and software.
There may be something to the argument that self-driving car features will reduce traffic fatalities even as they become responsible for a distinctive (but hopefully small) subset of those fatalities. But this article doesn't make that point.
The problem people have with features like Autopilot is that it may not be reasonable --- at least, as those features are currently designed --- to expect human drivers to model their behavior accurately enough to use them safely. This is an HCI problem that people in the computer industry should be very familiar with. I'm surprised to see it dismissed so easily.
Did you intend this to be a double-negative? It's pretty confusing as written, but would make sense as a single-negaitve.
The article is describing how a past issue with Audi caused a huge sales problem for the company, and is suggesting that 'autopilot' style technologies could cause the same style of event.
My bet is it will quickly fall apart. Ask your average person what "autopilot" implies and they're going to give you the sci-fi definition of a completely autonomous driving system, not many people know about the mechanics and limitations of autopilots in airplanes.
In short: as automation is better able to handle easy-to-predict scenarios, that increasingly leaves only rare, complex scenarios for the humans to deal with which, by definition, they are far worse at handling.
It wasn't a freak accident. The AI failed. The human driver was distracted and failed to respond in time. Two failures Tesla has to answer for. IMO the second one is the gravest. They shouldn't have deployed the technology, primitive as it is.
"The Design of Everyday Things" talks about this, that sometimes it's the designer of something's fault, rather than the user of the something. Also called "Human Error" by statements.
Search for "https://archive.org/stream/DesignOfEverydayThings/DesignOfEv... here https://archive.org/stream/DesignOfEverydayThings/DesignOfEv...
Point being, unless Tesla makes it safe to be used as autopilot, you probably shouldn't say it's autopilot. Otherwise it's not good enough and you would fault Tesla for the design rather than the users of the autopilot.
Incidentally I'm carrying that exact book in my backpack right now. I think the problem here is much deeper, and even opposite to what Norman discusses in the book. An autopilot is the epitome of "don't make me think" - I want to go from A to B, no need to worry about the driving. The problem at hand is that the technology is not mature enough to fully take over from humans, and we are not good at staying alert in this half-focused state - I don't own a Tesla but it must be incredibly boring. It's an interesting conflict of tech vs human nature.
Camera looks up from bumper, see a overhead sign, continues on.
Later it looks up from the bumper, and a trailer is down about the same area, continues on.
Nope... Need radar. Give me a break.
As far as I can tell, the only rare circumstance here is that Tesla's autopilot would be enabled on a road where there's cross traffic.
For most of my life I have lived somewhere with lots of snow, and where the sun hangs out close to the horizon during the entire day during the winter, and I have never understood why white cars and tractor trailers are legal on public roads. In some conditions you are lucky if you see them at 1/2 the range of a vehicle of just about any other color. Even compared to ligt grays/silver metallic it's a huge difference.
Maybe something with the faint silhouette not looking very vehicle-like making the brain ignore it, I don't know? While white vehicles can be harder to spot only with the sun being close to the horizon, add some drifting snow and they essentially becomes ghosts.
So while the accident clearly show Teslas autopilot is not good enough for hands free driving - of which it has made no claim to be - it might also be telling us a very simple fact: That white is a particularly bad colour for tractor trailers regardless if it is a human or a computer interpreting the visual field ?
the point of this tech is that it can be better than human, specifically wider viewfield at wider wavelength band. Humans have only narrow wavelength band ("visible light") only in reflected/passive mode in a 120 by 60 viewfield. The tech can do 360 by 360 viewfield in human visible - cameras(passive) and lidars(active), infrared - cameras and lidars, centimeter - radar, ultrasound radar. Most of this tech (except may be non-stereo low quality camera) would see the trailer. Tesla's main failure here is that they did viewfield smaller than even the human's one.
I know I'm not going to be one of Tesla's beta users. Fuck that.
Any system that controls the fate of human lives should have several fail-safes. i.e They shouldn't only have one way of detecting objects. Maybe video + sonar + laser. I don't know the details of their architecture but even a Microsoft Kinect could have detected a white trailer against the bright sky using it's infrared + video camera. Gravely disappointed in Tesla and even more for articles like this who don't hold them accountable for inferior technology.
We're very rarely well served by phrases like "this article seems like propaganda".
"Self-Driving Cars Will Thrive With More Regulation"
Level 2 automation is not really a self-driving car, and from the driver’s perspective, sitting there staring at the road and waiting for something to happen is probably worse than just driving the car. Moreover, since people don’t actually pay as much attention as they’re supposed to when using these features, these systems will probably also cause some accidents, even as they prevent others.
And another
Getting to level-three automation, where the car mostly drives itself most of the time, seems imminent. But the actual advantages of level-three automation over cruise control and early warning systems seem dubious. If people are distracted when the car needs them to take over, then level-three automation could end up being more dangerous than the older systems. And if people aren’t distracted, which is to say they are grimly staring at the road with nothing to do for hours at a stretch, then it seems worse than actually driving the car.
http://www.bloomberg.com/view/articles/2016-01-15/self-drivi...
http://www.bloomberg.com/view/articles/2015-12-01/winning-th...
Between the systems, the SOPs, and the training, there are layers of redundancy (slices of cheese) in almost everything. All have weaknesses, but not the same weaknesses. Still, occasionally the holes in the cheese line up and you have a crash. Usually after the investigation another layer of cheese is added.
In this case the Tesla had only two layers: the autopilot, and the driver. The driver wasn't paying attention so when the autopilot made a mistake there was nothing to save the situation.
Yes, Tesla should not call it "autopilot," but if you want to really prevent people from using the system in a way in which it's not intended, you need to enforce it in hardware.
I suppose I don't really know, but that was Elon's answer to "why didn't the radar see it?"
That's what puzzles me. I don't get how overhead road signs can be a problem, and it couldn't detect something at car height. Its just really odd.
As a side note, radar in cars and semi trucks are a difficult match. The height of tractor trailer trucks makes it difficult to figure out if they are actually obstacles or if you're really seeing a bridge or something. Normally there's enough hanging down in back that it's not an issue, but in this case the truck was perpendicular to the car. It would not surprise me if the radar went right underneath the truck and either didn't see it, or saw it but thought it was something like a bridge that the car could go under. These are difficult and complex systems to engineer.
I am driving an '14 Audi A6 and I can tell you, the radars + software on it are the most advanced I ever saw. They even catch a small cat when was trying to cross the street. The car actioned a emergency break before I even realized. The only time when this wasn't working, was during a snow blizzard, when the radar sensors got iced and they couldn't see anything. Otherwise, they are pretty wide range, at least 100m.
It also keeps the car in between the lanes - this is done with a camera, but if you take your hands of the wheel, you get a max of about 30 seconds of free driving to maybe open a bottle of water, before it get's really loud.
TIL everyone on Hacker News drives a much nicer car than I do. The crash avoidance measures run in tandem with the autonomous driving features, but the system (which is mainly licensed technology from Mobileye) just doesn't get any depth intelligence from it. Presumably if autonomous driving pointed you into a brick wall, the radar stuff would still trigger the AEB features (autonomous braking).
no such thing exist ...
It's really tragic what happened but as a driver you should be aware of what's happening. I've put ~15k miles of Autopilot on our Tesla and while it helps with the mental fatigue I was still watching the road 100% of the time exactly for reasons like this.
Also I think the killer app for Autopilot is rush hour traffic. Things happen much slower and this is where I found the larger percentage of mental fatigue happens.
Agree that on straightaways, the pings are pretty infrequent.
That will require a stage of using Autopilot for five minutes without your hands on the wheel, and it won't even beep at you.
Currently Tesla's cars act as though they've satisfied that assumption even though they haven't, and someone died over it. Tesla absolutely should not be acting like they've created a self-driving car with safety features that let you ignore the road when in fact they are driving assists and you still have to maintain awareness.
Daimler actually shipped something intended to do this and beep/yell at you if it thought you were "distracted" or tired, but I honestly never knew what it was measuring. If you could point a camera at the driver it would be trivial, but people would definitely have an issue with that!
> It's really tragic what happened but as a driver you should be aware of what's happening. I've put ~15k miles of Autopilot on our Tesla and while it helps with the mental fatigue I was still watching the road 100% of the time exactly for reasons like this.
Yes, even just adaptive cruise control is a big help and wonderful in traffic. But the ACC I'm familiar with will turn off as soon as you let go of the wheel. I'm happy to hear that the features have worked well for you, and it's nice that the Tesla yells at you, but it just needs to turn off.
https://www.mbusa.com/mercedes/technology/videos/detail/titl...
Why is that a problem? An autopilot system is designed to assist, not replace, human operators. And that's exactly what Tesla's Autopilot does.
> … but the larger issue is that the car does not enforce what the company often says in press releases, which is that you need to remain alert and with your hands on the wheel.
Cars without Autopilot (i.e. every car ever manufactured in the past 100 years) also do not enforce that the driver must hold the steering wheel. I don't see anyone complaining about that. It's the driver's responsibility to place their hands on the wheel.
I'd argue it's the manufacturer's responsibility that they ensure a driver using their "autopilot" feature keeps their hands on the wheel and maintains attention so I am not hit by one of their broken vehicles.
Other cars that do automated driving tasks alert drivers loudly when they take their hands off the wheel, there is no reason that shouldn't be standard.
Pushing the responsibility on to consumers when it's a solved problem in the industry is negligent and irresponsible on the manufacturer's part.
> Why is that a problem?
The US is not in the business of allowing companies to sell products that pose a safety hazard to users or people around them.
Tesla's version of autopilot is a new feature that government agencies take pretty seriously. NTSB is now investigating the Florida crash.
There are several Youtube videos of people abusing the tech. Other drivers do not want those people driving around them. To get people to use the system safely, the government can (a) ask police to check that people are paying attention while driving, or (b) require stricter hardware controls such as hands must be on the wheel. (a) is impractical
> Cars without Autopilot (i.e. every car ever manufactured in the past 100 years) also do not enforce that the driver must hold the steering wheel. I don't see anyone complaining about that.
Other car systems are irrelevant. This is new tech that had to be approved for use in the US. It was approved last October.
- Google has collected 2 million miles of data in 5 years
- others have collect 0 million miles of data
One might assume that each 10x of data doubles safety. So, a billion miles of data is perhaps 8x better than a million. It was bold of Tesla to ship autopilot, but it gives them a significant data lead while Google's caution means they may be squandering their once-thought-unassailable technology lead.
My thinking is that first you eliminate the 1% problems, then the 0.1% problems, then 0.01% and so on. So each 10x of scenarios tested is roughly one step of improvement.
Tesla has added some driver assist features which are far from unique to Tesla...
It's an apple to oranges comparison.
Project those data numbers forward, do you really think Google can maintain a technology lead with such a disparity? Maybe they can, but it's a reasonable question.
Tesla is implementing existing technology. Google is working on solving the next problem.
EDIT: Tesla has.
EDIT: No, they haven't... They have shipped industry standard driver assist features. By your logic, I assume you think my Kia Rondo has 'outshipped' Google, since it has cruise control.
Google isn't a car company, they don't need to ship a car. They need to ship a working AI.
It's beginning to look like Tesla saw there was a niche where existing hardware they could license would quickly give them a more "advanced" "Autopilot" system simply by being less conservative wrt. safety than other carmakers. Will this come back to bite them? Time will tell.
The other reason for the caution is that people's lives are at stake. Google's self-driving cars are significantly more advanced in terms of the sorts of situations which they can handle ocmpared to Tesla's. But one of the things which Google has found is that there is an uncanny valley in terms of driver safety as self-driving cars get more capable. People will tend to trust them more, even shouldn't, and that paradoxically, makes them less safe.
The first generation self-driving cars were lent out to Google employees, where they would be told that the driver would be video'ed for research purposes. And people were found reading e-mail, newspapers, etc., even though they were told to be ready to take over at any time. After all, what do you do if you lose a tie rod? Or you suddenly have a catastrophic tire blow out at highway speeds?
That's why the second generation car has no steering wheel, because it's been shown that it's hopeless to depend on the human being. Instead, you need to have dual redundancy for everything (much like in aircraft design), so that even if you have a hardware problem, that the car can always bring itself safely to a stop at the side of the road, even if the driver is asleep or totally not paying attention because they are too busy watching Harry Potter or some other movie.
Caution is good, when human lives are at stake. Even if they are doing something that they were emphatically being told Not to do, if they do it, maybe the designer of the Autopilot has at least some moral responsibility, even if the lawyers have carefully protected Tesla from any legal liability?
https://www.youtube.com/watch?v=iSasPVYzSQ0&t=150
It's a fancy cruise control, not self-driving. It doesn't understand anything but how to follow lanes (sorta, it doesn't work everywhere) and it has some ability to recognize obstructions in front of it and stop (again, sorta).
Google is teaching their car how to navigate city streets, deal with human drivers and pedestrians, navigate intersections, handle construction zones, deal with emergency vehicles and cops directing traffic, etc. All with a much better sensor package than in a Tesla Model S. Their system doesn't even need lane markings on the road, as they compare LIDAR data with a pre-marked reference map. Tesla's system, and every other system that doesn't do this, breaks down when lane markings are faded or non-existent.
Google's system would not have been prone to this type of accident five years ago, much less today.
The interesting thing is that no matter how much one anticipates situations a priori and builds them into the AI/model, there will likely exist situations not anticipated by the engineers.
For example, it is not clear to me how exactly a situation like https://en.wikipedia.org/wiki/Driving_direction#Sweden will be handled. Will this be "patchable" (never mind the logistics and security issues of providing updates to a nation's fleet), or will it require a full retraining of the AI? Or more generally, how robust/modular will the designed AI be to these kinds of situations?
The reason I like this example is because it is a classic instance where humans have inertia and thus have difficulty in consistently applying the "switch driving side" rule - I have seen international travellers requiring at least a few minutes at the wheel to reorient themselves.
In principle one would hope that a modular software solution can handle things more consistently.
Google has a lot more data than their autonomously driven miles (e.g. their street view data and presumably more) and they're still not comfortable enough with what they've got. Their initial results looked very promising but it seems they've become more cautious over time.
I don't think Google is squandering their lead. I think they've just gotten to that part of the curve where they realize this is a harder problem than what they initially projected. The others haven't gotten there yet.
They are using different models for their data, different types of data and so on.
Data itself eventually stops helping; although finding more edge cases is certainly helpful.
If Tesla gets its system right, Google will have a really hard time. Their desired lidar data monopoly would become a big pile of useless data if simpler systems could do the job at a similar safety level.
That said , if we look at the statistical likelihood of death in a car autonomous systems may very well be much better than humans, but whether we as a society are ethically okay with coming to terms with a piece of software that kills 1 / 100M is another question. A drunk driver that kills someone or themselves is obviously to blame. In this case who's fault is it? The drivers , tesla or the safety board for allowing autopilots to be sold?
Obviously the insurance. In Europe you need to have insurance to pay whenever is your fault.
With autopilot the only difference will be that the event of having an accident will be less probable. This means that paying the normal driver fee the insurance company will be making money if you autodrive.
But most of the time the fault will be on the other side, because the autopilot is not going to drive like a drunk or slept person. It will be deer or dogs or children on the road when they should not be there.
But an interesting situation would occur if the incentives from the insurance companies would be directed towards automated driving because of reduced risks and costs to the driver. That would be an interesting scenario
Tesla has two basic problems with their "autopilot". One is hype, and one is bad technical design.
Tesla's "autopilot" is just automatic lane keeping and automatic cruise control. Mercedes and BMW have been shipping that for years. The other car companies have hardware to check if the driver's hands are on the wheel. Tesla not only didn't do that, they hyped those basic functions as being an "autopilot", as if they had something comparable to what Google has. Tesla has significantly overhyped their technology, to a dangerous level.
Here's another Tesla crash, from yesterday.[1] This is a Tesla rear-ending the corner of a stalled van on a highway. This may be an "automation in wrong mode" error, and complicated excuses from Tesla will probably be forthcoming. That's not good enough. Airline pilots get into "wrong mode" errors, and they get extensive training, including simulator time. Drivers don't get that. This driver presumably thought the automation would do something sensible in that situation. It didn't.
The other big Tesla problem is a pure technical design error - the bumper-mounted radar is blind at windshield height. This is the immediate cause of the fatal accident involving under-running the semitrailer. It's also the cause of the parking accident with the truck with overhanging load sticking out the back.[2] Tesla's implementation has deadly blind spots. They need a radar at windshield-top height. It's quite likely, now that the NHTSA is looking into this crash, that Tesla will have to do a recall and retrofit one. (Along with hands-on-wheel sensors.)
Tesla was relying on stereo cameras too much. Depth from stereo is always iffy; there needs to be some detail on the target to get range. You can't range a uniform surface with stereo vision. (Human flyers have the same problem flying over ice.) A big white truck under some lighting conditions has that property. Cameras are useful, but backed only by today's algorithms, not enough. Teslas come with a sensor suite inadequate for the job.
From a software perspective, Tesla's software seems to be overoptimistic that there's no problem ahead. 3D vision systems know when they don't have range info for a large area. Tesla's software was willing to drive into that. It didn't note the absence of range info for the road surface; it may not even be profiling the road ahead. (We all had to do that in the DARPA Grand Challenge because we were operating off-road. That's a solved problem. Google does it with their roof-mounted LIDAR.) Tesla owners have asked for "pothole detection" as a feature. Lack of this may eventually take someone off a cliff.
The press is gradually picking up on this. Until recently, Tesla's autopilot was treated as a magic black box in the press. Now some automotive writers are starting to ask the right questions.
This is not a setback for automatic driving. This is just a Tesla problem.
[1] https://www.youtube.com/watch?v=qQkx-4pFjus [2] http://www.roadandtrack.com/new-cars/car-technology/news/a29...
> [1] https://www.youtube.com/watch?v=qQkx-4pFjus
minor nitpick but this video is from May, this didn't happen yesterday, did you mean to link another video?
And this is the same problem mentioned elsewhere in this thread. The adaptive cruise being 99.99% effective at stopping trains the driver that it will always work. Especially because Tesla doesn't force the driver to touch the steering wheel, they seem especially susceptible to this issue - and drivers do things like this: https://www.youtube.com/watch?v=5fDbfy0409U&t=0s
This is not an issue until it is.
One way to do that might be that if the car successfully avoids an obstacle on its own without the driver reacting at all, after the obstacle is passed the car pulls over and stops.
> This is not a setback for automatic driving. This is just a Tesla problem.
If the general public doesn't get the distinction it can become an automatic driving problem.
You want to make a bet on that?
This is just completely false. We've been putting them at bumper height for 10 years because that's where the system ends up working best with the greatest number of targets to see. NHTSA has been regulating these sensors for years and designing the tests for them; they are not new to this. Tesla is not going to have to do a recall on these, and spare me the idea that they are going to "retrofit a sensor at windshield height." This is just mindless speculation you made up.
Who's "we"?
Do you have a source for this? It was my understanding Google went with their roof-mounted approach because of the blind-spot issue. (They also went with Lidar over cameras because of the white-truck-in-bright-daylight problem.)
[1] http://www.pcworld.com/article/2025386/volvo-v40-watches-the... [2] http://www.fujitsu-ten.com/business/technicaljournal/pdf/38-...
Let's all hope that's how this is remembered 10 years from now (or whenever) when automatic driving is actually ready.
It's not a given that people will remember this as a Tesla-only problem. Instead it might be remembered as that time when autopilot got people killed.
And when it's actually ready... what Tesla is doing today, might delay or slow the adoption of real autopilot. And that would be a real setback for the real thing.
But that's an unfair comparison.. because humans drive in all conditions, while autopilot is more likely to be used on highways (and in good weather). Even human drivers are 3x safer on highways. So until Tesla is willing to compare like with like, we do not know how it compares.
It's also too early to waive off the damage that Tesla may cause. It's been 6 months since they rolled this thing out.
Over the next few years, they're going to give 300,000 cars to consumers with this enabled.
The few incidents that have been reported may just be the tip of the iceberg. We'll see how bad it gets when you have a few hundred thousand people driving around with what they think is autonomous driving.
This is a beta product that explicitly says not to use it without paying attention. there are alarms that are supposed to go off and the vehicle is supposed to slow down to a halt if you do not have your hands on the steering wheel.
I see it like the beginning of the airline industry. At first they weren't as comfortable and safe as they are now. any time they have an airline disaster they do a thorough investigation and if it is warranted they will issue new mandates that airlines need to follow. over time, every accident makes air travel safer from those mistakes/edge cases/etc.
Will they? Considering that those are driver-assistance systems it's always the human in charge, no?
The driver would also have died if he had just enabled "dumb" cruise control and noticed the white trailer.
Of course more advanced automation systems should do better and tesla should improve, but from a technical perspective it's still just an assistance system.
Hopefully in the future Tesla will add some additional checks to make sure the driver is actually using it as an assistance system instead of an auto-pilot. And when their auto-pilot actually becomes an auto-pilot they can remove those checks.
And then they can call it autopilot again...
I wouldn't be surprised if Tesla get slapped for marketing it as an autonomous driving system. Musk has publicly said their autopilot is twice as safe as a human driver. That is misleading at best considering it should never be used without a human driver.
Rear ending a semi is one of the most fatal types of car crashes. People should definitely not trust automatic cruise control when driving behind a trailer. And regular drivers should give themselves some extra space when driving behind a semi.
https://youtu.be/bT3G-kcKN70?t=1m9s
It's rumored that the next version will have three cameras looking forward, so camera can be used for distance calculations.
Of course they do, which is why it's such brilliant and irresponsible marketing.
- tesla does have hands on wheel sensors
- tesla does not use stereo cameras (autopilot uses a single forward facing camera combined with radar and sonar)
- the second crash 'from yesterday' you linked is from May
Tesla doesn't take the absence of hands too seriously. "I'm driving down the highway at 55, and I'm not looking, and I have no hands on the wheel".[3] Hands off driving in New York City traffic.[4] "I managed to go 50 miles without a single nag"[5]
Apparently Tesla does have only one camera on currently shipping products. They must be trying to do stereo from motion. There are rumors of a revision to the Model S with multiple cameras.[5]
Yes, the van crash was in May, not yesterday.
[1] https://forums.teslamotors.com/forum/forums/how-get-rid-hold... [2] http://mbworld.org/forums/new-s-class-w222/626411-steering-w... [2] https://www.youtube.com/watch?v=LS7pAC0OmLI [3] https://www.youtube.com/watch?v=M4BGlQoASyo [4] https://forums.teslamotors.com/forum/forums/nag-hands-wheel-... [5] http://cleantechnica.com/2016/05/04/new-tesla-model-s-2nd-tr...
This is not correct.
https://www.youtube.com/watch?v=Kv9JYqhFV-M
It is actually camera-only per comment of Mobileye's CEO.
1. https://twitter.com/elonmusk/status/748625979271045121
And lane changing. But yes, they never claimed the current system could do any more than that.
> Mercedes and BMW have been shipping that for years.
Yet, their systems on their 2017 models still isn't as good as Tesla's current system. [1]
> The other car companies have hardware to check if the driver's hands are on the wheel. Tesla not only didn't do that…
Tesla's system will indeed periodically prompt you to hold the steering wheel, and will disengage if the driver does not react to several warnings.
> … they hyped those basic functions as being an "autopilot", as if they had something comparable to what Google has
No. What Google has is called autonomous driving. You've completely misunderstood the meaning of the word "autopilot". From Wikipedia:
> An autopilot is a system used to control the trajectory of a vehicle without constant 'hands-on' control by a human operator being required. Autopilots do not replace a human operator, but assist them in controlling the vehicle, allowing them to focus on broader aspects of operation, such as monitoring the trajectory, weather and systems.
And that's exactly what Tesla's Autopilot does: assist the driver in controlling the vehicle, but does not replace a human operator.
[1] http://mashable.com/2016/06/16/2017-mercedes-benz-e-class-fi...
http://www.askthepilot.com/questionanswers/automation-myths/
Just because people mistakenly believe something doesn't make it right. The general public once thought the Earth was flat, and how did that turn out?
Tesla promotes it as autopilot. Words matter.
If Tesla itself selectively denies that the common understanding is the meaning they intend, then they appear to be trying to have it both ways: cashing in on the PR and cachet that the common meaning confers, while disclaiming the responsibility that goes along with owners actually believing it.
The unfortunate truth is, if I gave my mom a car with the autopilot feature, she'd end up eventually trusting it and not care about the implementation details that make Tesla's system not an automated driving feature. That's scary and dangerous to market to consumers.
Those constrains could be constant like a certain heading or a direction (not the same thing by the way), a series of vectors or a single vector (like a flight-path or an approach to a runway) or constraints that depend on dynamic variables like controlling the altitude and speed according to the weight of the aircraft or automatically avoiding traffic.
Of course the pilot must be standing by because a) autopilots do not keep track of everything (and aviation control interfaces are quite complex) and b) sensors can and do malfunction.
"Underreaction"[1] is much more dangerous for society than overreaction. I would cite digital privacy as an example of this.
[1] i'm not sure underreaction is a word or if i invemted it. Is underreaction so underreacted to that it is not even a word?
If you're accounting for micromorts then we might be at the point where TSA checks destroy more human life-hours than terrorism and normal plane crashes combined. Simply reverting airport security to pre-9/11 levels could save more lives than infinite spending increases for airplane crash safety.
So yes, at some point it becomes an overreaction when resources would be better spent on other ways of saving lives.
I'm not saying that we're there yet with self-driving cars. We probably are with planes.
If you look at AWS and the like, or services that tout "5 9s", you could make similar arguments. But reliability is itself a feature of a system.
If airplane crashes were more frequent, less people would rely on it (if you think it's hysterical now...).
Public Transit and package delivery are other things where reliability is super important. Not just public perception, but in a real way.
If you have to change trains twice, you can't have both be late right?
Also, death isn't just "subtract the man-hours from this person's society"....
On one hand, it's great that if I get in a crash I get crap loads of money for physical therapy and what not. On the other hand I wasted 2h every day of high school waiting and sitting on the bus and so did everyone else. Hours I'll never get back. Would rather have taken my chances :/
Eventually, the National Highway Traffic Safety Administration got involved, and wrote up a report which found that … yup, these drivers were stepping on the gas instead of the brakes, often with horrific results. That didn’t save Audi, for which sales collapsed and which almost pulled out of the U.S. market.
Despite thousands of hard working engineers, QA, Marketing, Sales, Legal and design people earnestly working to put out a safe, quality vehicle, Audi was almost undone by histrionic consumers. Really depressing when you think about it as a founder.
Autonomous cars will have issues. But we should hold the software makers accountable for those issues the same way we hold airplane makers / maintainers responsible in the rare event their planes crash.
As other commenters pointed out, Tesla likely released a half-baked technical solution too early, and created misleading (and in this case dangerous) marketing to go along with it: calling it "autopilot" and not requiring hands on the steering wheel. Tesla screwed up here, and shame on them.
This is not a binary world: I'm not saying that they should be bankrupted or the autonomous car industry shut down. When a plane crashes due to poor maintenance - I want that company to suffer market+legal consequences, I don't want the entire aviation industry to be shut down.
White trucks on a sunny sky are not rare enough to call this an anomaly. If Tesla had installed sensors on their cars to collect data but not enabled autopilot yet - would this man still be alive today?
The MVP model is great, but not for things where life is on the line. Google seems to be doing this right.
The problem is that Tesla's marketing here is (borderline) fraudulent. There should be and need to be legal consequences for that.
An autopilot is a system that assists, but does not replace, the human operator in operating a vehicle. Tesla's system does exactly this.
> … and nowhere near autonomous driving…
Nobody ever claimed it was autonomous.
In a Tesla it would be irresponsible to take your hands of the wheel or not to pay attention to the road for even a minute. Tesla's "Autopilot" is nothing like what people imagine an autopilot to be like.
A mechanical, electrical, or hydraulic system used to guide a vehicle without assistance from a human being.
"I set the autopilot to due south, so I could get some rest."
At any rate, there isn't much room for error in those numbers, and considering the above this almost looks like evidence that Autopilot is less safe than normal driving.
Well no, that would constitute an immediate and utter failure. More importantly though it's not significantly better than a human, which is disappointing, especially since they'd already limited the system to the simplest possible use case.
Normal consumers aren't interested in technical details or complicated explanations. They just want what the marketing promised them.
From that search, here's an interesting article from WIRED warning about the dangers (and lawsuit proneness) of the technology:
https://www.wired.com/2015/10/obviously-drivers-are-already-...