it seems we're living at the golden age of autonomous driving development as even Uber killing a woman due to disabled emergency braking didn't make it even to the negligence (granted Uber showing poor dynamic range video recorded by what seems like a cheap cell phone camera to the sheriff in that village (Phoenix) also played the trick - that sheriff can be fooled by such a video is the major part of why it is the golden age right now). The same way with Tesla - notwithstanding me being a big fan of Musk - self-driving without 3d scene building is pure arrogance and negligence which is enabled only by general public and government not-understanding of the things - thus the golden age.
Funny enough, it happens with human drivers too. Perhaps the fix is to block the entire lane instead of partially blocking it. At least the Tesla will stop for such a scenario, where a human still might not. Can't be that expensive to roll an additional rescue vehicle to sit in the lane with their lights on (I see it quite often in Florida, unsure about other states).
> According to recent data taken from the AAA Foundation for Traffic Safety, approximately 12 percent of all interstate highway deaths result from shoulder accidents. This means that an estimated 600 people a year are killed—and thousands more are injured—while making emergency stops off the highway.
It's unfortunately fairly difficult to find stats on accidents scoped solely to vehicles rear ending emergency vehicles partially obstructing a travel lane. The stat I present is a proxy.
Running into the back of a police cruiser that's partly in the lane with lights on is something humans generally only do before blowing a .1x on the breathalyzer.
The link you posted shows a clear correlation between low light conditions and hitting people on the shoulder. A police cruiser has blue lights and presumable the federally mandated reflective surfaces that all vehicles sold in the US have (usually in the tail lights).
This is a typical case of software making stupid mistakes a human would never make.
> The National Safety Council reports that cell phone use while driving leads to 1.6 million crashes each year. Nearly 390,000 injuries occur each year from accidents caused by texting while driving. 1 out of every 4 car accidents in the United States is caused by texting and driving.
Your statistics are for human drivers hitting cars and people. Not objects with blue lights (intentionally designed to be highly visible) that can be seen for miles at night being hit by vehicles that are augmented with computers that don't get distracted. Stop being intentionally misleading. An always-on computer hitting a police cruiser (the rear end of which conforms to known standards regarding number, position and size of reflective surfaces) is nowhere near equivalent to someone hitting something because they were looking at their phone.
Is it too much to expect technology to at least avoid the kind of accidents only incapacitated humans have?
Those lights are blinding as hell at night. The only reason a person wouldn't have hit the car is because it's the only thing you can see when trying to pass one.
Running into the back of a police cruiser that's partly in the lane with lights on is something humans generally only do before blowing a .1x on the breathalyzer. ... This is a typical case of software making stupid mistakes a human would never make.
Humans making this mistake do tend to brake at some point, just too late. There's still an accident, but damage is mitigated and survival odds, with seat belts and airbags, are good. Teslas are notorious for plowing into unrecognized stationary obstacles at full speed.
Many drivers don't get that limitation of Tesla's system. Which is why Level 2 systems that get it right most of the time are dangerous. Tesla has a really good lane keeping system, which creates the illusion that it can drive well. Until it can't.
That statistic is pedestrian deaths, not "hit stopped car on the shoulder" accidents.
"Pedestrians may enter the Interstate intentionally, often despite restrictions and controls, while drivers and motor vehicle occupants may become “unintended” pedestrians when their vehicle is disabled by a crash or other incident(Johnson, 1997)."
Yeah, we aren't going to rearchitect the highway system and teach every driver the new rules so Tesla can move a couple more units with it's false autonomous driving promises. They simply need to stop hitting parked cars with bright lights flashing.
Tesla is going to keep selling Autopilot, and average drivers will continue to drive poorly. I suggest a low cost change to emergency procedures that improves outcomes regardless of who has command of the vehicle. Autonomous driving will continue to improve, human drivers will not.
Let's be realistic, the law isn't going to catch up. Uber killed a person with a safety driver not paying attention and all safety systems turned off. Did the law change? It did not.
How many deaths in total have occurred related to Autopilot use? 6 [1]. About 100 people per day die in auto-related incidents. If you want to take it to the extreme, Tesla has killed far fewer people than Boeing's 737MAX autopilot (which has killed 346 people in a six month span).
If Tesla Autopilot is going to keep putting lives at risk then they really shouldn’t be allowed to keep selling it. I’m a bit baffled as to suggestions otherwise. If some lettuce gets contaminated with E. coli, we don’t say “oh well, caveat emptor, maybe start cooking your lettuce instead of eating it in salads,” we recall the lettuce.
We don't stop selling lettuce because you can get E. coli. To your point, we recall what we think we can, and billions of lbs continue to get sold. We have accepted that some people will get sick or die (even when recalled) because we value the availability and other benefits of lettuce over outright banning it.
There is always risk. You can de-risk, but it's usually impossible de-risk entirely in regards to life safety. You are at risk of death on the road with human drivers, and that risk is well quantified (NHSTA, NTSB, private auto insurance industry). Arguably, there is data to demonstrate that safety systems and lane keeping are safer when edge cases don't occur [1]. The question is: are those edge cases acceptable if the risk is reduced overall? Planes crash, but people still fly commercial. Again, we don't ban commercial air travel, but regulators and industry work together to ensure there are robust systems in place to remove as much risk as possible.
Should we ban the production and sale of lettuce too because a few people get sick or die [2]? Because it sure does sound like we say, “oh well, caveat emptor, maybe start cooking your lettuce instead of eating it in salads." (The same number of people have died from lettuce as from Tesla's Autopilot in roughly the same window of time [a decade, lumping together AP1, AP2, AP2.5, and FSD beta])
I own no Tesla securities. Can't smart people simply argue that progress has a cost and what realistic risk expectations are? Being pragmatic doesn’t make one a shill.
It's the usual AI bug, crashing on a rare edge case (i.e. not well represented in the training data) that seems straightforward logically to a human, leading to us being surprised/shocked/horrified that the AI missed the issue.
The most interesting part of this, is that this is a case that humans screw up on a pretty regular basis too. However we understand the human reason for doing so (lack of concentration), and understand that AIs don't suffer from the same fault. So when an AI screws it up we are still surprised.
The issue is that the AI will do the exact same thing each time it is presented with the same data (crash) whereas a human could go to remedial driver's education. And taking away the license of one inattentive driver gets them (theoretically) off the road... whereas if you have an AI crash into a fire truck and you said "stop it" - you still have all the other identical deployments out there with the same flaw.
And "add some partial blockage by a vehicle in the lane, retrain the AI, and then redeploy it" isn't a proper answer... wouldn't that be a different model AI that would need to be validated again?
You can turn this around though. It's harder to teach the AI, sure. But once you've taught it, it stays taught. With humans you have to deal with the continually renewing stream of young people who are new to driving.
> And "add some partial blockage by a vehicle in the lane, retrain the AI, and then redeploy it" isn't a proper answer... wouldn't that be a different model AI that would need to be validated again?
You're the one adding unnecessary constraints here. If the validation process harms more than helps, you don't do it. In all likelyhood the correct validation process is something lightweight that can be done quickly for simple changes.
This sort of training process is likely to be a continuous process, they're always ingesting new data, trying to learn how to drive better from it (and keep up with how patterns in the world change), and therefore continually deploying new models. Because that makes driving safer overall. Not a "this bad thing happened so let's figure out how to put out a bugfix for it".
> The issue is that the AI will do the exact same thing each time it is presented with the same data
That is not true at all these days, not to mention that even a slight change in illumination will result in completely different input data, the way current AIs work.
> crashing on a rare edge case (i.e. not well represented in the training data)
This isn't a case where it makes any sense to try to extrapolate from training data and call it a day. You don't need any training to avoid hitting stationary objects.
> The first job of automated driving is to drive only on flat surfaces. Doesn't matter why it's not flat. Then you worry about where to go. That's how the off-road DARPA Grand Challenge vehicles had to work. "Flat road" is a pure geometry thing. Not much AI needed.
Why use "AI" to solve a problem you don't have in the first place? This is the wrong tool for the wrong job.
(Compare also GPT-2 chess, in which a model trained on transcripts of chess games isn't able to stop itself from making illegal moves. The solution is straightforward - give your chess model a model of the rules of chess! And don't attempt to drive into spaces you cannot physically occupy.)
> You don't need any training to avoid hitting stationary objects.
You do, because you need to correctly conclude that there is going to be a stationary object there with high probability. As it turns out, that's difficult to do perfectly.
I'm having difficulty following your train of thought. It can be difficult to make conclusions about the future position of mobile objects. It can't be difficult to make conclusions about the current position of stationary objects.
> It can't be difficult to make conclusions about the current position of stationary objects.
You need to know that there is a stationary object in the first place, of a kind you care about (not a plastic bag). I.e. that it's not actually a moving object, an optical illusion, etc.
You also need to know that you are going to end up moving through it (it's not a parked car that you will avoid by following your lane).
That's... not trivial.
(going to be was probably poor phrasing on my part, sorry)
If one doesn't need any training to avoid hitting stationary objects, why do so many human drivers hit stationary objects? I can see arguments that the program shouldn't hit stationary arguments, but the ones I see apply to humans too with only minor changes.
Is there a relevant difference between what's "training" for an CPU driver and what's training for a human driver, and if so, what?
Yes, computers have problems that humans don't. But this isn't one of them. This is a problem that humans don't have, and that computers also don't have. (Or rather, a problem that humans do not have trouble telling computers how to solve.)
Yes, yes, humans have failure modes that computers don't, and those failure modes do lead to some problems. That's not relevant.
I repeat myself: "I can see arguments that the program shouldn't hit stationary arguments, but the ones I see apply to humans too with only minor changes." That humans have extra problems doesn't really matter, so long as humans require training to cope with those problems that are similar to those computers have.
For example, computers may use a vision device that sweeps across the surroundings and then use lots of postprocessing to build a complete scene. That's similar to what the human eye and brain do, except for the details of how the sweeping sweeps and how the postprocessing puts together a picture.
It isn't an AI bug, it is a lack of sensor input for the AI to work on. Elon's insistence that he can throw a couple extra cameras on a car and perform at a level of safety equal to what Google has figured out requires a dozen LIDAR sensors is criminal.
And this is for me a dealkiller issue. Because it means that there are going to be ways to e.g. obscure traffic signs in ways that no human will notice, leading to very nightmarish scenarios where "you expect vs the AI expects" and ... crash.
In my opinion until this is somehow solved "full self driving" in presence of other human drivers sharing the road is just a dream.
Well, one can argue that someone willing to drive after his license is revoked is also willing to lie about the Tesla crashing itself. It seems super reasonable to wait for Tesla side instead of rushing to conclusions.
> The company, for nearly 200 years of its existence (1602–1800), had effectively transformed itself from a corporate entity into a state or an empire in its own right.
> With its pioneering institutional innovations and powerful roles in world history, the company is considered by many to be the first major, first modern, first global, most valuable, and most influential corporation ever seen.
From all the Tesla autopilot videos posted online I have seen so far, it isn't hard to believe. The system needs constant driver intervention, even for simple everyday scenarios. Calling it anything more than driver assist is borderline criminal.
Was it a PR move to do spans the PR department? “Tesla, where every dollar goes towards making your car better!”. Or did they seriously disband their PR department!?
The article states the police said it was on autopilot -
> The Tesla's Autopilot driver-assist system was engaged when it struck the police car, a blue Dodge Charger with its emergency lights activated, police officials tweeted.
Or did I misunderstand and are you saying you don't believe the police report here?
Isn't one still fully responsible even when the car is in autopilot mode? I feel like blaming a Tesla for "crashing itself in autopilot" is a bit like saying a Mercedes "crashed itself" because you put it on cruise control / lane assist / collision detection mode so you could text from the driver seat or some such.
I mean, I could totally see some sort of unrelated class action suit against Tesla on the grounds of false advertising, but I'm skeptical Tesla is going to end up as a co-defendant if the charge is one random dude driving recklessly.
Holding Tesla responsible could happen in a number of ways, including via the regulators. NHTSA could even force them to change or deactivate the autopilot system. I'm not talking about a lawsuit. I'm talking about interruptions to their business. That's why I used the Boeing analogy.
That ignores that somehow a very high % of drivers in the US are on a plain suspended license, or none at all, and nobody particularly cares that they are or if they crash and injure. One block is immigrants with unclear status who don't bother to get one in the first place, another is the usual delinquents that don't bother to pay fines or tickets. Having straight up lost your license in the US from incompetence are a vanishing number, because well that is practically impossible in the US.
(I don't believe this group has a particular tendency to want to hit stationary objects, though.)
If anyone else was wondering what percent of drivers in the US have a suspended license it's 2.59% [1]. Which honestly surprised me because it seems lower than I was expecting.
Fun fact: North Dakota has the most drivers with a suspended license at 7.75% and is followed by Ohio at 5.62% [2].
Crash investigators don't need to take the drivers word for it. They have devices that plug into what is left of the ODB2 port or directly into the cars ECU and download the contents of the Event Data Recorder, which gives a usually 30 second snapshot of all the data leading up to an airbag deployment.
The problem with waiting for Tesla's side of the story is that not only do they have a very strong vested interest in one outcome, they have also been caught lying about this stuff before.
My license has been suspended for not renewing the registration/smogging disassembled vehicles used as parts donors in project cars.
I had assumed the reasonable thing would occur; the plates would expire and become suspended, a non-issue for a chassis sawz-alled to bits and sold for scrap.
Nope! Not in IL. While receiving a benign speeding ticket on a road-trip out of state, I learned my ostensibly valid DL was suspended, and what it feels like to be handcuffed, arrested, and processed at the station.
Why had IL suspended my DL? Well, I was forced to stay out-of-state and appear in court that week, eventually learning why IL had suspended my license to defend my innocence: the disassembled vehicle hadn't passed a smog test that year. They suspended the vehicle owner's DL for this nonsense. The judge in the state I had to defend myself in didn't even know what a smog test was. It was a clown show across the board, with my pocket footing the bill at every turn.
A suspended license is not a reliable indicator of anything other than bureaucratic fundraising by the state.
> I had assumed the reasonable thing would occur; the plates would expire and become suspended, a non-issue for a chassis sawz-alled to bits and sold for scrap.
That expectation seems to be at odds with everything I as a non car owner remember from my parents selling their cars. As far as I can find there often is no explicit requirement to deregister the car/plates, but with the exact problem you ran into: Even if everything ended up suspended the registration would still point at you, with a very good chance that you end up liable.
> hadn't passed a smog test that year.
Among the top reasons for people driving around with an expired registration apparently. So the police found exactly what it expected when it saw that expired registration with your name on it.
> So the police found exactly what it expected when it saw that expired registration with your name on it.
What are you talking about? I wasn't driving a vehicle with expired tags, the vehicle in my name triggering the DL suspension had been DIY scrapped and the plates destroyed. I was pulled over for speeding, driving a friend's car on a road trip during my turn at the wheel.
>That expectation seems to be at odds with everything I as a non car owner remember from my parents selling their cars.
Selling and letting registration lapse are two very different things.
Most states only criminalize driving a car with a expired registration. I have half a dozen cars with expired registration to my name and it's a non issue as long as I don't get pulled over driving them (which would be really hard to do since they either aren't complete enough to be drive-able or have long since been turned into tin cans).
I only come in to these threads to watch the Musk fanboys and Tesla apologists contort themselves into knots. This reply is a new high bar though, I love it, well done.
No, the apparent fact that the car was in Autopilot is what makes this interesting. Autopilot is touted as being effective – if not more effective than human operators – in avoiding this kind of accident.
No where is it advertised that autopilot handles partial lane obstructions or debris in the road, certainly better than a human.
Regardless what your opinion is on the naming, autopilot is a form of cruise control and legally and operationally requires driver attention, 100% of the time.
Oh, there is some disclaimer text, so it is OK that the chief executive of a company lies to its customers, investors, and regulators. Thanks for clearing that up!
A disclaimer? Oh, then it's all perfectly fine then! And it refers people to the owners manual to get all the nitty-gritty details. Of course, the owners manual is likely not updated when Autopilot is via OTA updates.
A disclaimer, like a clickwrap TOS, is not enough to take away Tesla's culpability. You can't sign away your life.
Yes, change the name, change the marketing, and get their CEO's Twitter under control.
I mean, imagine this in any other industry. CEO: "Our new FireSafety-O brand insulation is fire safe!" [Some houses catch fire] Company lawyers: "Well, didn't you read the post-purchase disclaimer that says it's made out of shredded newspaper and kerosene, and is only suitable for use in a non-oxidizing atmosphere? Ignore the CEO, he's deranged."
FSD has been sold since... years ago. I don't even remember when they started selling FSD, its just an option on Tesla cars for a long long time now.
That's the thing: the features of this are are named "autopilot" and "full self driving". What do you expect people to do when they spend $5000 or $10,000 on "full self driving" ?
If you call your product "Autopilot", or "Full Self Driving", it is 100% reasonable for a consumer to assume that it will handle not driving in to things in the road appropriately.
It's dishonest to dismiss this with "regardless of your opinion on the naming".
I believe a similar hypothesis is being tested as relates to Google Chrome's "Incognito Mode" and whether users should have expectation that the browser takes active steps to mask who you are to servers you access (as opposed to modifying its policies for storing cookies and locally-cached files).
Look, I understand people have issues with the naming. FSD is a separate issue and they've been selling it for years without having any sort of working software you can actually use, without any indication that it ever will work as advertised (an "actual" full self-driving system).
But if a plane on autopilot crashed because the pilot wasn't paying attention, the pilot would be at 100% fault. That's the origin of the naming, even if it's confusing to a layperson. I'm not here to argue about whether they should call it this or not.
Autopilot in an airplane is something used by people who have gone through explicit training and licensing in order to be allowed to fly an airplane. Even getting a basic private pilot's license, which has extreme restrictions on when you can fly, takes months.
Expecting the same level of competency and understanding of tools from an airline pilot, and the every day driver who hasn't passed a driving test since they were 16 years old, is ludicrous at best.
You actually raise a great point here. When the autopilot system in an aircraft fails, as was the case in Air France 447, government authorities step in. They ground every aircraft with that equipment, conduct investigations, ask other countries to also look at the data to see if they come to the same conclusions, and ultimately hold the manufacture responsible for failing to meet airworthiness standards. Pilots are generally _not_ found at fault for equipment failures unless it is something part of the pre-flight checklist.
Some autopilot systems, namely those in newer passenger aircraft, effectively do. They can take off, fly, and land an aircraft without human intervention.
They've had this capability for decades now; it was mentioned in a 5th season Mythbusters episode (the one about talking a civilian into a landing).
A suspended license doesn’t mean that an operator is functionally unable to avoid ramming into obvious obstacles. Unless there is evidence that the driver intentionally meant to ram the police car or other wise cause harm through gross negligence, then his suspended license (and the fact that the hit car belongs to a cop) is a detail, but not the story.
To put it another way, if a driver suffers a seizure on the highway, and Autopilot was able to avoid all collisions and bring the driver safely to the side of the road, what do you think the headline should be?
“Driver suffers seizure on interstate, automatic driving system steers car to safety”
Or:
“Man successfully drives on interstate while having an incapacitating seizure”
"We sold you a car that drives, we never said it can drive uphill!"
Regardless of you opinion, autopilot kills people, because its not possible to suddenly take over from an automoua car that decided to kill you, in 50ms. We have mountains of evidence on reaction time
> No, the apparent fact that the car was in Autopilot is what makes this interesting. Autopilot is touted as being effective – if not more effective than human operators – in avoiding this kind of accident.
And the fact we only hear about one of these kinds of crashes every few months essentially proves it's true.
Remember, more than 100 people are killed on the roads each and every day of the year in the USA. Dig into the stats, one Tesla crashing does not mean that overall Autopilot is not effective.
Teslas are meant to have collision avoidance systems, irregardless of Autopilot enabled or not, that are meant to avoid these crashes (for example when the driver is incapacitated).
It is possible for third parties to setup experiments that demonstrate the limits of Tesla's (or any car's) obstacle detection. When you are an expert in the field (such as vision), you can guess the limits, and setup an experiment (e.g., a white sheet hanging across the road).
Top Gear did something like that. They got a big inflatable that looks like a car, set it on an old airport runway, and drove at it in various vehicle with auto-brake.
> Autopilots do not replace human operators. Instead, the autopilot assists the operator's control of the vehicle, allowing the operator to focus on broader aspects of operations (for example, monitoring the trajectory, weather and on-board systems).
So it "only" encourages its users to take their eyes of the road for extended periods of time? I am sure the police will love to hear "I was just checking google maps for directions, but I had autopilot enabled so it is fine" after a crash.
It doesn't matter what it encourages. It matters what the contract between the person using the car and the car is. The contract is quite clear. The term is clear. There really should be no issue. I really don't get why people are so salty about this.
When you are piloting a vehicle it should be razor sharp that you are holding a weapon in your hand that is deadlier then a handgun.
> Critics say the name [Autopilot] misrepresents the driver-assist system's capabilities.
I can't disagree. Autopilot in a car should mean the same as it does in "Airplane!" or "Wall-E".
> The driver of the Tesla, identified as a 22-year-old man from Lansing, was ticketed for failing to move over and driving with a suspended license.
And there's the actual problem. You can blame Tesla all you want, but you'll have a hard time proving that the brand/model/features of car this person chose to drove has anything to do with the crash.
The only bag of air I want inflating into my driver’s seat is the airbag, and then only if it’s reasonably likely to save me from death or dismemberment
I think you are completely correct that what autopilot means in the public consciousness is much more than what it can actually do.
The funny thing is that what an autopilot does in a real airplane is actually pretty close to what the Tesla autopilot does, if you set it to fly you into a mountain it will happily do so (abet with a lot of alarms going off in a modern airliner).
Sure, a basic private aircraft wing-leveler or heading keeping system will fly you into a mountain, but the autopilots in passenger airlines can take off, fly a complete course, and land the plane.
They're monitored by trained professionals with thousands of paid hours and hundreds of hours of training, of course.
That is precisely the misconception I was talking about. It's simply not true for modern airliners (outside of a few technical demonstrations). There was a Global businesses jet which had auto-takeoff but that's about the only one I know of in service. The reason it's not used is there are no limits on required takeoff visibility so there is no operational capibilities to be gained by auto-takeoff and you need to be able to see enough to taxi to the runway in any case.
CAT-III autoland is a thing but it's actually only used in very limited circumstances, not least because it increases runway safety margins in a way which restricts runway utilisation. Many busy international airfields don't even have a CAT-III certified runway. The aircraft won't automatically cycle through to the correct mode to land (there are a bunch of other steps required to put then autopilot into a double/triple redundant mode etc).
I think the stubborn refusal to make use of LIDAR and other advanced sensors along with the idiotic claim that "humans can do fine with two eyes so so can we" will go down as one of the most genuinely foolish and bullheaded moves by Elon Musk. Having unavoidable surprises and accidents in the development of something like self-driving is understandable, and ultimately a few such things are frankly acceptable losses if the rate of tens of thousands of annual deaths by humans is cut. But they should be unavoidable and surprises, and there should be failsafes to try to avoid the worst. Driving into an object at full speed is obviously the most fundamental Bad Thing, Don't Do That in driving. Having a system that can cut into the more complex autopilot and at least slam the breaks every single time, even at worst within a short range where it's too late to avoid but can at least greatly minimize impact energy, should have been a day 01 foundational safety measure before embarking upon the rest. Huge point of self-driving should be being better than humans are. They should have FLIRs to deal with both moose and dark pedestrians at night, LIDAR/terahertz/ultrasonics, 360° cameras, whatever is needed to not just meet but actively avoid all kinds of crashes humans have. I hope Tesla really does get a bit slammed for cheaping out and then refusing to do a mea culpa on it, it's just so dumb.
Well in a traditional frontal collision avoidance system you rely on the very same mmwave radars and cameras (even a stereo pair in the Subaru system). Yes, L4+ self driving needs more for performance. A Tesla has more sensors than normal FCA/LKA system implementations; so IMHO the blame in this specific case is more likely with software than hardware.
Even amongst traditional manufacturers, Ford, Nissan, and Volvo have all opted for use of lasers along with mmWave. Volkswagon also has laser sensors on at least some models. And normal CAS are merely final emergency backups to normal involved human driver. They're added systems to help cover gaps in attention, and many simply depend on alerting the driver, not even doing breaking on their own. Whereas Tesla has been marketing autopilot as on the path to full level 5 since the beginning right down to the branding (vs "advanced driver assistance" or something), and whatever asterisks they've added since drivers keep depending on autopilot and autopilot vehicles keep crashing into stationary objects. Sure, maybe better software could avoid that, but the fact on the ground is that it hasn't. And for a life safety critical system, throwing hardware at a problem vs pure software is not wrong just because it's more expensive.
Tesla may be trying to optimize costs and be aggressive as they have been in other areas, doing the whole MVP/iteration thing, but this is one place where that just isn't necessarily the greatest strategy. They should be aiming to be better than human drivers not just as-good-as, and the challenges and limitations of ultra complex software based systems development are hardly some new revelation. They should have been prepared for that too with more hardware backstop.
To add even more detail, as there still appears to be a lot of confusion:
The radar systems in these vehicles send out a radio pulse in a broad approximate-cone forward. They get bounces back from everything that reflects radio in front of them. Distance from the object is calculated by time between pulse and response. Speed towards/away from the object is calculated from Doppler shift of the radio frequency.
There are two main things that these systems can't detect.
1. Speed of the object perpendicular to the direction of radio wave travel.
2. Location of the object within the approximate-cone the radio pulse travels in.
Note that thanks to the second, you can't calculate the first with higher-level object tracking, either.
So the data you get back is a list of (same-direction velocity component, distance) pairs. There's no way to distinguish between stationary objects in the road and stationary objects above the road, to the side of the road, or even the surface of the road itself.
Radar just doesn't provide the directional information necessary to handle obstacle detection safely.
This isn't the first time, right? Last one was a stopped truck somewhere in Europe.
We built a fancy computer vision model to detect and track football and it turns out a bald referee in a sunny weather can trick our model to think his head is the ball. So I wouldn't be surprised to know there are some flaws in their models too.
It's way safer to have other sensors and don't rely only on cameras if the purpose is engaged autopilot that takes control from the driver away.
133 comments
[ 2.9 ms ] story [ 183 ms ] thread> According to recent data taken from the AAA Foundation for Traffic Safety, approximately 12 percent of all interstate highway deaths result from shoulder accidents. This means that an estimated 600 people a year are killed—and thousands more are injured—while making emergency stops off the highway.
https://aaafoundation.org/wp-content/uploads/2017/12/Pedestr...
It's unfortunately fairly difficult to find stats on accidents scoped solely to vehicles rear ending emergency vehicles partially obstructing a travel lane. The stat I present is a proxy.
The link you posted shows a clear correlation between low light conditions and hitting people on the shoulder. A police cruiser has blue lights and presumable the federally mandated reflective surfaces that all vehicles sold in the US have (usually in the tail lights).
This is a typical case of software making stupid mistakes a human would never make.
> Distracted driving is dangerous, claiming 3,142 lives in 2019.
https://www.nhtsa.gov/risky-driving/distracted-driving
> The National Safety Council reports that cell phone use while driving leads to 1.6 million crashes each year. Nearly 390,000 injuries occur each year from accidents caused by texting while driving. 1 out of every 4 car accidents in the United States is caused by texting and driving.
https://www.nsc.org/road-safety/safety-topics/distracted-dri...
Is it too much to expect technology to at least avoid the kind of accidents only incapacitated humans have?
Humans making this mistake do tend to brake at some point, just too late. There's still an accident, but damage is mitigated and survival odds, with seat belts and airbags, are good. Teslas are notorious for plowing into unrecognized stationary obstacles at full speed.
Many drivers don't get that limitation of Tesla's system. Which is why Level 2 systems that get it right most of the time are dangerous. Tesla has a really good lane keeping system, which creates the illusion that it can drive well. Until it can't.
"Pedestrians may enter the Interstate intentionally, often despite restrictions and controls, while drivers and motor vehicle occupants may become “unintended” pedestrians when their vehicle is disabled by a crash or other incident(Johnson, 1997)."
Not if they keep hitting parked cars they won't, the law will catch up to them.
How many deaths in total have occurred related to Autopilot use? 6 [1]. About 100 people per day die in auto-related incidents. If you want to take it to the extreme, Tesla has killed far fewer people than Boeing's 737MAX autopilot (which has killed 346 people in a six month span).
[1] https://www.tesladeaths.com/
>> Not if they keep hitting parked cars they won't, the law will catch up to them.
> Uber killed a person with a safety driver not paying attention and all safety systems turned off. Did the law change? It did not.
Hmmm. You seem to have glossed over the verb aspect. How many people did Uber kill like that?
There is always risk. You can de-risk, but it's usually impossible de-risk entirely in regards to life safety. You are at risk of death on the road with human drivers, and that risk is well quantified (NHSTA, NTSB, private auto insurance industry). Arguably, there is data to demonstrate that safety systems and lane keeping are safer when edge cases don't occur [1]. The question is: are those edge cases acceptable if the risk is reduced overall? Planes crash, but people still fly commercial. Again, we don't ban commercial air travel, but regulators and industry work together to ensure there are robust systems in place to remove as much risk as possible.
Should we ban the production and sale of lettuce too because a few people get sick or die [2]? Because it sure does sound like we say, “oh well, caveat emptor, maybe start cooking your lettuce instead of eating it in salads." (The same number of people have died from lettuce as from Tesla's Autopilot in roughly the same window of time [a decade, lumping together AP1, AP2, AP2.5, and FSD beta])
[1] https://www.tesla.com/VehicleSafetyReport
[2] https://www.buzzfeednews.com/article/venessawong/the-decade-... ("2010s total: More than 1,010 people sickened, including 6 deaths.")
The most interesting part of this, is that this is a case that humans screw up on a pretty regular basis too. However we understand the human reason for doing so (lack of concentration), and understand that AIs don't suffer from the same fault. So when an AI screws it up we are still surprised.
And "add some partial blockage by a vehicle in the lane, retrain the AI, and then redeploy it" isn't a proper answer... wouldn't that be a different model AI that would need to be validated again?
> And "add some partial blockage by a vehicle in the lane, retrain the AI, and then redeploy it" isn't a proper answer... wouldn't that be a different model AI that would need to be validated again?
You're the one adding unnecessary constraints here. If the validation process harms more than helps, you don't do it. In all likelyhood the correct validation process is something lightweight that can be done quickly for simple changes.
This sort of training process is likely to be a continuous process, they're always ingesting new data, trying to learn how to drive better from it (and keep up with how patterns in the world change), and therefore continually deploying new models. Because that makes driving safer overall. Not a "this bad thing happened so let's figure out how to put out a bugfix for it".
That is not true at all these days, not to mention that even a slight change in illumination will result in completely different input data, the way current AIs work.
This isn't a case where it makes any sense to try to extrapolate from training data and call it a day. You don't need any training to avoid hitting stationary objects.
Compare this comment from 2018: https://news.ycombinator.com/item?id=17015528
> The first job of automated driving is to drive only on flat surfaces. Doesn't matter why it's not flat. Then you worry about where to go. That's how the off-road DARPA Grand Challenge vehicles had to work. "Flat road" is a pure geometry thing. Not much AI needed.
Why use "AI" to solve a problem you don't have in the first place? This is the wrong tool for the wrong job.
(Compare also GPT-2 chess, in which a model trained on transcripts of chess games isn't able to stop itself from making illegal moves. The solution is straightforward - give your chess model a model of the rules of chess! And don't attempt to drive into spaces you cannot physically occupy.)
You do, because you need to correctly conclude that there is going to be a stationary object there with high probability. As it turns out, that's difficult to do perfectly.
You need to know that there is a stationary object in the first place, of a kind you care about (not a plastic bag). I.e. that it's not actually a moving object, an optical illusion, etc.
You also need to know that you are going to end up moving through it (it's not a parked car that you will avoid by following your lane).
That's... not trivial.
(going to be was probably poor phrasing on my part, sorry)
Is there a relevant difference between what's "training" for an CPU driver and what's training for a human driver, and if so, what?
Computers have problems humans don't too of course... that's why we get these articles about "ridiculous crash happened".
I repeat myself: "I can see arguments that the program shouldn't hit stationary arguments, but the ones I see apply to humans too with only minor changes." That humans have extra problems doesn't really matter, so long as humans require training to cope with those problems that are similar to those computers have.
For example, computers may use a vision device that sweeps across the surroundings and then use lots of postprocessing to build a complete scene. That's similar to what the human eye and brain do, except for the details of how the sweeping sweeps and how the postprocessing puts together a picture.
In my opinion until this is somehow solved "full self driving" in presence of other human drivers sharing the road is just a dream.
It could be argued that technically he wasn't driving.
Well, one can argue that someone willing to drive after his license is revoked is also willing to lie about the Tesla crashing itself. It seems super reasonable to wait for Tesla side instead of rushing to conclusions.
https://apnews.com/article/legislature-legislation-local-gov...
Highlights:
> The company, for nearly 200 years of its existence (1602–1800), had effectively transformed itself from a corporate entity into a state or an empire in its own right.
> With its pioneering institutional innovations and powerful roles in world history, the company is considered by many to be the first major, first modern, first global, most valuable, and most influential corporation ever seen.
William Dalrymple has a tome of a book about the English East India Company, which I hope to check out someday soon: https://www.theguardian.com/books/2019/sep/11/anarchy-relent...
Jason Hickel is also an invaluable source: https://www.aljazeera.com/opinions/2018/12/19/how-britain-st...
> The Tesla's Autopilot driver-assist system was engaged when it struck the police car, a blue Dodge Charger with its emergency lights activated, police officials tweeted.
Or did I misunderstand and are you saying you don't believe the police report here?
Isn't one still fully responsible even when the car is in autopilot mode? I feel like blaming a Tesla for "crashing itself in autopilot" is a bit like saying a Mercedes "crashed itself" because you put it on cruise control / lane assist / collision detection mode so you could text from the driver seat or some such.
Same as any other situation. If one 737 Max crashes, Boeing may blame the pilots. But if another crashes, Boeing has to take some responsibility.
https://www.nhtsa.gov/sites/nhtsa.dot.gov/files/documents/un...
(I don't believe this group has a particular tendency to want to hit stationary objects, though.)
Fun fact: North Dakota has the most drivers with a suspended license at 7.75% and is followed by Ohio at 5.62% [2].
[1]: https://insurify.com/insights/cities-with-the-most-suspended... [2]: https://insurify.com/insights/the-10-states-with-the-most-su...
I had assumed the reasonable thing would occur; the plates would expire and become suspended, a non-issue for a chassis sawz-alled to bits and sold for scrap.
Nope! Not in IL. While receiving a benign speeding ticket on a road-trip out of state, I learned my ostensibly valid DL was suspended, and what it feels like to be handcuffed, arrested, and processed at the station.
Why had IL suspended my DL? Well, I was forced to stay out-of-state and appear in court that week, eventually learning why IL had suspended my license to defend my innocence: the disassembled vehicle hadn't passed a smog test that year. They suspended the vehicle owner's DL for this nonsense. The judge in the state I had to defend myself in didn't even know what a smog test was. It was a clown show across the board, with my pocket footing the bill at every turn.
A suspended license is not a reliable indicator of anything other than bureaucratic fundraising by the state.
That expectation seems to be at odds with everything I as a non car owner remember from my parents selling their cars. As far as I can find there often is no explicit requirement to deregister the car/plates, but with the exact problem you ran into: Even if everything ended up suspended the registration would still point at you, with a very good chance that you end up liable.
> hadn't passed a smog test that year.
Among the top reasons for people driving around with an expired registration apparently. So the police found exactly what it expected when it saw that expired registration with your name on it.
What are you talking about? I wasn't driving a vehicle with expired tags, the vehicle in my name triggering the DL suspension had been DIY scrapped and the plates destroyed. I was pulled over for speeding, driving a friend's car on a road trip during my turn at the wheel.
Reading comprehension fail...
Can you point out where you wrote that in your previous comment?
What the hell else is this supposed to mean? That I both scrapped the car and was driving around with the plates for some reason?
Plus, wouldn't it be spelled "sawzall-ed" (though, more likely, just "sawzalled")?
Selling and letting registration lapse are two very different things.
Most states only criminalize driving a car with a expired registration. I have half a dozen cars with expired registration to my name and it's a non issue as long as I don't get pulled over driving them (which would be really hard to do since they either aren't complete enough to be drive-able or have long since been turned into tin cans).
I suggest a better headline: a 22-year old man with a suspended license crashes into stopped Michigan police car.
Regardless what your opinion is on the naming, autopilot is a form of cruise control and legally and operationally requires driver attention, 100% of the time.
I'd say the CEO advertises it otherwise.
https://www.wired.com/story/elon-musk-tesla-autopilot-60-min...
https://www.tesla.com/sites/default/files/images/autosteer-e...
A disclaimer, like a clickwrap TOS, is not enough to take away Tesla's culpability. You can't sign away your life.
"Adaptive Cruise Control"
"Lane Keeping Assistance"
"Automatic Emergency Braking"
The same names (standardized names, really) used by other auto companies offering similar capabilities.
I mean, imagine this in any other industry. CEO: "Our new FireSafety-O brand insulation is fire safe!" [Some houses catch fire] Company lawyers: "Well, didn't you read the post-purchase disclaimer that says it's made out of shredded newspaper and kerosene, and is only suitable for use in a non-oxidizing atmosphere? Ignore the CEO, he's deranged."
> Update on the coast to coast autopilot demo?
> Still on for end of year. Just software limited. Any Tesla car with HW2 (all cars built since Oct last year) will be able to do this.
Date of tweets: 2017, almost 4 years ago. They've been advertising "coast to coast autopilot" and "Full Self Driving" features for years now.
That's the thing: the features of this are are named "autopilot" and "full self driving". What do you expect people to do when they spend $5000 or $10,000 on "full self driving" ?
It's dishonest to dismiss this with "regardless of your opinion on the naming".
In that case, the law has not yet decided.
https://arstechnica.com/gadgets/2021/03/judge-rules-5-billio...
But if a plane on autopilot crashed because the pilot wasn't paying attention, the pilot would be at 100% fault. That's the origin of the naming, even if it's confusing to a layperson. I'm not here to argue about whether they should call it this or not.
Expecting the same level of competency and understanding of tools from an airline pilot, and the every day driver who hasn't passed a driving test since they were 16 years old, is ludicrous at best.
They've had this capability for decades now; it was mentioned in a 5th season Mythbusters episode (the one about talking a civilian into a landing).
To put it another way, if a driver suffers a seizure on the highway, and Autopilot was able to avoid all collisions and bring the driver safely to the side of the road, what do you think the headline should be?
“Driver suffers seizure on interstate, automatic driving system steers car to safety”
Or:
“Man successfully drives on interstate while having an incapacitating seizure”
"We sold you a car that drives, we never said it can drive uphill!"
Regardless of you opinion, autopilot kills people, because its not possible to suddenly take over from an automoua car that decided to kill you, in 50ms. We have mountains of evidence on reaction time
And the fact we only hear about one of these kinds of crashes every few months essentially proves it's true.
Remember, more than 100 people are killed on the roads each and every day of the year in the USA. Dig into the stats, one Tesla crashing does not mean that overall Autopilot is not effective.
> Autopilots do not replace human operators. Instead, the autopilot assists the operator's control of the vehicle, allowing the operator to focus on broader aspects of operations (for example, monitoring the trajectory, weather and on-board systems).
When you are piloting a vehicle it should be razor sharp that you are holding a weapon in your hand that is deadlier then a handgun.
What a clown!
I can't disagree. Autopilot in a car should mean the same as it does in "Airplane!" or "Wall-E".
> The driver of the Tesla, identified as a 22-year-old man from Lansing, was ticketed for failing to move over and driving with a suspended license.
And there's the actual problem. You can blame Tesla all you want, but you'll have a hard time proving that the brand/model/features of car this person chose to drove has anything to do with the crash.
Shirley, you should keep your jokes PG! Wait, Airplane is PG? Oh man, the 80s were a different time...
An "Airplane! autopilot" requires... services... to operate. That was apparently PG in the 80s but not PG in today's society!
The funny thing is that what an autopilot does in a real airplane is actually pretty close to what the Tesla autopilot does, if you set it to fly you into a mountain it will happily do so (abet with a lot of alarms going off in a modern airliner).
Sure, a basic private aircraft wing-leveler or heading keeping system will fly you into a mountain, but the autopilots in passenger airlines can take off, fly a complete course, and land the plane.
They're monitored by trained professionals with thousands of paid hours and hundreds of hours of training, of course.
CAT-III autoland is a thing but it's actually only used in very limited circumstances, not least because it increases runway safety margins in a way which restricts runway utilisation. Many busy international airfields don't even have a CAT-III certified runway. The aircraft won't automatically cycle through to the correct mode to land (there are a bunch of other steps required to put then autopilot into a double/triple redundant mode etc).
Tesla may be trying to optimize costs and be aggressive as they have been in other areas, doing the whole MVP/iteration thing, but this is one place where that just isn't necessarily the greatest strategy. They should be aiming to be better than human drivers not just as-good-as, and the challenges and limitations of ultra complex software based systems development are hardly some new revelation. They should have been prepared for that too with more hardware backstop.
Namely, this user wrote:
To add even more detail, as there still appears to be a lot of confusion:
The radar systems in these vehicles send out a radio pulse in a broad approximate-cone forward. They get bounces back from everything that reflects radio in front of them. Distance from the object is calculated by time between pulse and response. Speed towards/away from the object is calculated from Doppler shift of the radio frequency.
There are two main things that these systems can't detect.
1. Speed of the object perpendicular to the direction of radio wave travel.
2. Location of the object within the approximate-cone the radio pulse travels in.
Note that thanks to the second, you can't calculate the first with higher-level object tracking, either.
So the data you get back is a list of (same-direction velocity component, distance) pairs. There's no way to distinguish between stationary objects in the road and stationary objects above the road, to the side of the road, or even the surface of the road itself.
Radar just doesn't provide the directional information necessary to handle obstacle detection safely.
fake it til you make it
We built a fancy computer vision model to detect and track football and it turns out a bald referee in a sunny weather can trick our model to think his head is the ball. So I wouldn't be surprised to know there are some flaws in their models too.
It's way safer to have other sensors and don't rely only on cameras if the purpose is engaged autopilot that takes control from the driver away.