> The safety agency informed Tesla weeks ago that it was looking into the spate of crashes in which Tesla vehicles operating on Autopilot failed to detect stopped emergency vehicles with flashing lights. The regulator originally said it was looking into 11 such crashes. A 12th occurred on Saturday, when a Model 3 hit a police cruiser that had stopped behind a car that had broken down on an interstate in Orlando, Fla.
12 crashes into vehicles with flashing lights on them doesn't sound like great success from Autopilot, gonna be honest.
I, and many others, take the position that there really isn't any practical gradation between more-or-less conventional driving, maybe with light lane switching assistance or something, and full on "Level 5" autonomous driving, because the idea that human brains can watch something for hours at a time only to take instant, sub-second actions every few hours is an insane model of human cognition.
I don't disagree with the idea of having a gradient to measure our progress up to full autonomy, but I haven't seen much to say that we should deploy the intermediate stages. Until autonomy can be reasonably guaranteed at at least human levels of safety under the assumption of no human control, it really shouldn't be deployed. "But the human had 673 milliseconds to take control and avert the accident! It's their fault!" is simply not a reasonable defense.
Shouldn't the objection be to anything less than Level 4, rather than Level 5? Level 4 requires that the vehicle can safely stop itself and wait for a human to take over when it is faced with a situation it can't handle.
(There's an argument that if you take away the routine parts of driving, humans won't be able to deal with anomalous situations, but that's a different argument than the attentiveness one.)
If that is possible, yes. I'm not convinced it is possible to create an autonomous vehicle that only panics in situations where there is a safe pull-over, or, if such a vehicle exists, that it'll be considered "autonomous" when it can't go past 25 or 35 mph or something. There's plenty of driving situations that may be confusing where pulling over and stopping isn't possible.
(And, to be clear, I'm not asking for perfect safety, I'm asking for "human or better", and also understand that there may be some interesting fuzziness near the border where the two have different strengths and weakenesses but come out statistically the same. However, I don't feel like we're there yet.)
Mixed control between automated and human for tasks that require persistent attention is bad. We have known this for decades. The example often given in engineering textbooks is 3 mile island. I suspect that the engineers involved in self driving systems know this too, but the liability is too great to make the switch all at once. It seems better to take baby steps into self driving, but history teaches us it should be all or nothing.
Far fewer objects to hit in the air: disengagement means "keep going this way," rarely "and crash into an eventual obstacle". Plus, their early safety record is written in blood like everything else is.
The airline autopilot doesn't have to negotiate anything like the kind of hazards that a car driving system does. Also, pilot training, the presence of a whole flight crew to monitor and catch errors make for a completely different situation.
They have their own records of fuck ups [0]. And for similar reasons as the Tesla AP crashes. Autopilot disengages and leaves the pilot in the lurch. If the pilot doesn't have situational awareness right away it can be fatal.
Airplane autopilots, depending on generation, are full touch. The pilot programs the waypoints. The pilot is alerted and agrees to every course correction and change.
Airplane autopilots don't disengage. They do what you told them to do. They are simple and predictable systems that at most simply follow a predetermined course exactly.
That's not entirely true, there are situations where they do disengage automatically (e.g. due to missing/conflicting sensor information). By the nature of the thing it is a) rare and b) the pilots have more time to react because it won't be a "you're going to hit a truck in 1 second" situation unless something went badly wrong before. But there is still some risk to the handover situation making it more likely pilots misinterpret the current state of the plane or do not realize which automatic support is still active (not just autopilot, also other avionics support (e.g. if I remember correctly one aspect around the Air France 447 crash is that the pilot likely did not realize that the plane had disengaged support systems that would keep him inside a safe flight envelope in normal circumstances).
> The driver is responsible for stopping the car. You literally agree to it when you enable AP. People are too comfortable and not paying attention.
Autopilot encourages people to not pay as much attention while driving, until a critical situation comes up at which point they're expected to be fully in control. That's a design flaw that isn't fixed with a liability shifting "Agree" button.
I haven't used basic cruise control in years. It's actually convenient not to have to pay attention to the speed but I find, on roads with even moderate traffic, it comes at the cost of prioritizing maintaining a constant speed rather than flowing with traffic and generally staying in a lane.
You have no telemetry and no idea what actually happened, car might have veered without enough warning. Which is the whole point of having an investigation, really.
> 12 crashes into vehicles with flashing lights on them doesn't sound like great success from Autopilot, gonna be honest.
I'm sorry, but how can you form an opinion so quickly? 12 crashes out of what? Any number? 12 crashes out of 10000000 scenarios sound like a pretty good record, while 12 crashes out of 15 scenarios indeed doesn't sound like a great success.
But what information do you have that the rest of us doesn't?
I still remember the days when the SciFi of the day sold robotic drivers as "they don't get tired, they don't make mistakes, they are relentless in their pursuit".
Instead of that, we got robots that are apparently unable to avoid crashing into large vehicles with flashing lights. Sometimes.
As usual, the future never ceases to disappoint me.
Ah yes the 1960s. Where politics and economics were also utopian and dishonest. When your average American job could impossibly support a family of 4. 16 year olds drove dodge chargers and the economy was so strong from all of the various nefarious engineering that you could buy products that were made to last 50 years.
"I'm sorry, but how can you form an opinion so quickly?"
It doesn't really matter. They're stationary objects; there's absolutely no reason, good or bad, for the algorithm to determine that the correct trajectory change is into the stationary object.
Humans crash into stationary objects because they're lacking attention almost 100% of the time, whether it's because they're half asleep, drunk, texting, or distracted by just about anything else. Autopilot shouldn't have this issue, so there's no reason for it to fail to recognize the obstacles unless it's been poorly trained, and there's absolutely no reason for it to recognize an obstacle as a valid path or destination.
If you mean "the software is too stupid to be useful here, so let's ignore a whole class of issues, potentially with lethal consequences; abracadabra, it's SEP now", I agree. OTOH, this stretches the meaning of "good" a little bit.
The reason is "our code is too stupid, because is not an actual intelligence and better sensors are expensive"? This is a bad reason from my point of view.
You claim that you have some super duper AI but it can't detect a truck or a wall from a shadow or bag , then your AI is stupid and buy some better sensors and improve your AI.
My takeaway from that article was that there's not enough trust in the raw inputs and possibly a missing continuous weighting system of the sensory interpretations. (humans do this naturally, weighing their various sensory interpretations in real-time simultaneously)
I can't find the specific comment now, but on previous autonomous driving discussions here, it's been asserted that Tesla (and many other systems) simply can't reliably detect stationary objects at speed. If the systems actually did a naive, "that looks like a stationary object, I'm going to avoid it" they'd barely be able to move in a highway setting.
That seems like an enormous caveat to the system overall. Do you know if there is any public justification for going along that route, omitting radar, other sensors, etc.?
AFAIK, it can detect them just fine, but it is bad at discriminating those its collision avoidance part should pay attention to from those it shouldn’t pay attention to (e.g. the road surface, traffic signs on the side of or over the road), so they (¿mostly?) chose to ignore all of them.
Most collision avoidance systems aren't that sophisticated. Radar based systems like those used in Volvo's Pilot Assist suffer from the same inability to discern stationary objects.
Most systems are self contained modules with low power micro-controllers that can emit basic warning and alert data onto the CAN BUS. Tesla's is actually taking the input from it's sensors and attempting to do more in-depth analysis and actually produce a 3D space.
The fact that they're still suffering from the same problem as other manufacturers shouldn't be an admonishment of their technology or attempts unless or until it's proven that their approach is fundamentally worse.
Well, no. No system that is unable to detect and map out a 3D maps of obstacles and plot a course through it without hitting stationary, well visible objects, is not suited to being used in a self-driving capacity. Whether it's from Tesla or anyone else.
If their approach fundamentally means they decided to work around mapping their surroundings in 3D correctly, as the GP suggested, there is a strong reason to admonish.
There are systems that can detect stationary objects, Tesla does not want to use those more reliable systems, so they use the programming methodology of tweaking tweaking the code when a bug appears and hope that was the last one.
Sure some AI gurus will claim that humans can detect this objects only with 2 eyes but they forget to mention that behind those 2 eyes is a brain an actual intelligence that is aware of the world and it's place in the world. So if you can't create an actual intelligence then at least put the effort in feeding correct data into your AI and remove the line of code "if static object ignore because too hard to detect if is a truck or a shadow/paper bag/sign"
> tweaking the code when a bug appears and hope that was the last one
Tesla should also be prohibited from pushing updates. When the software is updated, the code should be tested for at least X thousand miles and X months before a new version may be pushed. Letting an authority do the pushing avoids the temptation of manufacturers to push updates prematurely. It also solves the problem of unexpected changes for the user at random times.
If safety issues are detected, the automobile (or at least the autopilot) should be disabled until the code has been fixed and fully tested.
From what I understand there is also some code that is there to ignore static objects because maybe are signs or bridges and the AI is too stupid. So you have bad sensors with bad code combo.
> There are systems that can detect stationary objects, Tesla does not want to use those more reliable systems
What systems are you referring to? Lidar and radar systems can't reliably detect stationary objects as hazards, because most of the world is stationary and consequently gets filtered out.
Filtering out is the problem that develoeprs/engineers should solve right? I can't accept the idea that the car can hit a wall because "our devs are too incompetent or we are to cheap to use better hardware" , a wall or a truck does not look like a bridge or a shadow or a sign, if the AI is too stupid then is not ready, even if is better then a drunk teen that drives a 15 years old car in some very specific situations.
The lidar/radar detects the objects the AI should label them and decide, if it can't do it then throw more video cards at it and more training or find some competent people that could use more deterministic code to reduce the complexity before the AI attempts to run.
Yes, in principle there are systems that could detect stationary hazards. But as far as I'm aware the best lidar and radar approaches are currently no better than Tesla's RGB approach. It seems that the AI is much more important than the sensor.
The sensors could detect a solid object, the issue seems to be that the sensors will detect many things and the AI or the hardware is incapable to handle the amount of sensors data.
Though the Google car seems to have strong enough hardware to keep track of many vehicles and objects at a big enough distance to not be surprised if something appears or disappears for a frame.
Not sure what people want for a sensor, to detect and label the objects for you? or calculate velocities and predict the future? That seems to be the software developers job but you might need competent people and not throwing big data into a black box and retrain until it mostly works.
The problem is with the sensors, not the AI, and occurs across the industry and isn't exclusive to Tesla. Volvo's Pilot Assist has the same short comings.
How a code that decides what to ignore or not is a sensor issue? Sensors tell you that there is something in front, sure maybe is a sudden thing and there is no velocity/acceleration vector for a few moments but the object is detected ...the software DECIDES what to ignore.
The example where a car changes slane and the code ignores any static object in front of that car seems a clear case of a crash waiting to happen, the car in front might change the lane to avoid something. (and the fact that some other tech that is honestly labeled lane assist has this issue does not excuse a self claimed autopilot to have same flaws."
Let me summarize and let me know where I am wrong:
"Because the software is bad and it triggers too many false positives the coders added a "clever" hack , we ignore most static objects."
> How a code that decides what to ignore or not is a sensor issue?
I guess "Sensor Module" would be a more accurate description.
Most automotive systems are designed as a series of modules that function independently but jointly form a system. They either broadcast or consume messages over the CAN BUS.
The Volvo V60 for example has a Forward Sensor Module, the radar, which outputs a handful parameters like Primary Target Range, Voltage, and Automatic Alignment Offset. It's the responsibility of other body modules to consume that data. The Closing Velocity Module might then broadcast Acceleration related messages which intern would be picked up by the Adaptive Cruise Control Module. Or it might broadcast Collision Warning Messages which the Stereo Module reads and responds by blaring a warning noise out the speakers and also the Brake Control Module receives and applies the brakes.
The horrible truth is that if autonomous driving can save 1+ life, it's better than what we have now so we should keep it and accept that crashes will occur much like they do now with humans in sole control.
There's no reason to kill autonomous driving just because gen 1 - 50 isn't perfect but is an improvement on current driving methodology.
Giving a one-line sentence of what happened disregards other variables such as, lighting conditions, where the stationary object was (Over a hill, round a sharp bend, etc.), how much stopping distance was there. A lot of Tesla hate is purposefully done from EV haters & Musk haters. No one can argue that there isn't an establishment who wants to tear down and prevent Tesla & Musk from succeeding. Teslas will catch on fire (like other cars do), Teslas will crash (like other cars do), Auto-pilot will fail (while also being overall an improvement on driving safety).
We can't assume the stationary object was 300 feet away on a straightaway in a perfect visibility conditions, we have to judge whether a human would've performed worse and whether the same cases are occurring over and over. 12 crashes over a few months is a pretty tiny fraction when there are probably more than 12 humans crashing into stationary objects every hour.
>>
The horrible truth is that if autonomous driving can save 1+ life, it's better than what we have now so we should keep it and accept that crashes will occur
>>
As that '1 life' saved may be countered by a number of lives lost due to the system that might otherwise have survived.
“The horrible truth is that if autonomous driving can save 1+ life, it's better than what we have now”
This is not true at all and we have much higher standards than this. This is like saying if a vaccine cures Covid but manages to kill people at a rate slightly less than Covid, we should embrace it.
>can save 1+ life, it's better than what we have now so we should keep it and accept
So I take it you would be fine with the nightmare version of Apple's phone scanner, continuously scanning your devices for thoughtcrime? After all, "If it saves just one child..."
Any time I hear emotional appeals that ignore the cost side of the equation, I look for a financial motive.
Let's rephrase. If the important upsides and downsides are all represented in the number of traffic fatalities, and that number goes down, then that's good.
The problem with the Apple system is significant downsides that are completely separate from CSAM.
>if autonomous driving can save 1+ life, it's better than what we have now so we should keep it and accept that crashes will occur
This argument ignores the reality of driving: by being a competent, attentive, careful driver you can avoid many classes of accident. Crashes are all about fault, and as a safe driver you can avoid most at-fault accidents. You get to avoid all the self-inflicted damage of drunk/incapacitated driving, unsafe driving for weather conditions, extreme speeding / reckless driving, etc. You still have to deal with the externalities of those, but some fraction of the time you can notice and anticipate these people on the road and stay away. Not always of course, so that still leaves some number of deaths per mile, but I'd bet that threshold is way lower than the "save one life" bar. So if you're a good driver, autonomous cars at the "current death rate" safety bar would make you less safe.
What you get by ignoring this and using the current death rate as a bar is a "transfer of deaths": you take all the deaths that were the fault of the driver (driver dies due to being drunk/high, very reckless driving, etc) and transfer them to random innocent people who were not at fault.
> This argument ignores the reality of driving: by being a competent, attentive, careful driver you can avoid many classes of accident.
It’s funny you use the word reality since this is completely at odds with driving in America. You’re comparing an idealized world of competent drivers to the real world where there are basically no barriers to getting a license and everyone is on their phone behind the wheel. Unless you advocate for stricter licensing and more police enforcement against inattentive driving with equal vigor, this line of reasoning doesn’t hold.
The systems you're referring to are radar based systems. Tesla stopped putting radar in the Model 3/Y in favor of it's vision system.
It would be interesting to see a breakdown of the 12 incidents and the Tesla Models involved. Were the incidents involving radar/vision blended, or vision only vehicles or both?
If the incidents only involved older vehicles that were radar equipped, that would be good PR for the vision only system. If the incidents were in vision only systems, it would be bad. I suspect it'll probably be a little of both.
> It doesn't really matter. They're stationary objects; there's absolutely no reason, good or bad, for the algorithm to determine that the correct trajectory change is into the stationary object.
Are you saying it doesn't matter how often it happens? I'd be a lot more worried about things if something happens always compared to it happening in some cases. Sure, stationary objects should be driven into, I agree. But if this only happens 1 time over the course of 10 years, I'd be a lot more lenient than if it happens once per week, no matter if the reason is "good or bad" (not sure what that means?)
These are situations where failure is not suppose to happen when the driver is paying attention. Given that the auto-pilot is always paying attention when operational, this becomes an unacceptable failure mode because it failed in the most rudimentary way - failure to perceive stationary obstacles and a display of complete lack of situational awareness.
For emergency responders on highways, there are means to reduce the risk of additional collisions by increasing visibility to oncoming drivers (markers, lights, etc.). If the auto-pilot is not able to perceive these even when paying attention, we do have a problem.
It’s not that simple. 12 out of ten million is extremely bad if the average driver takes part in that lottery 100,000 times a day. 12 out of 15 is nothing to worry about if the event is extremely rare.
I guess this lies somewhere between these extremes, but wouldn’t know.
We also don’t know how many of these accidents were prevented by attentive drivers.
When human drivers cause accidents, the causes are collected and analyzed, and laws are proposed to prevent future accidents from happening. It doesn't matter if it's 12 or 12 thousand, or even zero crashes... if there's fear, then society will demand something be done.
The fact that AI can give no easily understood excuses, and improvements to the algorithms is a trade secret means that any fear cannot be abated, aside from outright banning of AI driving.
We just did a 1,500 mile round-trip journey to visit family. Because of the Tesla stuff, I was casually aware of roadside emergency vehicles moreso than usual. We passed 28 police/fire/emergency vehicles that were next to our travel lane.
I saw a number of Tesla vehicles on the road as well, each passing these same emergency vehicles, but not always in the same lane.
By this point, there must be 1M+ incidents of Tesla vehicles passing roadside emergency vehicles in an adjacent lane.
12 crashes is 12 crashes too many, statistically speaking, but it's also a tiny tiny percentage of the total number of cases.
IMO, it is good that there is federal investigation into this. My main concern is that is seems our government agencies at large are not very good when it comes to properly understanding anything technology related, or making relevant decisions/rulings/etc. on these kinds of things.
> IMO, it is good that there is federal investigation into this
> seems our government agencies at large are not very good when it comes to properly understanding anything technology related, or making relevant decisions/rulings/etc. on these kinds of things.
>I saw a number of Tesla vehicles on the road as well, each passing these same emergency vehicles, but not always in the same lane.
How do you know all of them were on autopilot?
>12 crashes is 12 crashes too many, statistically speaking, but it's also a tiny tiny percentage of the total number of cases.
It's also 12 more than any normal attentive human has gotten into, and an average attentive human driver is the absolute floor for self-driving on public roads unless Tesla is going to go to the lengths GM has to ensure the person behind the wheel is remaining attentive.
>IMO, it is good that there is federal investigation into this. My main concern is that is seems our government agencies at large are not very good when it comes to properly understanding anything technology related, or making relevant decisions/rulings/etc. on these kinds of things.
I think they're good at understanding if Tesla loses complete public confidence in self driving that is a net-loss for humanity and technology, as well as a prime source of political shitstorm when people ask why something wasn't done sooner.
> It’s also 12 than any normal attentive human has gotten into
Are you comparing a fleet of cars driving millions of miles to the performance of one ostensibly good driver? Ok, I guess we’ll need perfect systems, see you after the heat death of the universe.
There's never going to be a perfect comparison when the same AI can drive every car in the fleet at the same time, whereas one human brain can drive at most one car at a time.
Yeah but the issue is at hand is specifically 12 cars. You can put 12 humans in cars and watch them brake before hitting emergency vehicles. If they’re claiming AI is better than humans then it’s important to understand why these 12 are so bad.
I get this argument in abstract, but I don't think it's fair when we're talking about obviously unsafe behaviour that literally no human driver should ever do, except in cases of like, extreme medical distress.
It's about the unknown unknowns. If you have Teslas slamming into stopped emergency vehicles with their lights on, and you don't know why, what else don't we know about their ability to perceive and reason about the world that they're navigating in?
This cop[1] tells a lady not to stand between the vehicles because she will get killed if his patrol car gets hit, and seconds later it happens and he says "just like that".
This is the reality of the situation and, frankly, should be a top-level comment. I don't really have a horse in this overall race but IMO your factual perspective is spot-on -- this has happened, plenty of times, by actual humans.
Just like anything else, it may not be common (or it may be 'uncommon') but, to say that it never happens, is plainly, clearly and objectively false.
The failure modes of a new driving system are going to be distributed differently than the old system. This is exactly the same flawed argument as "we should ban all human drivers because in the situation where they get drunk they have lots of accidents and self-driving cars don't have that problem".
That said, I suspect there is some systematic issue with Tesla's system and flashing lights, and I bet they are working like demons to identify and fix it already.
Nope, he is simply saying that a human of sound state of mind does not have these accidents.
The authorities are simply viewing Tesla's credibility as undermined.
They ask themselves, if it can't do this, how can it get anything right, or there must be many more flaws.
The humans do not like to pass over control to machines unless the machines do it clearly better.
And in this case, the authority does not want to have their credibility shot by looking away from the Tesla issue, it cannot be pushed under the metaphorical carpet anymore, there is a bulge under the carpet visible to anyone who wants to see.
No, he's asserting that a normal attentive driver won't crash into a stationary emergency vehicle under remotely normal weather/road conditions. And, personally, I'm not going to trust a system that isn't as good as a normal attentive driver.
>I'm not going to trust a system that isn't as good as a normal attentive driver.
As I understand it, we're not expected to trust the current incarnation of Tesla Autopilot. We're supposed to be ready to intervene in case it screws up.
I'd be interested to know how long the drivers of the 12 Teslas in these accidents had to intervene but didn't (how long was it obvious Autopilot had screwed up before the impact).
My problem with this is that we have little proof that people are any good at this sort of task (i.e. monitoring something without intervening for really long stretches of time).
So if data show that people keep failing at it we have to take a look and decide that it's not a good system, not that the people are at fault.
The challenge, which no one has a good answer for, is that if you do a "good" job of automating something, people will zone out. It's just how we're wired.
Does it have to better than an attentive driver or the average driver? I figure if the Tesla has less crashes per mile than the average driver than that's a success?
Even if the accidents a Tesla does get into are ones that a normal attentive driver wouldn't get into, if it avoids just as many accidents that a normal attentive driver would get into isn't that enough?
While that is true, there are also people who don't drink and drive, don't text while driving, don't drive while overly fatigued, etc. So it's reasonable for those people to not want to use a system that's only as good as an "average" driver.
> So it's reasonable for those people to not want to use a system that's only as good as an "average" driver.
But... no one is forcing anyone to use this system. They elect to. Thus this is the wrong question, but rather if drivers should be given the option to use the system.
If anything it might be reasonable to infer that poor drivers are more likely to use this system, so the standard could plausibly be below that of an average driver while still improving overall safety.
I hate to put it this way, but I think it'll be easier to get people to use autopilot (that performs better than your average driver) than it will be to get people to get gud at driving.
You're an attentive driver right? You've probably noticed people lane changing without looking, causing or almost causing collisions. Or jockeying for position fruitlessly. Or being drunk or getting sleepy. Or getting aggro. Back when I was driving a lot I would have been grateful for anything that meant I wouldn't have to drive around people like that.
I dunno where I'm going with this exactly, I suppose we've had similar experiences and it makes me a little surprised that you'd be against automated systems bringing the skill floor up.
I would even assert that normal inattentive driver won't crash in stationary emergency vehicle... Or if they do they should permanently have their right to drive removed. I would consider giving attention to flashing lights minimum standard.
Approximately 12 percent of all interstate highway deaths result from shoulder accidents. This means that an estimated 600 people a year are killed—and thousands more are injured—while making emergency stops off the highway.
I don't, but statistically speaking, some of them likely were.
Tesla has shipped about 1.6M units globally, with most of those in the US. Let conservatively estimate half of those stayed in the US, and half of that half have used autopilot frequently enough for a high degree of opportunity of these kinds of crashes. That is still 250,000 vehicles. 12 of those being involved in crashes is a pretty low number, and we also do not know how many scenarios they avoided that a human might have missed.
> It's also 12 more than any normal attentive human has gotten into
What is your definition of "normal attentive human"? There are tons of studies showing how humans suck at long-term tasks that are repetitive or non-engaging (like highway driving). People get distracted all the time while driving and cause accidents. Sure an "attentive" human would not run into a police vehicle, but there are not enough of those on the road most days.
I'm sorry but the general computer called your brain is not like the purposebuilt computer of Autopilot. 12 is too many for a computer who's only job is to literally look at the road and drive.
Maneuvering around congested streets I agree is a judgement call that humans and computers can get wrong. Not hitting stationary objects doesn't take a supercomputer to solve it.
I believe that Autopilot and Full Self Driving is perfectly safe. Where is the evidence to the contrary? I am willing to pay $5000/month/car for Full Self Driving.
> By this point, there must be 1M+ incidents of Tesla vehicles passing roadside emergency vehicles in an adjacent lane.
> 12 crashes is 12 crashes too many, statistically speaking, but it's also a tiny tiny percentage of the total number of cases.
That perspective is little more than spin. Most serious flaws are like that.
Any failure that isn't a "tiny tiny percentage of the total number of cases" is pretty serious. If it was even a small percentage (say 0.01%), Tesla autopilot would be banned from the road, and maybe Elon Musk would go to jail (or at least Tesla would probably get bankrupted) for releasing such a hazardous product.
The fact that Tesla and Elon have been fighting these autopilot errors in completely the wrong way is the primary reason the feds should be involved.
I have been using autopilot in Nissan. If the car were to crash into an ambulance light going off I would not be able to predict it and stop it from happening quickly enough. I just have a whole lot of trust in some basic things the car does.
And this is one of those cases where they should have been able to get this right after the first crash but they didn’t and now the feds have to get involved. A lesson on why we can’t have nice things.
We should not allow the Fed to stifle innovation and technological progress. If we are not willing to take risks, then we will never be able to reach out true potential of becoming a truly advanced space faring civilization and a multi-planetary species. The NHTSA does not have the technological capability to understand advanced AI used in Autopilot and Full Self Driving and they should stop their interference immediately. I have Tesla and I bought the Full Self Driving package. I believe it is far more valuable than the current price. I am willing to pay 30 times more for Full Self Driving.
Hopefully the NHTSA is also going to investigate the many other cars with traffic aware cruise control, which have the same problem with stationary vehicles.
I was driving my Rav4 with TACC enabled, and it would have crashed into a column of stopped cars if I hadn't noticed in time and intervened.
Tesla has something like 1000x greater rate of hitting stationary emergency vehicles. The fact that all the other cars have the same issue with tacc actually points even more strongly to something being wrong with Tesla.
>> The safety agency informed Tesla weeks ago that it was looking into the spate of crashes in which Tesla vehicles operating on Autopilot failed to detect stopped emergency vehicles with flashing lights. The regulator originally said it was looking into 11 such crashes. A 12th occurred on Saturday, when a Model 3 hit a police cruiser that had stopped behind a car that had broken down on an interstate in Orlando, Fla.
> 12 crashes into vehicles with flashing lights on them doesn't sound like great success from Autopilot, gonna be honest.
It's really surprising what people will put up with. IIRC, recently "autopilot" had some regressions in the situations it could handle, and Elon Musk tweeted something to the effect of "Yeah, this version sucks since we're retraining all our NNs."
Like how did something that half-baked ever make it to production in a safety critical system?
> The regulator originally said it was looking into 11 such crashes. A 12th occurred on Saturday
But there are 4500 crashes every year, and that's only including ambulances (https://www.ems.gov/pdf/GroundAmbulanceCrashesPresentation.p... pg 22). So why is Tesla being investigated for being involved in 0.25% of these crashes when they made up 3% of car sales in 2020?
Also, radar ignores stationary objects on any car. Why is only Tesla being investigated when this exact same flaw exists in every other car manufacturer that uses radar based cruise control?
I think the right stat to measure is - of all the Tesla cars out there, how many had crashes, and compare this against the average for other car brands/models.
They're looking specifically into crashes happening on autopilot though, so using that 3% figure assumes people are using the tesla in autopilot mode 100% of the time
Tesla is already putting a lot of resources into improving Autopilot and FSD. I'm not sure how regulators are going to help that process, and I think there are a couple possible pitfalls:
(1) Will they take relative risk into account? If the car sometimes crashes but less often than humans, will that be sufficient, or considered a flaw because it's not perfect and regulators hate looking bad?
(2) Will they take progress into account? If the car drives worse than humans but its accident rate drops 10% per year, then after some number of years we'll accumulate fewer total accidents than if we hadn't allowed progress to continue. That's important because progress won't continue without lots of cars on the road using Autopilot and FSD betas.
I never crashed into a stationary vehicle or had a crash. I want a FSD vehicle to be at least as safe as me, not some bad or drunken driver, that's what FSD should be measured against.
If the FSD vehicle is safer than the average driver, but not as safe as you, then you should want the FSD to be allowed by regulators. The average safety of other drivers will increase, lowering your chance of an accident, without degrading your own driving if you don't use FSD yourself.
It isn't whataboutism to ask that statistics include the necessary context. The person you are replying to doesn't give a full and proper context, but that doesn't make attempting to provide that context "whataboutism".
Sorry, but that - "So why is Tesla being investigated for being involved in 0.25% of these crashes when they made up 3% of car sales in 2020?" - pointing at others/numbers to detract from your own mistakes clearly is whataboutism.
Go read about what "whataboutism" is and where it came from. It is a rhetorical technique where you deflect criticism by asserting that the group you are being criticized by is hypocritical because they do the same thing they are accusing you of doing.
There is no implication of hypocrisy here, nor could there be since the NHTSA doesn't make cars...
Instead this is a comment asking a legitimate question about the basis for the investigation given the overall rates of occurence for the incident in question. Other repliers attempted to answer this question, you decided to try to shut down discussion by mis-applying a rhetorical label.
They should, but people here are afraid that this looks like a government overreaction that won't actually find an accurate result.
IMO, they are wrong and this will likely lead to guidance for all automakers, including Tesla. I doubt anything that comes out of this will be a problem for Tesla, if anything it might end up being harder for the others.
The answer seems obvious. Legacy car makers will continue to invest heavily in disinformation and lobbying to try to break Tesla. They can't compete on product, especially when you consider a dealer-free buying experience.
> So why is Tesla being investigated for being involved in 0.25% of these crashes when they made up 3% of car sales in 2020?
How does the proportion of sales in one year have anything to do with the proportion of crashes that involve cars on the road which were purchased at any time? Furthermore, this is Teslas on autopilot mode--not Teslas in manual driving mode
>> So why is Tesla being investigated for being involved in 0.25% of these crashes when they made up 3% of car sales in 2020?
> How does the proportion of sales in one year have anything to do with the proportion of crashes that involve cars on the road which were purchased at any time? Furthermore, this is Teslas on autopilot mode--not Teslas in manual driving mode
Also, a faulty design is a faulty design, regardless of the proportion of total accidents it causes. All faulty designs should be investigated. The theoretical existence of a "faultier" design isn't a get out of jail for all other designs.
Unfortunately, Tesla has a lot of apologists whose main concern seems to defend it for all criticism, even if the arguments required to do that are specious and unreasonable.
Which other car manufacturer has its assistance system crash into emergency vehicles more often than that? If there is one, that certainly should be given the same attention, yes.
Maybe the driving data has nothing to do with crash, or to put it another way, the log of the driver's route and actions up to the point of the crash, is the point.
Hopefully the TSA doesn't try to kill as many people as the FDA has. The big lesson from the pandemic was how poorly these agencies are run, and their complete willingness to see thousands die by implementing restrictions.
Let's ignore the larger point (which is that this is just one of a million possible edge cases where autopilot could systematically fail), I could absolutely see Tesla just bodging in a "If you see emergency vehicle lights flashing disengage" - it'd perfectly reasonable to say that you shouldn't be using autopilot in that situation because you should be paying attention and ready to react to the emergency vehicle.
It should warn and turn off in a LOT of situations imo; there was another accident in a construction zone where the car hit a barrier, for example. And there's the shared videos of people literally sleeping or sitting in the back seat of a Tesla on autopilot; the car should detect that and disable autopilot for an X amount of time.
Of course, at the same time the car should not play police; if someone is caught doing that, their license should be taken away from them.
As much as it would be a solution to Tesla's problem, it would impact Tesla's overall plan and strategy. Automatically disengaging when emergency vehicles are detected is a huge barrier in their selling strategy ("It can drive itself, it's just not allowed yet to for legal reasons, but soon").
I don't think you're wrong policy wise specifically.
I do think that this 'auto pilot ... but you have to manage it' system is not workable for most humans. Paying attention when you sorta don't have to ... is just not paying attention for most people.
It's a space I don't think we manage well outside of say a real pilot in a plane trained to do his thing (and there's actually a lot of cases with trained airline pilots losing situational awareness and auto pilot does the wrong thing and the pilots don't notice soon enough / auto pilot makes bad choices for a surprisingly long time before anyone notices).
I think this might be a space where it is all or nothing auto pilot wise.
If the driver isn’t paying attention already, this can’t work in the worst case. The roads and their speed limits are designed for the driver to safely come to a stop if there’s an obstacle in the road. If you add another 3-10 seconds for the system to disengage, warn the user, have the user focus in the road and act appropriately, you are crashed into the emergency vehicle again.
Biden, when he's not eating ice cream and awake, hates Tesla because they aren't a union shop. This and other things are just a shakedown. You can see by the echo chamber here who's the bot army and who's not.
Remember, the primary purpose of Tesla is to save the planet. But I guess the unions trump that. Fuckers.
I assume (?) Teslas have interior microphones. Maybe not in this case but how easily will cabin interior data be turned over to authorities to paint a picture of what goes on inside and outside a vehicle or nearby vehicles leading up to an accident?
It may be easier to just ask for and release everything. In the name of safety, of course.
The absolving argument that it's still the drivers responsibility to intervene has flown out of the window with this request.
The federal safety agency is not having any of that excuse anymore.
They might find that Tesla did no wrong here, further enabling Tesla to get away with this. But this looks a lot like they want to regulate the feature a bit more.
If self regulation via companies and humans does not work, the government will step in.
Curious to see the outcome, they haven't ordered the immediate disablement of the feature, they might have a neutral view.
The question is, how much of that is Tesla’s fault? They’re an EV company, not a psychology consultancy. We don’t hold gun manufacturers or alcohol companies liable when people do stupid things with their products, despite the best efforts of their marketing divisions. Granted “pull trigger get bullet” is much simpler than the Autopilot response.
We ask airplane makers to make adjustments to better indicate problems / auto pilot status when humans don't do the right thing all the time. And that's for professionals who get tons of training.
We have all sorts of car / road safety features because humans are bad at driving.
We can't reprogram the humans, can change the environment they operate in.
Fundamentally any life form has a slower reaction when they have to notice an issue and start to take motor control. Versus already being in control and simply not causing the issue.
They are not just "an EV company" they are making a guidance system for thousands of high speed metal death machines. They BETTER consult some psychologists. This is a really bizarre thing to say.
You either misread my comment or you’re attacking a straw man argument. I said “they’re not a psychology consultancy”, NOT “I hope they don’t consult psychologists”.
1. You are trying to shift blame away from Tesla and onto the drivers
2. Tesla is doing the same thing by pretending not to understand the brains of carbon based life forms, hoping most people accept it on face value and don't think too hard.
Placing the issue on individual psychology: "why did this guy shoot his own head on purpose, why did this guy drive drunk"
instead of on fundamental human psychology: "oh surely the drivers will simply take over control instantly and without warning after staring straight and doing nothing for 2 hours"
...is the whole problem. They need to consult psychologists instead of crying that they aren't responsible for their own system design.
So it's entirely Tesla's fault. They aren't making cars for Lt. Commander Data. They're making cars for human beings
> We don’t hold gun manufacturers or alcohol companies liable when people do stupid things with their products
I believe we would if the gun design meant it could fire when it wasn't supposed to or if random batches of the alcohol contained a toxic chemical but was label warning the consumer that it was up to them to figure out if the bottle they want to consume was safe.
There's a big difference between users "do[ing] something stupid" and manufacturers "making something that puts unreasonable demands on the user to ensure safety."
> We don’t hold gun manufacturers or alcohol companies liable when people do stupid things with their products
We do though.
gun manufacturers sell firearms that need ~10lbs force to shoot (~2x normal) to law enforcement so keep accidental discharges down.
alcohol companies maybe but there are "Alcohol Server Liability Laws" where a bar/etc can be held liable for damages caused by somebody who got drunk at their bar/etc.
It's absolutely their fault. Folks have been saying from day 1 that aviation has dealt with this very problem. Telsa should know since they're trying to claim they mean autopilot in the technical rather than colloquial sense.
The only reason they haven't faced that is because it's completely devestating to the technology they're trying to market.
It's the counter thesis to the " i drive more alert when drunk" , I know what you mean. Might be that there is no in-between from full self driving and fully humanly operated.
Of course, liability under the current state of affairs will always be on the human, but I think it might be reflected in insurance premiums one day where it becomes economically difficult to pay the premium.
I agree, all the brands should be observed, Tesla simply has the largest attach surface currently, due to most media exposure and let's be honest, loudest and most misleading marketing strategies.
IMO this is a good thing, and should be applied to all manufacturers of driver-assist technology. Tesla has certainly been at the bleeding-edge of overly optimistic and confusing marketing for their product, but others should be on alert too.
FTC should have obliterated Tesla from orbit the minute they called cruise control "Autopilot", and the Tesla beta opt-in fund "Full Self Driving", and any time Elon made any claim that the vehicle was going to be self-driving by X date.
This would have saved several lives, untold injuries, and millions in damages. It would have been disastrous for Elon's wealth and the market cap of Tesla, but then we could at least pretend there's still rule of law in the US.
Thankfully other jurisdictions have actual regulators and are telling Tesla that you cannot use fraudulent names, but who knows how much longer it will take for the US to join the rest of the world in this.
This was definitely a poor name in retrospect, but I can see why they chose the name. The current functionality of autopilot is in fact pretty similar to aviation autopilots. Unfortunately most of the public thinks that an autopilot can fly a plane unassisted, so they naturally expect "Autopilot" to drive their car unassisted.
Is this only for Autopilot data or also for Full Self Driving data? Hasn't Full Self Driving already passed all approvals and perfectly safe? Why would they try to regulate something that is already perfectly safe? I have never heard of Full Self Driving crashing. I think Tesla should charge more for this advanced tech. I think $5000/month/car is a fairer price.
I'm curious what the data requirement will be. Is it just the segmented map? Is it the actual model plus all inputs for n-seconds leading up to the crash? Is it all training data that meets some threshold of similarity to the incident?
It would be nice if the investigation of these incidents resulted in an open standard for analysis and quality control of self-driving systems.
Another point is, legal human operator liability or not, the agency has been alerted of 12 such incidents which could never, ever occur unless a human driver is asleep, fully drunk/under the influence of drugs, some sort of fight happened in the car or was distracted from heavy sunlight to the eyes or had to dodge a hazard or animal on the road.
Their way of thought is simply that if the software can't deal with what is trivial for humans, maybe a recall or better look is in order.
I know, this is not trivial for software, it just looks to not technical people like something that should never happen, and if it does , they loose all the trust in it.
Obviously, there are many bright engineers working at Tesla, not sure how this was deployed and how this slipped QA repeatedly.
This, the authority asks Tesla, before the authority has to answer to the voters , congress and media shows.
This not a feature that Autopilot or any other of the lane keeping systems have at the moment. If we are going to say that this is an issue then pretty much all lane keeping systems would have to be outlawed.
If it is really impossible for people to pay attention without their hands on the wheel at all the time then all lane keeping must be abolished or a standard needs to be established what 'paying attention' means.
Crashing into emergency vehicles happens often even without lane keeping system as well, so its a univeral problem when you have big stationary objects standing around on driving lanes.
What do people on HN actually want or expect in terms or regulation?
Here are some suggestions:
- Only allow any kind of lane keeping if a comprehensive testing that they can detect any obstetrical of any kind and have emergency breaking. This of course would simply mean less such system would be deployed as most companies don't have practical scalable solution for this as of yet.
- Only allow any kind of lane keeping if a comprehensive driver monitoring system is implemented that makes sure driver is alert. Additionally you could require wheel touches every so often in addition to that, if 'hands free' driving should not be allowed.
Tesla is in a good spot here, they have the internal cameras and already have a driver monitoring system yet despite that they have not allowed hands free driving.
Tesla new Vision Autopilot or the more extensive FSD stack (that will do highway driving with v10) should solve the stationary object problem.
I'm sure that if there was any kind of regulation, Tesla would simply comply and move on. In the worst case if regulation would somehow insist on having things named in a specific way, Tesla would just rename their features.
However the point here seems to be non of this is technological problem with Tesla, is simply a situation with no regulation and so every manufacturer picks their own solution. Tesla is the most prominent and most talked about so they are the whipping boy of anybody that wants more regulation.
>- Only allow any kind of lane keeping if a comprehensive testing that they can detect any obstetrical of any kind and have emergency breaking. This of course would simply mean less such system would be deployed as most companies don't have practical scalable solution for this as of yet.
probably this. it's a nice feature but it lulls you into a false sense of security and just doesn't seem to mesh well with human nature. It works well enough most of the time that people start trusting it too much over time
I'm just not sure this is objectively true. Many people on HN seem convinced that there is nothing between all human and all machine that is better but I don't really think this is based on a solid evidence.
We should not allow the Fed to stifle innovation and technological progress. If we are not willing to take risks, then we will never be able to reach out true potential of becoming a truly advanced space faring civilization and a multi-planetary species. The NHTSA does not have the technological capability to understand advanced AI used in Autopilot and Full Self Driving and they should stop their interference immediately. I have a Tesla and I bought the Full Self Driving package. I believe it is far more valuable than the current price. I am willing to pay 30 times more for Full Self Driving.
191 comments
[ 2.9 ms ] story [ 214 ms ] thread12 crashes into vehicles with flashing lights on them doesn't sound like great success from Autopilot, gonna be honest.
I don't disagree with the idea of having a gradient to measure our progress up to full autonomy, but I haven't seen much to say that we should deploy the intermediate stages. Until autonomy can be reasonably guaranteed at at least human levels of safety under the assumption of no human control, it really shouldn't be deployed. "But the human had 673 milliseconds to take control and avert the accident! It's their fault!" is simply not a reasonable defense.
(There's an argument that if you take away the routine parts of driving, humans won't be able to deal with anomalous situations, but that's a different argument than the attentiveness one.)
(And, to be clear, I'm not asking for perfect safety, I'm asking for "human or better", and also understand that there may be some interesting fuzziness near the border where the two have different strengths and weakenesses but come out statistically the same. However, I don't feel like we're there yet.)
Yeah, and that's how it's going to be with car autopilots. Which is why it pisses me off that people want to stop progress on it.
Car AP is way harder than plane AP and saying "But planes can do it, why is Tesla so bad at this" is facetious.
Airplane autopilots, depending on generation, are full touch. The pilot programs the waypoints. The pilot is alerted and agrees to every course correction and change.
[0] https://en.wikipedia.org/wiki/Air_France_Flight_447
Autopilot encourages people to not pay as much attention while driving, until a critical situation comes up at which point they're expected to be fully in control. That's a design flaw that isn't fixed with a liability shifting "Agree" button.
If you do this three times in a drive you have to power off and on the car again to be able to use autopilot again.
I've done maybe 40.000 km on autopilot and it's very aggressive compared to my other car (ioniq electric) when it comes to attention.
The fact that some people purposely defeat this system with weights on the wheel is another matter.
But saying that tesla encourages you to not paying attention is not correct.
All the articles with industry analysts insisting that other driver monitoring systems are nearly perfect have sometimes blinded me to that fact.
A lot of people don't like any of these systems for a lot of the same reasons that people didn't like basic cruise control in the early days.
I'm sorry, but how can you form an opinion so quickly? 12 crashes out of what? Any number? 12 crashes out of 10000000 scenarios sound like a pretty good record, while 12 crashes out of 15 scenarios indeed doesn't sound like a great success.
But what information do you have that the rest of us doesn't?
Instead of that, we got robots that are apparently unable to avoid crashing into large vehicles with flashing lights. Sometimes.
As usual, the future never ceases to disappoint me.
I believe you're thinking of terminators, not automated drivers:
https://quotegeek.com/quotes-from-movies/the-terminator/1136...
Though the "will not stop, ever...", even for an emergency vehicle, might be relevant.
It doesn't really matter. They're stationary objects; there's absolutely no reason, good or bad, for the algorithm to determine that the correct trajectory change is into the stationary object.
Humans crash into stationary objects because they're lacking attention almost 100% of the time, whether it's because they're half asleep, drunk, texting, or distracted by just about anything else. Autopilot shouldn't have this issue, so there's no reason for it to fail to recognize the obstacles unless it's been poorly trained, and there's absolutely no reason for it to recognize an obstacle as a valid path or destination.
https://arstechnica.com/cars/2018/06/why-emergency-braking-s...
(thanks for the link correction)
You claim that you have some super duper AI but it can't detect a truck or a wall from a shadow or bag , then your AI is stupid and buy some better sensors and improve your AI.
Well you don't ask for much
Most systems are self contained modules with low power micro-controllers that can emit basic warning and alert data onto the CAN BUS. Tesla's is actually taking the input from it's sensors and attempting to do more in-depth analysis and actually produce a 3D space.
The fact that they're still suffering from the same problem as other manufacturers shouldn't be an admonishment of their technology or attempts unless or until it's proven that their approach is fundamentally worse.
If their approach fundamentally means they decided to work around mapping their surroundings in 3D correctly, as the GP suggested, there is a strong reason to admonish.
Sure some AI gurus will claim that humans can detect this objects only with 2 eyes but they forget to mention that behind those 2 eyes is a brain an actual intelligence that is aware of the world and it's place in the world. So if you can't create an actual intelligence then at least put the effort in feeding correct data into your AI and remove the line of code "if static object ignore because too hard to detect if is a truck or a shadow/paper bag/sign"
Tesla should also be prohibited from pushing updates. When the software is updated, the code should be tested for at least X thousand miles and X months before a new version may be pushed. Letting an authority do the pushing avoids the temptation of manufacturers to push updates prematurely. It also solves the problem of unexpected changes for the user at random times.
If safety issues are detected, the automobile (or at least the autopilot) should be disabled until the code has been fixed and fully tested.
A self driving system needs to have a 3d understanding of it's environment. It's not acceptable to crash into paintings by Wile E. Coyote.
What systems are you referring to? Lidar and radar systems can't reliably detect stationary objects as hazards, because most of the world is stationary and consequently gets filtered out.
The lidar/radar detects the objects the AI should label them and decide, if it can't do it then throw more video cards at it and more training or find some competent people that could use more deterministic code to reduce the complexity before the AI attempts to run.
Though the Google car seems to have strong enough hardware to keep track of many vehicles and objects at a big enough distance to not be surprised if something appears or disappears for a frame.
Not sure what people want for a sensor, to detect and label the objects for you? or calculate velocities and predict the future? That seems to be the software developers job but you might need competent people and not throwing big data into a black box and retrain until it mostly works.
https://www.wired.com/story/tesla-autopilot-why-crash-radar/
How a code that decides what to ignore or not is a sensor issue? Sensors tell you that there is something in front, sure maybe is a sudden thing and there is no velocity/acceleration vector for a few moments but the object is detected ...the software DECIDES what to ignore.
The example where a car changes slane and the code ignores any static object in front of that car seems a clear case of a crash waiting to happen, the car in front might change the lane to avoid something. (and the fact that some other tech that is honestly labeled lane assist has this issue does not excuse a self claimed autopilot to have same flaws."
Let me summarize and let me know where I am wrong:
"Because the software is bad and it triggers too many false positives the coders added a "clever" hack , we ignore most static objects."
I guess "Sensor Module" would be a more accurate description.
Most automotive systems are designed as a series of modules that function independently but jointly form a system. They either broadcast or consume messages over the CAN BUS.
The Volvo V60 for example has a Forward Sensor Module, the radar, which outputs a handful parameters like Primary Target Range, Voltage, and Automatic Alignment Offset. It's the responsibility of other body modules to consume that data. The Closing Velocity Module might then broadcast Acceleration related messages which intern would be picked up by the Adaptive Cruise Control Module. Or it might broadcast Collision Warning Messages which the Stereo Module reads and responds by blaring a warning noise out the speakers and also the Brake Control Module receives and applies the brakes.
A autonomous driving system which can't detect stationary obstacles is not an autonomous driving system: It's a murder machine.
If they can't do this, they shouldn't operate, full stop.
There's no reason to kill autonomous driving just because gen 1 - 50 isn't perfect but is an improvement on current driving methodology.
Giving a one-line sentence of what happened disregards other variables such as, lighting conditions, where the stationary object was (Over a hill, round a sharp bend, etc.), how much stopping distance was there. A lot of Tesla hate is purposefully done from EV haters & Musk haters. No one can argue that there isn't an establishment who wants to tear down and prevent Tesla & Musk from succeeding. Teslas will catch on fire (like other cars do), Teslas will crash (like other cars do), Auto-pilot will fail (while also being overall an improvement on driving safety).
We can't assume the stationary object was 300 feet away on a straightaway in a perfect visibility conditions, we have to judge whether a human would've performed worse and whether the same cases are occurring over and over. 12 crashes over a few months is a pretty tiny fraction when there are probably more than 12 humans crashing into stationary objects every hour.
>> The horrible truth is that if autonomous driving can save 1+ life, it's better than what we have now so we should keep it and accept that crashes will occur >>
As that '1 life' saved may be countered by a number of lives lost due to the system that might otherwise have survived.
This is not true at all and we have much higher standards than this. This is like saying if a vaccine cures Covid but manages to kill people at a rate slightly less than Covid, we should embrace it.
So I take it you would be fine with the nightmare version of Apple's phone scanner, continuously scanning your devices for thoughtcrime? After all, "If it saves just one child..."
Any time I hear emotional appeals that ignore the cost side of the equation, I look for a financial motive.
The problem with the Apple system is significant downsides that are completely separate from CSAM.
This argument ignores the reality of driving: by being a competent, attentive, careful driver you can avoid many classes of accident. Crashes are all about fault, and as a safe driver you can avoid most at-fault accidents. You get to avoid all the self-inflicted damage of drunk/incapacitated driving, unsafe driving for weather conditions, extreme speeding / reckless driving, etc. You still have to deal with the externalities of those, but some fraction of the time you can notice and anticipate these people on the road and stay away. Not always of course, so that still leaves some number of deaths per mile, but I'd bet that threshold is way lower than the "save one life" bar. So if you're a good driver, autonomous cars at the "current death rate" safety bar would make you less safe.
What you get by ignoring this and using the current death rate as a bar is a "transfer of deaths": you take all the deaths that were the fault of the driver (driver dies due to being drunk/high, very reckless driving, etc) and transfer them to random innocent people who were not at fault.
It’s funny you use the word reality since this is completely at odds with driving in America. You’re comparing an idealized world of competent drivers to the real world where there are basically no barriers to getting a license and everyone is on their phone behind the wheel. Unless you advocate for stricter licensing and more police enforcement against inattentive driving with equal vigor, this line of reasoning doesn’t hold.
It would be interesting to see a breakdown of the 12 incidents and the Tesla Models involved. Were the incidents involving radar/vision blended, or vision only vehicles or both?
If the incidents only involved older vehicles that were radar equipped, that would be good PR for the vision only system. If the incidents were in vision only systems, it would be bad. I suspect it'll probably be a little of both.
Are you saying it doesn't matter how often it happens? I'd be a lot more worried about things if something happens always compared to it happening in some cases. Sure, stationary objects should be driven into, I agree. But if this only happens 1 time over the course of 10 years, I'd be a lot more lenient than if it happens once per week, no matter if the reason is "good or bad" (not sure what that means?)
These are situations where failure is not suppose to happen when the driver is paying attention. Given that the auto-pilot is always paying attention when operational, this becomes an unacceptable failure mode because it failed in the most rudimentary way - failure to perceive stationary obstacles and a display of complete lack of situational awareness.
For emergency responders on highways, there are means to reduce the risk of additional collisions by increasing visibility to oncoming drivers (markers, lights, etc.). If the auto-pilot is not able to perceive these even when paying attention, we do have a problem.
I guess this lies somewhere between these extremes, but wouldn’t know.
We also don’t know how many of these accidents were prevented by attentive drivers.
The fact that AI can give no easily understood excuses, and improvements to the algorithms is a trade secret means that any fear cannot be abated, aside from outright banning of AI driving.
I saw a number of Tesla vehicles on the road as well, each passing these same emergency vehicles, but not always in the same lane.
By this point, there must be 1M+ incidents of Tesla vehicles passing roadside emergency vehicles in an adjacent lane.
12 crashes is 12 crashes too many, statistically speaking, but it's also a tiny tiny percentage of the total number of cases.
IMO, it is good that there is federal investigation into this. My main concern is that is seems our government agencies at large are not very good when it comes to properly understanding anything technology related, or making relevant decisions/rulings/etc. on these kinds of things.
> seems our government agencies at large are not very good when it comes to properly understanding anything technology related, or making relevant decisions/rulings/etc. on these kinds of things.
Not sure how you think it's good then.
How do you know all of them were on autopilot?
>12 crashes is 12 crashes too many, statistically speaking, but it's also a tiny tiny percentage of the total number of cases.
It's also 12 more than any normal attentive human has gotten into, and an average attentive human driver is the absolute floor for self-driving on public roads unless Tesla is going to go to the lengths GM has to ensure the person behind the wheel is remaining attentive.
>IMO, it is good that there is federal investigation into this. My main concern is that is seems our government agencies at large are not very good when it comes to properly understanding anything technology related, or making relevant decisions/rulings/etc. on these kinds of things.
I think they're good at understanding if Tesla loses complete public confidence in self driving that is a net-loss for humanity and technology, as well as a prime source of political shitstorm when people ask why something wasn't done sooner.
Are you comparing a fleet of cars driving millions of miles to the performance of one ostensibly good driver? Ok, I guess we’ll need perfect systems, see you after the heat death of the universe.
It's about the unknown unknowns. If you have Teslas slamming into stopped emergency vehicles with their lights on, and you don't know why, what else don't we know about their ability to perceive and reason about the world that they're navigating in?
Humans do a lot of dumb, irresponsible things on the road, but this is not one of them.
This cop[1] tells a lady not to stand between the vehicles because she will get killed if his patrol car gets hit, and seconds later it happens and he says "just like that".
>this is not one of them.
Yes[2], this[3] is[4] one[5] of[6] them[7].
[1] https://www.youtube.com/watch?v=fZDIwaRWCl0
[2] https://www.youtube.com/watch?v=97D3llj0KSQ
[3] https://www.youtube.com/watch?v=pV5FrWTATDY
[4] https://www.youtube.com/watch?v=loex5U9zD0s
[5] https://www.youtube.com/watch?v=khz9hlpvhko
[6] https://www.youtube.com/watch?v=uyaa6zEHZ2Q
[7] https://www.youtube.com/watch?v=piT4lZBGU4w
Just like anything else, it may not be common (or it may be 'uncommon') but, to say that it never happens, is plainly, clearly and objectively false.
That said, I suspect there is some systematic issue with Tesla's system and flashing lights, and I bet they are working like demons to identify and fix it already.
The authorities are simply viewing Tesla's credibility as undermined. They ask themselves, if it can't do this, how can it get anything right, or there must be many more flaws.
The humans do not like to pass over control to machines unless the machines do it clearly better.
And in this case, the authority does not want to have their credibility shot by looking away from the Tesla issue, it cannot be pushed under the metaphorical carpet anymore, there is a bulge under the carpet visible to anyone who wants to see.
Can you clarify this please? Are you asserting that prior to the 12 Tesla incidents, there weren't any crashes into stationary emergency vehicles?
>I'm not going to trust a system that isn't as good as a normal attentive driver.
As I understand it, we're not expected to trust the current incarnation of Tesla Autopilot. We're supposed to be ready to intervene in case it screws up.
I'd be interested to know how long the drivers of the 12 Teslas in these accidents had to intervene but didn't (how long was it obvious Autopilot had screwed up before the impact).
So if data show that people keep failing at it we have to take a look and decide that it's not a good system, not that the people are at fault.
Even if the accidents a Tesla does get into are ones that a normal attentive driver wouldn't get into, if it avoids just as many accidents that a normal attentive driver would get into isn't that enough?
A lot of people probably feel the same.
But... no one is forcing anyone to use this system. They elect to. Thus this is the wrong question, but rather if drivers should be given the option to use the system.
If anything it might be reasonable to infer that poor drivers are more likely to use this system, so the standard could plausibly be below that of an average driver while still improving overall safety.
It’s not just autopilot.
I don't, but statistically speaking, some of them likely were.
Tesla has shipped about 1.6M units globally, with most of those in the US. Let conservatively estimate half of those stayed in the US, and half of that half have used autopilot frequently enough for a high degree of opportunity of these kinds of crashes. That is still 250,000 vehicles. 12 of those being involved in crashes is a pretty low number, and we also do not know how many scenarios they avoided that a human might have missed.
> It's also 12 more than any normal attentive human has gotten into
What is your definition of "normal attentive human"? There are tons of studies showing how humans suck at long-term tasks that are repetitive or non-engaging (like highway driving). People get distracted all the time while driving and cause accidents. Sure an "attentive" human would not run into a police vehicle, but there are not enough of those on the road most days.
> 12 crashes is 12 crashes too many, statistically speaking, but it's also a tiny tiny percentage of the total number of cases.
That perspective is little more than spin. Most serious flaws are like that.
Any failure that isn't a "tiny tiny percentage of the total number of cases" is pretty serious. If it was even a small percentage (say 0.01%), Tesla autopilot would be banned from the road, and maybe Elon Musk would go to jail (or at least Tesla would probably get bankrupted) for releasing such a hazardous product.
I have been using autopilot in Nissan. If the car were to crash into an ambulance light going off I would not be able to predict it and stop it from happening quickly enough. I just have a whole lot of trust in some basic things the car does.
And this is one of those cases where they should have been able to get this right after the first crash but they didn’t and now the feds have to get involved. A lesson on why we can’t have nice things.
Hopefully the NHTSA is also going to investigate the many other cars with traffic aware cruise control, which have the same problem with stationary vehicles.
I was driving my Rav4 with TACC enabled, and it would have crashed into a column of stopped cars if I hadn't noticed in time and intervened.
What's your denominator? Is this per year or per kilometre driven with TACC active?
> 12 crashes into vehicles with flashing lights on them doesn't sound like great success from Autopilot, gonna be honest.
It's really surprising what people will put up with. IIRC, recently "autopilot" had some regressions in the situations it could handle, and Elon Musk tweeted something to the effect of "Yeah, this version sucks since we're retraining all our NNs."
Like how did something that half-baked ever make it to production in a safety critical system?
But there are 4500 crashes every year, and that's only including ambulances (https://www.ems.gov/pdf/GroundAmbulanceCrashesPresentation.p... pg 22). So why is Tesla being investigated for being involved in 0.25% of these crashes when they made up 3% of car sales in 2020?
Also, radar ignores stationary objects on any car. Why is only Tesla being investigated when this exact same flaw exists in every other car manufacturer that uses radar based cruise control?
I think every crash, where an investigation could prevent another crash in the future, should be investigated.
Shouldn't Tesla "celebrate their mistakes" (yes, morbid, sorry) and see this as an opportunity to improve their product?
(1) Will they take relative risk into account? If the car sometimes crashes but less often than humans, will that be sufficient, or considered a flaw because it's not perfect and regulators hate looking bad?
(2) Will they take progress into account? If the car drives worse than humans but its accident rate drops 10% per year, then after some number of years we'll accumulate fewer total accidents than if we hadn't allowed progress to continue. That's important because progress won't continue without lots of cars on the road using Autopilot and FSD betas.
Go read about what "whataboutism" is and where it came from. It is a rhetorical technique where you deflect criticism by asserting that the group you are being criticized by is hypocritical because they do the same thing they are accusing you of doing.
There is no implication of hypocrisy here, nor could there be since the NHTSA doesn't make cars...
Instead this is a comment asking a legitimate question about the basis for the investigation given the overall rates of occurence for the incident in question. Other repliers attempted to answer this question, you decided to try to shut down discussion by mis-applying a rhetorical label.
IMO, they are wrong and this will likely lead to guidance for all automakers, including Tesla. I doubt anything that comes out of this will be a problem for Tesla, if anything it might end up being harder for the others.
NHTSA is pretty reasonable most of the time.
How does the proportion of sales in one year have anything to do with the proportion of crashes that involve cars on the road which were purchased at any time? Furthermore, this is Teslas on autopilot mode--not Teslas in manual driving mode
> How does the proportion of sales in one year have anything to do with the proportion of crashes that involve cars on the road which were purchased at any time? Furthermore, this is Teslas on autopilot mode--not Teslas in manual driving mode
Also, a faulty design is a faulty design, regardless of the proportion of total accidents it causes. All faulty designs should be investigated. The theoretical existence of a "faultier" design isn't a get out of jail for all other designs.
Unfortunately, Tesla has a lot of apologists whose main concern seems to defend it for all criticism, even if the arguments required to do that are specious and unreasonable.
And Tesla also doesn't call it 'full self driving autopilot'. They simply call it Autopilot.
Of course, at the same time the car should not play police; if someone is caught doing that, their license should be taken away from them.
I do think that this 'auto pilot ... but you have to manage it' system is not workable for most humans. Paying attention when you sorta don't have to ... is just not paying attention for most people.
It's a space I don't think we manage well outside of say a real pilot in a plane trained to do his thing (and there's actually a lot of cases with trained airline pilots losing situational awareness and auto pilot does the wrong thing and the pilots don't notice soon enough / auto pilot makes bad choices for a surprisingly long time before anyone notices).
I think this might be a space where it is all or nothing auto pilot wise.
Ultimately a human problem.
Remember, the primary purpose of Tesla is to save the planet. But I guess the unions trump that. Fuckers.
It may be easier to just ask for and release everything. In the name of safety, of course.
Check out how long it took for even simple safety features to make it into all cars.
> Then the waste majority of cars simply wouldn't come with lane assist. Deploying lidar on such scale is simply not practical at the moment.
IMHO, lane assist isn't "self driving" any more than cruise control is.
I think you get into self driving territory when the car makes lane changes or turns at intersections.
They might find that Tesla did no wrong here, further enabling Tesla to get away with this. But this looks a lot like they want to regulate the feature a bit more. If self regulation via companies and humans does not work, the government will step in.
Curious to see the outcome, they haven't ordered the immediate disablement of the feature, they might have a neutral view.
I'm not sure that's fixable on the human end.
We have all sorts of car / road safety features because humans are bad at driving.
We can't reprogram the humans, can change the environment they operate in.
Fundamentally any life form has a slower reaction when they have to notice an issue and start to take motor control. Versus already being in control and simply not causing the issue.
They are not just "an EV company" they are making a guidance system for thousands of high speed metal death machines. They BETTER consult some psychologists. This is a really bizarre thing to say.
1. You are trying to shift blame away from Tesla and onto the drivers
2. Tesla is doing the same thing by pretending not to understand the brains of carbon based life forms, hoping most people accept it on face value and don't think too hard.
Placing the issue on individual psychology: "why did this guy shoot his own head on purpose, why did this guy drive drunk"
instead of on fundamental human psychology: "oh surely the drivers will simply take over control instantly and without warning after staring straight and doing nothing for 2 hours"
...is the whole problem. They need to consult psychologists instead of crying that they aren't responsible for their own system design.
So it's entirely Tesla's fault. They aren't making cars for Lt. Commander Data. They're making cars for human beings
I suspect insurance would be quite expensive.
I believe we would if the gun design meant it could fire when it wasn't supposed to or if random batches of the alcohol contained a toxic chemical but was label warning the consumer that it was up to them to figure out if the bottle they want to consume was safe.
There's a big difference between users "do[ing] something stupid" and manufacturers "making something that puts unreasonable demands on the user to ensure safety."
We do though.
gun manufacturers sell firearms that need ~10lbs force to shoot (~2x normal) to law enforcement so keep accidental discharges down.
alcohol companies maybe but there are "Alcohol Server Liability Laws" where a bar/etc can be held liable for damages caused by somebody who got drunk at their bar/etc.
The only reason they haven't faced that is because it's completely devestating to the technology they're trying to market.
Exactly, there were some pretty good 99% Invisible podcasts on that topic:
https://99percentinvisible.org/episode/children-of-the-magen...
https://99percentinvisible.org/episode/johnnycab-automation-...
I think the word you're looking for is "dishonest."
Sometimes, "coming soon" is a great answer, vagueness is a powerful tool to help with time demands of clients.
This would have saved several lives, untold injuries, and millions in damages. It would have been disastrous for Elon's wealth and the market cap of Tesla, but then we could at least pretend there's still rule of law in the US.
Thankfully other jurisdictions have actual regulators and are telling Tesla that you cannot use fraudulent names, but who knows how much longer it will take for the US to join the rest of the world in this.
It would be nice if the investigation of these incidents resulted in an open standard for analysis and quality control of self-driving systems.
Their way of thought is simply that if the software can't deal with what is trivial for humans, maybe a recall or better look is in order. I know, this is not trivial for software, it just looks to not technical people like something that should never happen, and if it does , they loose all the trust in it. Obviously, there are many bright engineers working at Tesla, not sure how this was deployed and how this slipped QA repeatedly.
This, the authority asks Tesla, before the authority has to answer to the voters , congress and media shows.
If it is really impossible for people to pay attention without their hands on the wheel at all the time then all lane keeping must be abolished or a standard needs to be established what 'paying attention' means.
Crashing into emergency vehicles happens often even without lane keeping system as well, so its a univeral problem when you have big stationary objects standing around on driving lanes.
What do people on HN actually want or expect in terms or regulation?
Here are some suggestions:
- Only allow any kind of lane keeping if a comprehensive testing that they can detect any obstetrical of any kind and have emergency breaking. This of course would simply mean less such system would be deployed as most companies don't have practical scalable solution for this as of yet.
- Only allow any kind of lane keeping if a comprehensive driver monitoring system is implemented that makes sure driver is alert. Additionally you could require wheel touches every so often in addition to that, if 'hands free' driving should not be allowed.
Tesla is in a good spot here, they have the internal cameras and already have a driver monitoring system yet despite that they have not allowed hands free driving.
Tesla new Vision Autopilot or the more extensive FSD stack (that will do highway driving with v10) should solve the stationary object problem.
I'm sure that if there was any kind of regulation, Tesla would simply comply and move on. In the worst case if regulation would somehow insist on having things named in a specific way, Tesla would just rename their features.
However the point here seems to be non of this is technological problem with Tesla, is simply a situation with no regulation and so every manufacturer picks their own solution. Tesla is the most prominent and most talked about so they are the whipping boy of anybody that wants more regulation.
probably this. it's a nice feature but it lulls you into a false sense of security and just doesn't seem to mesh well with human nature. It works well enough most of the time that people start trusting it too much over time
I'm also not sure how to verify that.