108 comments

[ 0.24 ms ] story [ 198 ms ] thread
From the sounds, it seemed to detect it at the last moment. Can Model S owners confirm?

Also, there are variable settings for breaking, right? Has anyone tested what happens if they're intentionally set too low to prevent accidents?

So many experiments I'd run with a Model S and an EM mocked up crash pad...

This is a known issue. Bottom right of page 64:

https://www.teslamotors.com/sites/default/files/Model-S-Owne...

> Warning: Traffic-aware cruise control may not brake/decelerate for stationary vehicles, especially in situations when you are driving over 50 mph (80 km/h) and a vehicle you are following moves out of your driving path and a stationary vehicle or object is in front of you instead. Always pay attention to the road ahead and stay prepared to take immediate corrective action. Depending on TrafficAware Cruise Control to avoid a collision can result in serious injury or death.

Is there a technical explanation anywhere for why this is so hard for the system to handle? I get that if a car ahead of you suddenly swerves out of the way, revealing a stopped vehicle that was completely blocked from view previously, that the vehicle (like a human) can't react until it sees it. But it's clear the vehicle had time to react and didn't until the crash was imminent.
I'm not completely sure but I believe it has to do with the decision to not use LIDAR, in favor of just a camera and some other basic "radar" system. The cost of LIDAR is pretty expensive, yet coming down. I imagine in their AP 2.0 hardware update, it will include LIDAR for all directions. Like Google's car.
My Subaru (which just uses cameras) definitely would've stopped for this. Perhaps the difference is that the Subaru uses two cameras to get a stereo view, whereas the Tesla is trying to do it from a single camera.
Some companies are working on solid state lidar in the price range of $250.
Think barriers around a curve. The car doesn't know that you're going to turn so it must ignore stationary objects(for now). To the sensor suite following an S-curve might look the same(car following going right). That said it seems like AP/AEB aren't cross-sharing data and could do better there.

[edit] Looks like AEB did engage at the last minute, but that only happens when there will be a collision.

So at this point that the lane-keeping (automatic steering) wasn't engaged? How did the guy let this happen if he was steering at the time?
if you have to stay aware of road conditions with a risk of serious injury/death for not doing so, what is the purpose of using an auto-pilot feature?

I suppose you could go hands free because in the event of a road hazard you might just have to brake (although coming to a stop on a highway is dangerous - the person behind you might not stop).

For me the value of an auto-pilot feature is the gain of time. One can respond to emails, read, field a call w/out risking the safety of themselves or others, etc.

...unless said situation above happens and you slam into the back of another vehicle...the traffic slinky effect is harsh.
Good to hear that Tesla knows it's an issue.

Bad to hear that they released it knowing this was an issue, because I wouldn't expect myself to be able to react to this.

Humans are not capable of handling these types of nuances well at speed. Being an active driver and being an inactive driver that needs to stay alert enough to catch the few cases where you run into "known issues" is just not going to work well.

As I'm typing it I'm suddenly having a wave of guilt for all kinds of software bugs I've written where the answer was "it's a known issue".

This is how the main character's father dies in the movie Disturbia, in the opening scene. The car they're following suddenly swerves aside to reveal a stationary truck. This is why you always leave a large gap between you and the car in front of you. It's almost impossible to avoid otherwise if the car in front of you can't be seen around.
Tesla is not a self-driving car at this point. The fact that people rely on this as the primary means of breaking is completely crazy.
When you only have to brake in 1% of the use cases, and the "Autopilot" handles the other 99%, is it really that crazy that a driver does not instinctively recognize that 1%?

Or is it crazy to expect a user to be able to react instantly to the 1% use case?

This is not very different from any normal car with cruise control.
(comment deleted)
Perceptually it's very different. And it's perception that matters a whole lot here, unfortunately.
A normal car with normal cruise control does not keep itself in a lane. Nor does it adjust for the speed of other cars. All it does is make your speed a constant; you still have to steer and initiate any and all braking or overtaking maneuvers.
Some do adjust speed. The point being, neither is a fully automated auto-pilot - the human behind the wheel is always responsible for stopping the vehicle or steering away from obstacles in an emergency.
It isn't a matter of recognizing the 1%. You should be braking for yourself 100% of the time. If you occasionally fail to do so, the car will manage to save you 99% of the time.

It is crazy to see a situation in which you should brake and to intentionally choose not to with a non-autonomous car.

However, adaptive cruise control is not a new thing. Other car brands does not have this issue.
I disagree: Most ACCs have exactly the same limitation.
If you read the driver's own comments on YouTube, he did react to the situation in time. His decision of action however was in error. He decided to take partial control of the vehicle by grabbing the steering wheel to change lanes and merge right. What he needed to do was take full control of the vehicle by tapping the brake to disengage the cruise control aspect of the AP.
Am I the only one thinking this issue is going to seriously injure/kill someone sooner rather than later, and then Tesla will be having a big problem?

If a person was behind that stopped van, trying to figure out what the van's problem was, or if they had the van jacked up on one corner to change a blown tire, this same crash could easily have been fatal.

This already happens with "dumb" cars, and anecdotally, I've heard this independently from multiple tow truck drivers / roadside emergency responders that they specifically park their vehicles, and stand in specific spots, to avoid getting sideswiped or smushed. You also see this with state police conducting traffic stops in the highway: they pull their vehicle over, but not completely in line with the person they pulled over, giving them a little protection by putting their vehicle in the way... or they approach on the side of the vehicle that isn't adjacent to traffic.

Stationary vehicle, in a traffic lane, on a major artery, is a bad situation whether or not your car is piloting itself.

This is an artefact of the sensors (radar) used. Since resolution and processing is insufficent right now, everything that is stationary is ignored.

I have a recent BMW with ACC that shows the same behaviour.

It's not an ultra rare event (I experience it roughly once every other day) and it's also not a problem as a driver.

Also with ACC: Just swerve right and ACC will happily crash you into any other stationary object like a lamp post or a wall.

> Depending on TrafficAware Cruise Control to avoid a collision can result in serious injury or death.

What is the sense of it then? If you can't depend on it, you need to keep alert constantly, with your hands on the wheel and foot ready to hit the brakes.

Which essentially defeats the purpose of TrafficAware Cruise Control.

[Edit: a user was kind to point that Tesla's Autopilot ISN'T an autopilot, and should NOT be relied upon to avoid accidents, requiring indeed full attention when driving]

Yes. You are required while driving a vehicle to keep your hands on the wheel and foot ready to hit the brakes. With any semi-autonomous driving system, as a driver, YOU are the one that is responsible for all actions that the vehicle takes.
Thanks for the clarification, this actually sounds like a feature I'd stay away from. Not only it doesn't solve all cases enough to let me not worry about it, but also creates the false impression (even if I'm the one creating that impression myself) that I don't need to worry about it.

It's almost like american football helmets.

I totally agree. I'm curious to see the statistics as the marketing of "semi-autonomous" vehicles gets more widespread, but I suspect there is a dangerous sweet-spot of capability of semi-autonomous vehicles.

My concern is that semi-autonomous vehicles become good enough that humans trust them to drive (that is, people consider the vehicle to be driving rather than providing an aide to driving such as automatic-shifting, and standard cruise-control); yet not so capable as to be a true autonomous vehicle. Do these features simply making driving safer across the board, or is there a point where the human stops taking responsibility and accidents increase?

Given how many people text and otherwise don't pay full attention to their driving today, does anyone seriously believe that as increasingly capable adaptive cruise control/lane keeping/automated braking/etc. systems become common that large numbers of people won't completely disengage from driving? M

Most of them may even know that they're supposed to be actively driving but, seriously, that's not how humans are wired. And tsk, tsking about how people shouldn't do that isn't going to change anything.

It will be "interesting" to see how this plays out because it's apparent that we're rapidly heading for a point where we'll have cars that are "pretty good" at highway autopilot that are in no way certified or approved for hands-off operation by drivers playing a video game.

Humans have been making mistakes with every new assistive technology. When ABS was optional, people with ABS cars would pump the brakes and end up not stopping in time. With cruise control, there are urban legends of RV drivers going back to make a sandwich.

Eventually people figure this stuff out, but as was discussed in other Tesla threads, there is a risk of entering an uncanny valley where it's actually more dangerous to have the assistive tech in an emergency, but safer in non-emergencies.

There might be a weird HN edit thing going on, but it seems like you are answering a "what is..." question with a Yes.

Not that I have an answer per se -- if I still have to be driving, inputting the obvious strategy doesn't help me much because I already know how to do that. Collision avoidance is pretty much the only thing I have anxiety about, so saying it's not supposed to be used for that means this is a very very expensive mid-range sedan (to consumers like me).

There was no HN edit thing going on. I did answer a "what is the point" with a "yes". I haven't yet decided whether I did this for rhetorical reasons or because it's early in the morning.

I was more responding to the second statement with, essentially: "Yes, it does default the 'autonomous driving' purpose of TACC".

Regular cruise control is even more pointless.
I actually like cruise control, since it's use case is very specific, and always works: keeping a certain speed.
It's useless in the city or densely populated areas. It's great on the open road, e.g. in the south western US, where you can drive in a straight line at 70mph for several hours without braking at all.

In that case your right foot thanks you for cruise control.

Back when I did more longish drives on thinly traveled roads in the South, I used it more. I've gotten out of the habit of using it because in the Northeast, even in rural areas, there tends to be too much traffic for it to be really worthwhile.
It's not like the TrafficAware Cruise Control is meant for completely autonomous driving. It's supposed to be a convenience. To not pay attention to the road, regardless of how good your cruise control is, is not just idiotic, it's life threatening.

Have they not mentioned, time and time again, that this is not an autonomous car? I have adaptive cruse control in my Mazda, automatic breaks, but they're not a replacement for paying attention to the road.

The problem is that Tesla's taken systems intended as driving assists and branded them using terms like "Autopilot" that imply that a human doesn't need to be in the loop. Then you end up with people who haven't read the manual (because no one actually reads the manual) and let their car ram itself into a parked van because the brakes were "the autopilot's job".

I'm of the opinion that Tesla probably needs to back down on their "autonomous" stuff-- their half-baked semi-autonomous stuff is getting enough bad press that it's going to kill true autonomous tech before it can even make it onto the road.

Exactly this, someone needs to tell the Tesla PR department to scale back. Same goes for the 'bio weapon' defense thing.
>The problem is that Tesla's taken systems intended as driving assists and branded them using terms like "Autopilot" that imply that a human doesn't need to be in the loop.

Even in aviation, 'autopilots' don't mean the pilot can leave the cockpit unmanned. Human pilots are needed at all times in case the autopilot disengages due to unexpected conditions.

The distinction being that, when an airplane's autopilot disengages, the pilot's typically going to have a couple minutes to evaluate the situation and determine how to react appropriately, because airplanes typically don't just fall out of the sky. Tesla's "autopilot" gives you seconds to react if you wait until it disengages, if it knows to disengage at all.
Most Tesla purchasers aren't pilots; the meaning communicated to them but the branding "autopilot" isn't based on experience in aviation, it's based on popular culture.
I understand that cruise control is not autonomous driving. Still, I would expect cruise control to NOT crash into a stationary object right in front of the car. If you read the disclaimers of TrafficAware in the manual, the only possible conclusion seems to be that you simply cannot use this feature. If you can't trust the system to brake in time, it means you have to do the braking yourself. Every time.

The bottom line: TrafficAware is bad technology.

Adaptive cruise control's purpose is to let you not have to break just because your cruise control is set slightly too fast for the car in front of you.

It's not intended to be an autopilot. Just a better cruise control

Telsa is just hyping up the same shit that all automakers can do right now.

> It's not intended to be an autopilot.

Tesla brands this feature exactly as 'Autopilot'.

Because Telsa wants to make you think its doing something that BWM isn't. But it's entirely sales puffery.
It's fine and good for everyone to claim that the driver must keep alert constantly and it's their responsibility. However, this feature is inconsistent with that statement as it is damned near impossible for most people to stay engaged and alert when they are not actively engaged in a process. This is a known failure of the human brain and in fact is the reason that on road trips I don't use cruz control and tend to speed because when I'm engaged, I'm paying attention.
This would be a bad situation even for a human driver.
The car ahead of the Tesla handled the situation just fine.
It did. The first car in the lane to encounter an obstacle will have no trouble avoiding it. The driver probably saw the obstacle from very far away.

However, if you, the human, are following said car, you may have fractions of a second to react, after the car moves out of the way and you can finally see the obstacle.

Granted that with certain exceptions, such as following a large van or truck, it is not possible, but good defensive driving means _also_ paying attention to what's in front of the car in front of you. I often react to things before the driver in front of me does (which means I'm not reacting as abruptly, which means the knucklehead 20 feet behind me also doesn't hit me).

EDIT after watching the gif; yeah, this was pretty much the opposite case: big van stopped, clearly visible over the compact car in front. Gross negligence on the part of the driver to not see this and respond _before_ the car did.

How? Every other driver on that road is able to avoid this obvious obstacle in broad daylight.
I wouldn't even classify this as an issue, this is CRUISE CONTROL, not auto-pilot. It's meant to meter your speed not your heading, and the heading is what needed adjusting. Driver was clearly not paying attention or he would've followed the car the Tesla was likely tracking for speed.

I don't think the tech is at fault here at all, if you were using regular cruise control in this scenario nothing would've changed.

I mean if the CC slammed on the brakes every time it passed a bridge pylon or a stationary car THAT would be broken.

> this is CRUISE CONTROL, not auto-pilot

I give the same response as above. Tesla names this feature 'Autopilot'. The tech is not at fault (though it could be better), but the user expectation and understanding of the tech is faulty, and I think part of this is the name Tesla is using. In this case their ambition is ahead of common sense.

See 'Autopilot' as a major feature on their site:

https://www.teslamotors.com/models

What we have here, is a car that behaves like a T-Rex. It can see and react to moving obstacles, but as soon as something stands still, it loses all perception.

In all seriousness, how is this an issue at all?

It clearly sees obstacles (like a wall in a curve) and takes corrective action. How does it not do something (if even just applying the brakes) when there's an obvious obstruction right in front of it?

I can't imagine the amount of anxiety it would cause drivers if they knew this. Rather just drive myself.

Its like the Microwave saying that most of the time it'll work fine but may blow up the house sometimes, please pay constant attention and stop immediately if you see smoke.

But... that's already true. Microwaves burn all the time (for small values of all the time). You DO have to pay attention, at least periodically. I never set it going and leave the room for instance.
If anyone thinks that such a warning automatically absolves Tesla of responsbility for ensuring this sort of stuff doesn't happen then they're grossly mistaken. My big rant 2 weeks ago [1] covered exactly this. Real-world design means that you don't just slap a disclaimer or warning sticker on and expect that to be enough. You must, as the designer, anticipate misuse of your design that is encouraged by proper use of it and work to eliminate that capacity of misuse.

In the same way that modern web server systems try to sanitise database inputs, etc. rather than just blame the developer for not knowing best-practices themselves, real-world engineering is all about understanding how the system may be misused and designing to eliminate the capacity for misuse.

Yes, you'll never eliminate misuse completely, but the court of law is littered with examples that didn't bother to try and so couldn't defend against a claim of negligence in their design engineering process.

Simply saying in a manual that 'hey, this may cause you to crash in these circumstances - pay attention!' isn't enough.

[1]: https://news.ycombinator.com/item?id=11680576

It's like sugary drinks and smoking cigarettes: You can jump up and down and scream the obvious to people, but they will deny it for short-term pleasure.

From a corporation's perspective, in an at-least-partially subsidized healthcare system, such as the U.S. and EU (to different extents, duh!) the health issues are an externalized cost.

It's still a great track record, but I can't shake my misgivings about automotive autopilot beta testing being an open-road thing. This seems like begging for trouble, with the biggest issue being the occasional ambiguity about a momentary brake-tap turning a system off.
But even with occasional issues like this one, it's already half as likely to cause accidents than a person[0]. At this point it'd actually be a worse for us to not test this on open road.

[0]: http://www.telegraph.co.uk/technology/2016/04/25/elon-musk-t...

Lying with statistics. Compares cruise control with human drivers. But think about that for a second...

Highways have an accident rate about 1/3rd of urban roads.

Where is cruise control mostly used? Highways.

How can you compare human drivers on urban roads AND highways with a system that is nearly always used only on highways.

If you compare Tesla's stat with humans on highways only.. Tesla is actually less safe.

That stat is complete bullshit.

Adaptive cruise control is not autopilot. It does not steer for you. It's not in beta, it's been a production feature on high end cars for 10 or so years.

Regular cruise control has always deactivated with a momentary brake tap. Been this way since cruise control started coming on cars.

Based on my experience, when adaptive cruise control is braking, manually braking has unexpected consequences.

A tap can disable the current braking completely, resulting in an unexpected change in velocity. Frequently a bad idea, since the ACC is braking for a reason.

Applying normal braking pressure can add your braking pressure to the existing braking pressure, resulting in an abnormally large braking force, sometimes enough to initiate ABS on dry roads. Very disorientating, and it makes you unpredictable to drivers behind you.

The solution is to tap, then press a second later, but this requires you to be aware of the need to take manual control early in the situation.

Adaptive cruise control is also not 'Regular cruise control'.
Read the YouTube comments -- the OP has responded to many questions there.
I cannot wait to be killed by some idiot putting 100% faith in his Tesla to do his job of driving for him.
Probably less likely than being killed by an idiot putting 100% faith in their driving ability.
I'm sure it will eventually be safer than human drivers once the technology is there but for a time there are going to be deaths that could've been prevented had the driver not overestimated the autopilot's abilities.
If you're worried about getting killed by other drivers, you should simply avoid driving a car. Hundreds of thousands of people die that way every year, and 0% of those cases have involved Teslas so far.
Would you prefer an 18 year old snapchatting at 107 MPH in a Mercedes with a pregnant passenger? Resulting in permanent brain damage for her crash victim?

https://www.washingtonpost.com/news/morning-mix/wp/2016/04/2...

The point is no matter how many accidents autopilot gets into, its safer than human drivers. And it will only improve.

Autopilot could kill ~30K people a year, and that's still less than the number of people killed by people driving in the United States alone.

I'm not sure how this story is relevant, it's well known that human drivers are more dangerous than self driving car. This particular case, while tragic, is not unique.

But this isn't a self driving car - it's automated cruise control. It's not going to let you snapchat and drive safely. You still need to be aware, and rhetoric that presents it as a solution to distracted driving is dangerous, leading to situations exactly like this one.

Sure it's still safer; that's not the point.

(comment deleted)
> Sure it's still safer; that's not the point.

That's entirely the point.

When it is safe enough you can operate the vehicle without also being a responsible driver, then I'd agree with you. But it's not, as demonstrated by this thread.
> Snapchat has a filter that allows users to record their speed of travel, and she wanted to see how fast she could go. So McGee accelerated, then accelerated some more

Really? Snapchat has a feature that is basically designed to get people to use their phone while driving/moving?

Mine picks up the stationary vehicles 9 out of 10 times, but you do have to watch it. This happens most frequently at stop-lights on roads where the speed limit is above 55 MPH and I'm approaching the stationary car stopped at the light.
Cruise control in the left lane? Not paying attention? Ticks all the boxes.
Cruise control that will AUTOMATICALLY DETECT AND FOLLOW THE SPEED LIMIT(!!!) in the LEFT(!!!) lane.

HR would have a field day with all this box checking

apparently tesla drivers are just beta testers
They made that pretty clear when they released a beta version of the software.
"production ready" software seems very difficult without all the data they are gathering from the beta release.
There are a bunch of these videos on YouTube. The best part about them is watching the Tesla Owners' Club (Internet Comment Subdivision) crawl out of the woodwork to explain away why the software worked perfectly and if only the meatbag behind the wheel had read section 74.3(f) of the operating instructions they would OF COURSE have known about this.
While I would hope automakers don't push the technology before it's ready, and relying on disclaimers does seem more CYA than an actual safety measure, it's easy to be defensive when an issue like this has a high probability of being treated irrationally. A meatbag in a conventional vehicle getting into an accident doesn't make the news. An autonomous vehicle completing a trip without incident doesn't make the news. The relative accident rates of autonomous vehicles and meatbag-piloted vehicles doesn't make the news. These are sensitive times for public acceptance of the technology.
The Tesla has a huge screen in the dash. Why not force the driver to read or listen to a summary of the most important sections before they engage cruise control?
I noticed a recent Merc TV ad had a disclaimer along the lines of '*vehicle may not automatically brake in all circumstances'
Wow, what kind of awful sensor technology and low sampling rate are they using for this? You could probably build something that worked better with an Arduino.

Vehicles equipped with these features are actually more dangerous, based upon the preliminary data. I'm all for new technology, but if you want to use clearly "beta" -- and that's putting it lightly, as its clear there are definite hardware/sensor/sampling problems here, not software -- you should be required to carry high risk insurance.

It's not fair to the rest of us drivers who have to be on the road. We did not agree to be Tesla's beta testers too, and its only a matter of time before someone is seriously injured or killed by one of these "features".

I'm sure someone here will chime in and say, "but its safer than a human driver, a reckless teenager". To that I say, that's why teenager's insurance rates are so high! We need to increase the insurance rates here to reflect the risk presented to both the driver and general public of this reckless, half-backed autonomy.

I'm not an owner, but even I know that the Adaptive Cruise control is for use on clear highways and can't cope with much.

This was a situation where the driver plainly needed to take over, had plenty of time to take over, and the issue was sufficiently obvious that even a non-alert driver who was looking forward would see they needed to take over.

There's an argument to be made that relying on partial automation is a fundamentally bad idea because the driver will not become sufficiently alert in time to save themselves. This is not an example of such a situation.

As the driver said in his comments, when he'd seen the system stop 1,000 times in the past he had a false sense of security that it was actually going to stop for him like it was supposed to.
If the driver had used it off-highway 1,000 times in the past, but then the Tesla made an error off-highway, is it Tesla's fault or the driver's fault?

This isn't even a Tesla issue: most car makers now ship cars with adaptive cruise control with stop-and-go. If they stop 95% of the time in this situation, do we really have to ban them because some drivers insist that past experiences constitute a valid test?

(comment deleted)
In a continuation of regulation being outpaced by technology. I feel the current law doesn't properly cover this. My opinion is that, If a car manufacturer is providing a service that controls the brakes and acceleration or turning of your car in an autonomous manner they should be responsible for any accidents that arise because of it.

It is not responsible to expect humans that are sitting behind the wheel of a car that covers 90% of situations to pay attention for the 10%. This sort of situation puts the human at eases and gives them false confidence in a vehicle.

Even if the fine print SAYS they are in control. They are adding a layer between the human and the car that delays their reaction speed a little more. Because the human is not trained to drive in a buggy ass car, they aren't trained to respond to this sort of thing.

Liability for these sorts of accidents should fall on Tesla, and they should not be having their drives be guinea pigs. If a driver wants access to this autonomous mode they should have to sign a release because by default Tesla should be responsible. Again the law isn't current with this.

I also want to say I love what tesla is doing but I think the bad PR of irresponsibly integrating autonomous driving when it isn't ready (or they don't have the proper sensor bank to make it work) is going to delay the acceptance of self driving cars in the public in general.

Tesla's self driving car is to Googles self driving car as the Blue Origin landing was to Space X's landing . But in general people don't know or see the difference they are just cars that drive themselves or rockets that land. Also google's car is typically driven with a trained driver.

I also want to say I'm a huge proponent of self driving cars.

The technology in question is cruise control, not auto pilot or auto steer or anything of the sort. Mercedes' have had radar enabled cruise control since the 80's, and the principle is the same: The vehicle will keep pace with traffic and if you watch the video, that's exactly what it did. The driver maintains full control of the vehicle and it's clear by the sound of the conversation that the driver wasn't paying attention to the road, ergo, driver's fault 100%.
Does it brake and accelerate? If yes, then the car is responsible for braking and accelerating and any damage done while it is braking and accelerating.

Does it turn? No, then no responsibility for turning.

If I can rely on the car to break and go, then I should be able to RELY on the car to do so. This seems like a feature that makes people complacent and will end up hurting self driving car adoption in the long run.

Look at how many people harp on the few accidents Google has been in, and they have a well trained driver in it. Tesla is just putting a 1/2 measure in place so that they can leverage their users as testers and pawning off responsibility when the users turn out to be TERRIBLE testers.

> Does it brake and accelerate? If yes, then the car is responsible for braking and accelerating and any damage done while it is braking and accelerating.

Except that this car's manual, along with every other car I've ever owned and seen that had active cruise control, specifically states that the car will not stop for stationary objects. People need to RTFM. It comes with the vehicle for a reason.

The car appears to accelerate as soon as the car in front drivers around the van. I would think it would wait a second to reevaluate the situation before rapidly accelerating into the van.
Who's got a mirror? Video is private.
Too much assistance make people less aware of danger especially when there are slight perturbations to the handled case:

It happens in airplane industry (where people forgot some basics because they use to much autopilots)

It happens in science when people use too much black magic statistic metadata (and forget to check the details in the slight differences of inputs).

But be aware that more automation of dangerous industry also will have the same effect ; mass transport, nuclear plant, chemical plants, medical practice, energy grids.

Well, aren't coding with google&SO and/or frameworks kind of a growing autopilot for most coders?

More accurate title: Tesla successfully emergency brakes to gently tap the rear end of a van instead of ramming it in what would be a potentially fatal accident