I still don't understand the entire premise here. Who cares if a driver can deliberately defeat these things? That's not the threat model!
A driver nag is an attentiveness reminder. You're driving a car. You're expected to be paying attention. No one else is responsible for safety. So the car wants (for both safety and liability reasons) to make sure you're doing what you're supposed to do. And it works. The car nags me and I tug the wheel to prove I'm there. Most of the time I'm paying attention anyway, but sometimes it's because I got distracted. It works.
If a driver deliberately wants to behave unsafely, there's no way to stop that, because there's an infinite number of ways drivers can behave unsafely. Do we want a system to detect when drivers speed up on yellow? Tesla could absolutely implement that. Why aren't we demanding it? Prevent drivers from approaching cars in adjacent lanes with higher than a 30mph speed delta? Could do that too.
Prevent accelerating in the wrong direction on a road?
Possible.
But no, all that stuff would be dumb, because it doesn't correspond to an actual safety problem with the hardware. We recognize it's driver behavior at fault, and we don't want our cars regulating behavior.
Why is this different?
I submit it's different because "Actually Elon Musk is an Asshole and TSLA went to the moon". Which isn't a very good premise for automobile safety designs.
Is there any evidence that drivers are abusing Autopilot etc. any more than cruise control and causing a higher rate of accidents due to inattentiveness?
You have to get 11 paragraphs in before you get to this:
> Our evaluation does not provide specific insight into the Texas crash, but safety advocates and researchers at CR say that it does show that driver monitoring systems need to work harder to keep drivers from using the systems in foreseeably dangerous ways.
By this standard, cruise control should never have launched in any vehicle until technology was capable of verifying driver attentiveness with eye tracking.
> Who cares if a driver can deliberately defeat these things? That's not the threat model!
Faking a "malfunction" and then suing could be part of the threat model to the car maker and/or the insurance companies that provide liability coverage.
Because the big shiny text that people pay $10,000 for says "Autopilot". Between the name and the large price tag, it conveys that it's higher than L2 autonomy. Putting in the fine print that it's not, doesn't change things.
Which is the most ridiculously meta argument. I have people all over the internet (amusingly never in real life) tell me to my virtual face that I clearly don't understand the product that I paid ten thousand dollars to buy, and that it doesn't do what I know damn well it does.
I dont get it. do you confirm the argument that buyers of tesla autopilot understand its an advanced driver assist system which at this time requires drivers attention? or you do not?
asking becuase the message yours responds to says exactly that: people whoy actually buy autopilot do know very well that it is a system that requests and sometimes even requires driver's attention.
so if you confirm that, then how is a meta argument, ridiculously so?
> Who buys Full Self Driving for 10k and thinks you can’t pay attention?
People who understand how English works? Like, if I spend 10k on something that is "Full Self Driving", I'm paying for the ability for it to drive itself, fully. That is to not watch the road. It's right there in the name. Otherwise call it "Kinda Self Driving"
But given the accidents where people turn on FSD and go to sleep, clearly not everyone thinks that. And the better FSD is, the more crashes are avoided and the higher that percentage is.
> You very clearly buying the right to have access to the newest features as the role out. This is very clearly spelled out when you buy it.
All I know is in my personal experience, everyone who bought FSD feels conned. Maybe you went into it eyes wide open, and if so good for you. Maybe they expected a faster rollout of features, maybe they didn't notice that part of the sales pitch at all.
How big is the disclaimer when you buy FSD? I'm not sure that any disclaimer is really worth the horribly misleading name.
>Do you have any actual data about people deliberately go to sleep?
Well, there was the one where someone went into the back seat to sleep. And there were several other high-profile crashes where people turned on FSD and passed out at the wheel at over 100MPH.
And, we only have data on when the FSD fails, and crashes ensue. Heck, we probably only have data on fatalities because the news doesn't want to get sued.
There were 17 accidents with Autopilot and non with FSD. One of those was a death where somebody fell asleep on the highway.
There are literally 100s of videos of the Autopilot doing an avoiding maneuver that prevented potentially accidents, some potentially deadly.
> Maybe they expected a faster rollout of features, maybe they didn't notice that part of the sales pitch at all.
I think its fair to say you expected the features faster, but the claim its not spelled out is false. If people don't read what is very clearly written on the website when you buy it, then that is their fault.
I understand that it's a proper name and that in English I shouldn't read too much meaning into proper names. The Big Mac isn't all that big. The Nike Dunk didn't let me dunk basketballs as a kid, but it was a cool shoe. One of my favorites was the ole Six Dollar Burger that by the end was costing you more than $6.
Who cares if a driver can deliberately defeat these things?
This is really to do with the expectations. Consider the safety belt, there is an expectation that with the safety belt on, the driver will not go through the windscreen in case of an accident. It is not the case that the driver has defeated the seat belt feature by not using the seat belt. A reasonable person will not have the expectation of having the safety provided by seatbelt without put it on.
Things are not as clear cut for assisted driving as can be seen in this thread.
What author fails to, or pretends not to, understand is that the responsibility clearly lies with the driver. If you "cheat" or ignore the system and there is any type of negative outcome, it is the driver's fault. That's all the incentive you need, and it's the same incentive we have for people driving today.
Level 2 is not intended to be autopilot. It's intended to be safer than driver unassisted.
This comparison just sold me on how morally wrong it is what Tesla is doing. Intentionally misleading and marketing to customers a feature called Autopilot that is only a marginal improvement on what other cars already offer. What if Volvo started calling their (clearly not independent) feature Autopilot and saying it was the future of hands-free driving? Seems inexcusable.
Because the comment is exactly what Tesla argues you have to do but then the unnamed called it morally wrong. Yet, when other systems are found vulnerable to driver hacking, the unnamed falls back to the same argument that Tesla uses, namely driver responsibility.
Go ahead and flag this comment as well if you want Hacker News.
Tesla
“While using Autopilot, it is your responsibility to stay alert, keep your hands on the steering wheel at all times and maintain control of your car.”
Unnamed regarding Tesla Autopilot
“Morally wrong”, “intentionally misleading”
Unnamed regarding this report about other car makers
“ the responsibility clearly lies with the driver. If you "cheat" or ignore the system and there is any type of negative outcome, it is the driver's fault”
Furthermore, when another comment points out that other automakers have similar naming to Autopilot, said unnamed says it is just marketing pressure.
In fact, the article, as read by a human, highlights Cadillac’s Super Cruise.
Now the human who did actual research went to Cadillac’s site which touts
“ Offered on the 2018-2020 CT6, 2021 CT4, CT5 and Escalade, and forthcoming 2023 LYRIQ, available Super Cruise is the first true hands-free driving-assistance feature for compatible roads (2021 CT4 and CT5: late availability expected summer 2021).”
First true hands-free driving-assistance feature for compatible roads
Would this be considered morally wrong or driver responsibility?
How about dealer with both hands behind their head while saying "Look ma, no hands".
“ Offered on the 2018-2020 CT6, 2021 CT4, CT5 and Escalade, and forthcoming 2023 LYRIQ, available Super Cruise is the first true hands-free driving-assistance feature for compatible roads (2021 CT4 and CT5: late availability expected summer 2021).”
“Well since I can use my hands, I’m going to have a cookie.”
unwraps cookie with two hands
Second Cadillac video ad, 0:50
Sofia Vergara doing her lipstick with two hands
—
Hahahahahahahahaha!
In one case, you have a CEO who takes their hand away but the fine print says always keep your hands on the wheel. Unnamed entity finds that morally wrong.
In the other, all material from the company says that you can take your hands off the wheel. In addition, you have videos of an engineer responsible for Super Cruise doing so as well as a dealer driving with both hands behind their head as well as advertising when someone unwraps a cookie with both hands as well as Sofia Vergara doing her lipstick. Apparently, this is not morally wrong.
This comparison just sold me on how morally wrong it is what Cadillac is doing. Intentionally misleading and marketing to customers a feature called Super Cruise that is only a marginal improvement on what other cars already offer. What if Volvo started calling their (clearly not independent) feature Super Cruise and saying it was the future of hands-free driving? Seems inexcusable.
So the summary is that humans can still outsmart current technology that monitors drivers attention. Not bad for humans.
But declaring Level 2 as garbage is stretching it. I am driving Level 2 (ACC plus lane assists plus speed sings recognition plus a few other things) for a few years now and I definitely get less tired on long drives. I recently drove coast to coast and it was more boring than tired because of the Level 2.
As another "level 2 driver" with the same features as OP has, I don't understand how someone could even begin to think it would be safe to "distract themselves" at this early level of automation. The temptation to stay alive wins over any temptation to distract myself.
When I hear reports of accidents in Teslas because the driver "took a nap" or watched a movie or whatever, it's hard to feel empathy, because that's just reckless driving at this level.
Of course Tesla is to blame when they marketed, and maybe are still marketing, level 2 capabilities as being on par with an "Auto Pilot".
Because I know it's just an "assist". Basically the camera sees the white lines in the road and tries to bounce between them while keeping a safe distance to the car in front with a radar.
My car also requires me to put actual pressure on the wheel at all times, I think a 10-15 second lapse is allowed, then it starts to beep.
We do have the anecdotal stories of people buying RVs with "Cruise control", enabling it on a freeway and going to the back to make a cup of coffee. Then being all surprised_pikachu.jpg when they're being dug out from the wreckage.
Every thirty seconds, the Tesla demands attention. If you're distracted, it beeps like crazy and a few times of that, the system disabled itself for the drive.
I also don't "feel" like it can drive itself so paying attention still is warranted.
Rather than saying Level 2 is garbage, we should look at the stats and determine whether it is or is not safer than driving without automation. A perfectly attentive driver will beat automation, but as all the annual deaths while driving show, that is certainly not how universally drive.
I mean obviously, that's why people buy them. Everybody thinks they're an incredibly safe driver and most of them privately think the safety aspect is the smallest reason they buy these systems. They get bought because people want to text and watch Netflix on the way to work.
So I believe there's a tradeoff here. People drive to work all the time staring at their phones and only glancing up at the road every few seconds. They always have the option of turning the assist systems off and going back to distracted driving the old fashioned way. Obviously companies have no financial incentive to do this, but I posit that not making it impossible to use assist systems while completely distracted is actually a net benefit for road safety.
I'd rather have Honda Sense driving a car down the interstate by itself than some dude who's only looking up when the road curves.
> People drive to work all the time staring at their phones and only glancing up at the road every few seconds.
Do you know people who drive like this? Driving this way is (a) very illegal and (b) recklessly endangering other people. I doubt it's especially common, at least in most of the US.
I don't know how common it is, but people do drive like this. From a 2012 news article [1] about Rob Ford, the mayor of Toronto at the time:
Reporter: "Sir, there's a picture that went out on Twitter this morning of you reading while still driving on the Gardiner [Expressway]."
Ford: "Yeah, probably. I'm busy."
Reporter: "So you read while driving?"
Ford: "Yeah, probably, yeah. I'm try[ing] to catch up on my work and you know I keep my eyes on the road, but I'm a busy man."
Later in the article:
In October, Ford was accused of illegally dialing numbers on his cellphone and talking on it as he steered his gold minivan westbound along Dundas Street West near Spadina Avenue.
I've had it show up in my own family when I noticed my Plex server was in use while I knew the person using it was on the road. Needless to say that resulted in an angry discussion.
I have a very different definition of "easy" compared to Consumer Reports.
> Fisher next placed a small, weighted chain on the steering wheel, to simulate the weight of a driver’s hand, and slid over into the front passenger seat without opening any of the vehicle’s doors, because that would disengage Autopilot. Using the same steering wheel dial, which controls multiple functions in addition to Autopilot’s speed, Fisher reached over and was able to accelerate the vehicle from a full stop.
Yes, the good old easy method of putting a weighted chain to the steering wheel and climbing over to the passenger seat.
Why? The only people doing this are “journalists” seeking clicks (or attacking Tesla on behalf of unions or unionized auto makers) or TikTokkers making fake videos.
It’s not possible to watch Netflix on your Tesla screen when the car is not in park. If you want to construct a Rube Goldberg device on your steering wheel so you can watch Netflix on your iPhone in the passenger seat, you have bigger problems than crashing.
So what? This is the most absurd line of inquiry. Did you know that seat belts don’t work if you don’t put them on? The question is not can something be circumvented. The question is if the system is safe. There is no evidence presented here to suggest driver assist systems are a net hazard.
The truth is that some car manufacturers actually request it to be somewhat fool proof, but very few.
It costs hardware to run and cost to develop/test, and car manufacturing is all about price.
But also:
Today the algorithm knows the 3D geometry of your head, and knows that it does not makes sense with a flat face. But it is more important to get it to work good on everyone than to catch foolers.
Disclaimer: i'm one of all the developers behind camera monitoring.
64 comments
[ 2.7 ms ] story [ 135 ms ] thread"tests show human drivers can be fooled too"
A driver nag is an attentiveness reminder. You're driving a car. You're expected to be paying attention. No one else is responsible for safety. So the car wants (for both safety and liability reasons) to make sure you're doing what you're supposed to do. And it works. The car nags me and I tug the wheel to prove I'm there. Most of the time I'm paying attention anyway, but sometimes it's because I got distracted. It works.
If a driver deliberately wants to behave unsafely, there's no way to stop that, because there's an infinite number of ways drivers can behave unsafely. Do we want a system to detect when drivers speed up on yellow? Tesla could absolutely implement that. Why aren't we demanding it? Prevent drivers from approaching cars in adjacent lanes with higher than a 30mph speed delta? Could do that too. Prevent accelerating in the wrong direction on a road? Possible.
But no, all that stuff would be dumb, because it doesn't correspond to an actual safety problem with the hardware. We recognize it's driver behavior at fault, and we don't want our cars regulating behavior.
Why is this different?
I submit it's different because "Actually Elon Musk is an Asshole and TSLA went to the moon". Which isn't a very good premise for automobile safety designs.
It's because the premise is so weak. I can put a brick on the gas pedal, so no car is safe!
Is there any evidence that drivers are abusing Autopilot etc. any more than cruise control and causing a higher rate of accidents due to inattentiveness?
I was especially disappointed in Consumer Reports, which insinuated that these methods of "fooling" Autopilot were somehow connected to a fatal crash in Texas. https://www.consumerreports.org/autonomous-driving/cr-engine...
You have to get 11 paragraphs in before you get to this:
> Our evaluation does not provide specific insight into the Texas crash, but safety advocates and researchers at CR say that it does show that driver monitoring systems need to work harder to keep drivers from using the systems in foreseeably dangerous ways.
By this standard, cruise control should never have launched in any vehicle until technology was capable of verifying driver attentiveness with eye tracking.
Faking a "malfunction" and then suing could be part of the threat model to the car maker and/or the insurance companies that provide liability coverage.
Because the big shiny text that people pay $10,000 for says "Autopilot". Between the name and the large price tag, it conveys that it's higher than L2 autonomy. Putting in the fine print that it's not, doesn't change things.
But I do, I did, and it does. So there.
asking becuase the message yours responds to says exactly that: people whoy actually buy autopilot do know very well that it is a system that requests and sometimes even requires driver's attention.
so if you confirm that, then how is a meta argument, ridiculously so?
People who understand how English works? Like, if I spend 10k on something that is "Full Self Driving", I'm paying for the ability for it to drive itself, fully. That is to not watch the road. It's right there in the name. Otherwise call it "Kinda Self Driving"
But given the accidents where people turn on FSD and go to sleep, clearly not everyone thinks that. And the better FSD is, the more crashes are avoided and the higher that percentage is.
First of all it informs you that it is not jet complete. There is a complete list of what features it has and what features it doesn't have.
You very clearly buying the right to have access to the newest features as the role out. This is very clearly spelled out when you buy it.
> But given the accidents where people turn on FSD and go to sleep
Do you have any actual data about people deliberately go to sleep?
I have seen no numbers what so ever to confirm what you are saying.
All I know is in my personal experience, everyone who bought FSD feels conned. Maybe you went into it eyes wide open, and if so good for you. Maybe they expected a faster rollout of features, maybe they didn't notice that part of the sales pitch at all.
How big is the disclaimer when you buy FSD? I'm not sure that any disclaimer is really worth the horribly misleading name.
>Do you have any actual data about people deliberately go to sleep?
Well, there was the one where someone went into the back seat to sleep. And there were several other high-profile crashes where people turned on FSD and passed out at the wheel at over 100MPH.
And, we only have data on when the FSD fails, and crashes ensue. Heck, we probably only have data on fatalities because the news doesn't want to get sued.
There are literally 100s of videos of the Autopilot doing an avoiding maneuver that prevented potentially accidents, some potentially deadly.
> Maybe they expected a faster rollout of features, maybe they didn't notice that part of the sales pitch at all.
I think its fair to say you expected the features faster, but the claim its not spelled out is false. If people don't read what is very clearly written on the website when you buy it, then that is their fault.
This is really to do with the expectations. Consider the safety belt, there is an expectation that with the safety belt on, the driver will not go through the windscreen in case of an accident. It is not the case that the driver has defeated the seat belt feature by not using the seat belt. A reasonable person will not have the expectation of having the safety provided by seatbelt without put it on.
Things are not as clear cut for assisted driving as can be seen in this thread.
Level 2 is not intended to be autopilot. It's intended to be safer than driver unassisted.
-----
darkerside on June 8, 2018 [–]
This comparison just sold me on how morally wrong it is what Tesla is doing. Intentionally misleading and marketing to customers a feature called Autopilot that is only a marginal improvement on what other cars already offer. What if Volvo started calling their (clearly not independent) feature Autopilot and saying it was the future of hands-free driving? Seems inexcusable.
Go ahead and flag this comment as well if you want Hacker News.
Tesla
“While using Autopilot, it is your responsibility to stay alert, keep your hands on the steering wheel at all times and maintain control of your car.”
Unnamed regarding Tesla Autopilot
“Morally wrong”, “intentionally misleading”
Unnamed regarding this report about other car makers
“ the responsibility clearly lies with the driver. If you "cheat" or ignore the system and there is any type of negative outcome, it is the driver's fault”
Furthermore, when another comment points out that other automakers have similar naming to Autopilot, said unnamed says it is just marketing pressure.
1+1=2, not 3 as per guidelines
Now the human who did actual research went to Cadillac’s site which touts
“ Offered on the 2018-2020 CT6, 2021 CT4, CT5 and Escalade, and forthcoming 2023 LYRIQ, available Super Cruise is the first true hands-free driving-assistance feature for compatible roads (2021 CT4 and CT5: late availability expected summer 2021).”
First true hands-free driving-assistance feature for compatible roads
Would this be considered morally wrong or driver responsibility?
https://www.wired.com/story/elon-musk-tesla-autopilot-60-min...
https://youtu.be/zq43DQ9yVJY
Engineer who created it without hands on wheel!
https://youtu.be/rfV8fqYaHx8
How about dealer with both hands behind their head while saying "Look ma, no hands".
“ Offered on the 2018-2020 CT6, 2021 CT4, CT5 and Escalade, and forthcoming 2023 LYRIQ, available Super Cruise is the first true hands-free driving-assistance feature for compatible roads (2021 CT4 and CT5: late availability expected summer 2021).”
And more, directly from Cadillac.
—
Cadillac website, headline description of Super Cruise
https://www.cadillac.com/ownership/vehicle-technology/super-...
“Designed to take your hands and breath away”
—
Cadillac Super Cruise ads on YouTube
https://youtube.com/playlist?list=RDQMwyZZ5Gf05dA&playnext=1
First Cadillac video ad, 1:11
“Well since I can use my hands, I’m going to have a cookie.”
unwraps cookie with two hands
Second Cadillac video ad, 0:50
Sofia Vergara doing her lipstick with two hands
—
Hahahahahahahahaha!
In one case, you have a CEO who takes their hand away but the fine print says always keep your hands on the wheel. Unnamed entity finds that morally wrong.
In the other, all material from the company says that you can take your hands off the wheel. In addition, you have videos of an engineer responsible for Super Cruise doing so as well as a dealer driving with both hands behind their head as well as advertising when someone unwraps a cookie with both hands as well as Sofia Vergara doing her lipstick. Apparently, this is not morally wrong.
Hahaha
Jajajaja
Then Tesla should change the name and offer refunds to anyone who bought Full Self Driving.
Arguably, the incentives today aren't strong enough given the level of pedestrian fatalities
https://www.caranddriver.com/news/a37260363/driver-assist-sy...
But declaring Level 2 as garbage is stretching it. I am driving Level 2 (ACC plus lane assists plus speed sings recognition plus a few other things) for a few years now and I definitely get less tired on long drives. I recently drove coast to coast and it was more boring than tired because of the Level 2.
When I hear reports of accidents in Teslas because the driver "took a nap" or watched a movie or whatever, it's hard to feel empathy, because that's just reckless driving at this level.
Of course Tesla is to blame when they marketed, and maybe are still marketing, level 2 capabilities as being on par with an "Auto Pilot".
My car also requires me to put actual pressure on the wheel at all times, I think a 10-15 second lapse is allowed, then it starts to beep.
We do have the anecdotal stories of people buying RVs with "Cruise control", enabling it on a freeway and going to the back to make a cup of coffee. Then being all surprised_pikachu.jpg when they're being dug out from the wreckage.
I also don't "feel" like it can drive itself so paying attention still is warranted.
Rather than saying Level 2 is garbage, we should look at the stats and determine whether it is or is not safer than driving without automation. A perfectly attentive driver will beat automation, but as all the annual deaths while driving show, that is certainly not how universally drive.
So I believe there's a tradeoff here. People drive to work all the time staring at their phones and only glancing up at the road every few seconds. They always have the option of turning the assist systems off and going back to distracted driving the old fashioned way. Obviously companies have no financial incentive to do this, but I posit that not making it impossible to use assist systems while completely distracted is actually a net benefit for road safety.
I'd rather have Honda Sense driving a car down the interstate by itself than some dude who's only looking up when the road curves.
Do you know people who drive like this? Driving this way is (a) very illegal and (b) recklessly endangering other people. I doubt it's especially common, at least in most of the US.
> Fisher next placed a small, weighted chain on the steering wheel, to simulate the weight of a driver’s hand, and slid over into the front passenger seat without opening any of the vehicle’s doors, because that would disengage Autopilot. Using the same steering wheel dial, which controls multiple functions in addition to Autopilot’s speed, Fisher reached over and was able to accelerate the vehicle from a full stop.
Yes, the good old easy method of putting a weighted chain to the steering wheel and climbing over to the passenger seat.
The problem with building idiot-proof things is that idiots are way too inventive.
It costs hardware to run and cost to develop/test, and car manufacturing is all about price.
But also:
Today the algorithm knows the 3D geometry of your head, and knows that it does not makes sense with a flat face. But it is more important to get it to work good on everyone than to catch foolers.
Disclaimer: i'm one of all the developers behind camera monitoring.
This is false. Comma.ai/OpenPilot allows this. Video seems under-researched.