I’m pretty content that “the day when AI in cars can handle ALL conditions on the road, ALL OF THE TIME” (emphasis mine) won’t happen in your lifetime, or indeed the lifetime of any currently gestating fetus.
The insurance industry expects the average insured driver to file a claim every 18 years. That's covering all conditions, all of the time. No "hand off to someone else when I get scared" because humans can't do that.
That's honestly pretty good.
Wake me when autonomous driving can get within a factor of two of that.
human can't handle "ALL conditions ALL OF THE TIME" either
this is a false goal that doesn't exist in the real world
cars, planes, trains, all automation of any kind, all the code you ever wrote, nuclear power plants...all this and more operates with some fallibility that is deemed acceptable given the benefits
no one is excusing AV accidents that seem to defy common sense (randomly driving into barriers etc)...but humans fail in these ways too, every day
In case you’re living under a rock, it already happened. Are they perfect and driving everywhere in every city? No… but here in SF it is not unusual to see cars with no one in them, they’re driving all over the city, at all times, in parking lots, through school zones, etc
Also you’re trying to discredit a well credentialed individual (who worked in automotive safety regulation) based upon age/gender and other irrelevant factors. I feel the need to call that out as toxic.
The rollout beyond sf has been … slow. Which begs the question, how autonomous are autonomous vehicles? Is this a technology or an economics problem, or both?
I agree with the fact that "experts" are quite often wrong and it's too easy to be derailed by appeal to authority.
On the other hand she has a point. As long as the tech won't be 99.99xxxx safe, it will introduce moral hazard (she calls it "risk-homeostasis" but I don't think that kind of neologism is particularly expressive).
At a meta-level, coining some new term could make someone raise an eyebrow. Perhaps she just didn't know about it.
I don't think the poster is trying to discredit her based on age, just contextualizing her comment about it not happening in her lifetime by estimating what period that's likely to refer to.
The comment about her appeal to authority on the basis of being a fighter pilot is also more a counterpoint to crediting her additionally on that basis above any non-expert, since that skillset isn't actually particularly related to autonomous driving.
There have been self driving vehicles in well constrained areas since the 90s or even earlier. They have been capable of complex route navigation, obstacle avoidance and responding to a variety of conditions. The technology hasn't really matured an order of magnitude past the automated vehicles that have been moving things around in factories for decades. The technology improvement has come in computing power and sensor tech where now it's just cheaper and easier to include this in more vehicles. There need to be major leaps before this can really become widespread. I would say at best we are at the Commodore 64 era of self driving cars. I think it's going to take even longer to get to the iphone era considering the result of failure could be death.
> This woman appears to be in her fifties...claiming autonomous driving will not happen in her lifetime
She doesn't claim that. You claim she claimed that. Why make things up?
In the interview she talks about existing autonomous driving applications that are operating today. How can you go from reading that part of her statements to your gross misinterpretation?
This aphorism can be applied to pretty much the entire AI industry right now. Similar to Web3 industry before it, except AI is potentially far more dangerous.
Many of the top AI experts are now coming out sounding warnings. AI Experts have predicted a chance of 10-20% human level extinction from AI, have a history of underestimating AI, but companies are rushing to the precipice, and it is just the early days.
I have a feeling that she has the eyes of the highly trained professional driver (pilot) with good senses and ability to react well.
Not something that can be told about the bulk of the drivers on the road.
Although I see the point of and agree with the necessity of 'crytal clear' situation, knowing what exactly can be expected from the assitance, I believe that the average driver should have as much help as possible. The roads are pretty unsafe yet, because of the human drivers.
I disagree that drivers should be assisted, and so does the interviewee in the article. She uses the phrase "risk homeostasis" to describe the phenomenon that if people feel they are getting helped they will engage in more risky behavior.
I think roads and cars should be made to look less safe than they are, so the average driver will pay more attention.
Did any of you read the article? She acknowledged there are some cases where it’s working in well defined environments.
But she clearly shows how we are currently in the most dangerous phase “car drives except handing off to human drivers in a crisis”.
“ The policy should be that either the computer is driving or you are driving. And by driving I mean steering—people do fine with regular cruise control. The act of keeping your hands on the wheel and guiding the car’s lateral motion is enough to keep your brain engaged. So, no L3 [full self-driving, but the driver must be ready to take the wheel], which is too confusing, and no hands-free L2 [partial self-driving]. I am not against the passing of control per se, but there should just be two modes of operation, with crystal clear feedback about which mode you are in.”
Shit like this makes you realize how behind the curve NHTSA and other government regulators are. Why aren't they putting more guardrails around self driving? What do they stand to lose by enforcing common sense ideas like the ones in the article?
Because the USA is a country ruled by large corporations and/or wealthy individuals, not by the people. The large corporations wants to be free to do self driving experiments, so they do it.
They person interviewed, Missy Cummings, was appointed by the Biden administration to be a senior safety advisor to NHTSA. Elon Musk directed his fans (who are also largely shareholders) to viciously attack her which included sending death threats [1]. She was then ordered to recuse herself from any Tesla related matters due to a petition driven by those same Tesla shareholders.
Out of context, the tweet is largely innocuous as plenty of people might tweet such things. However, immediately following that tweet Missy Cummings was harassed ferociously by the Tesla shareholder base as seen here [1].
One of the leaders of the petition immediately following the tweet was Omar Qazi of WholeMarsBlog who is such a rabid fan and shareholder that he sees no issue with endangering children to support Tesla’s stock price as seen here [2] and the corresponding article. He also advocates for harassing any critics with vicious, baseless accusations [3].
Even if we assume that Elon is not directly organizing the attacks or by proxy, he has deliberately shaped a shareholding fanbase who will at the slightest mention. And, even if we assume that such a virulent base was purely accidental, which it never is, public figures have a moral duty to consider the likely consequences of their actions or speech.
His followers, many of which he engages with and responds to on Twitter such as the aforementioned Onar Qazi, are well known to viciously attack people he identifies as critics. These are not fringe elements, but the core of his fanbase. His actions are all but guaranteed to result in harassment of his critics, which is to his benefit, and he makes no effort to stop such interpretation, but rather harnesses it to suppress his opponents.
You conveniently forgot to mention several things that don't fit your narrative. Why are you so eager to place the blame on Tesla fans for unearthing evidence, instead of calling out Cummings for her unsuitability for the role? Are you actually suggesting that NHTSA requested her recusal based on a simple petition, rather than the hard-hitting evidence it put forth?
Cummings had a seat on the board of a LIDAR company and was paid by them [1].
She has had research funding from competitors in the industry Toyota, Uber, and Volkswagen [2].
She had associations with Tesla short-sellers, which raises questions about her impartiality [1].
Notably, she made public remarks suggesting an inclination to punch Musk [1].
Moreover, her legal team tried to suppress public discourse by attempting to get the petition on Change.org removed [1].
Also, should someone like this really hold an impartial position like the senior advisor of NHTSA? [3][4].
Let's not overlook these potential conflicts of interest just because they are inconvenient to acknowledge.
Hah, tl;dr, but link 1 is sooo neutral, it describes with flowery language about how great Tesla self driving is, and how shitty LIDAR is, after going on attack about her investment in a LIDAR company...
You mean Veoneer, the LIDAR company where she was formerly on the board of directors and formerly a shareholder [1]? Where she resigned from the board on October 25, effective November 1, and divested all of her shares prior to accepting the appointment to remove the potential financial conflict of interest?
It sure is convenient how the blog post by Omar Qazi, a well known shareholder and propagandist who literally spends all of his time marketing for Tesla, made on November 21 fails to mention that event that occurred on October 25. Almost as if he has a massive conflict of interest and will say and do anything to support the value of his Tesla stock, such as endangering children [2]. I can not think of a more unreliable source other than Elon Musk himself (Elon Musk being a more unreliable source due to his even bigger conflict of interest).
And yes, a expert with decades of experience in safety engineering having a informed opinion supported by actual research they have done [3] on products is exactly someone who should have a position like senior advisor to NHTSA [4].
Fair enough, I'll concede that she did resign from the board and sold her shares. However, you've sidestepped the remaining five points in my argument. If these don't raise red flags for you about her suitability for an impartial role, then perhaps we fundamentally differ in our understanding of what's tolerable and what's not.
It seems intellectually lazy to assume that the present continues reliably into the future. You need a more concrete argument than "we've been making progress so far so we'll keep making progress", or if you're on the other side of the argument, you need something better than "this hasn't happened yet so it will never happen".
If the progress of neutral networks-based AI has thought me anything then I expect more and more applications where it's amazingly capable some time, capable most of the time and completely fails still relatively often.
I am really skeptical at this point if it can ever master something to the point where we can safely trust our lives with it.
Two entirely separate fields, both with a similar problem: not good enough. Impressive, but makes mistakes and an 80% solution is not what you want for self-driving cars.
I would struggle to risk my life and my family on even a 99% solution. Is a FSD wants to drive me head on to a railway one day, which has happened, I think I'd rather have been at the wheel. That's just me though. I view my life as mission critical.
The progress recently has been with LLM. Great when you want to talk to your car or car needs to read street sign. Not useful if want to drive the car. It is possible that the techniques transfer over but self-driving is pretty different domain. My guess is that can share general techniques but the LLM breakthrough isn’t useful for self-driving.
Fun story, I was in a loaner car from a dealership which had lane assist. I was driving in the desert around a corner against a cliff so I couldn't see around it very well. Right around the corner there was a small boulder in the road which I had to swerve to avoid, except the lane assist fought me on it since it would cross the painted lines. It wasn't like a near death experience or anything, but it really made me think on how the systems meant to protect us can be what causes a life threatening problem if they aren't designed well. If I had to swerve to avoid another rock or a wayward vehicle, compensating for my own vehicle's unexpected response to driver input is at the bottom of the list of things I'd want.
Even without any obstacles a rented jeep Cherokee felt like it was constantly trying to kill me on mountain roads. The lane assist also turned itself back on every time the car started.
Another baffling feature was that the car refused to move after stopping at stop sign. Turns out there’s a “safety feature” to disable the engine if the seatbelt is not buckled. Thankfully I was just looking for parking and the only consequence was some inconvenienced cars behind me but that could get really bad in the wrong situation.
On my dad's car, the middle back seatbelt has a pretty odd behaviour. It doesn't stretch at all if the car is moving, but it is allowed to contract. This means:
- if the person sitting in the middle has failed to buckle up or at least stretch the belt enough before the car started moving, now they can't do it until the car stops again;
- whenever the person in that place breathes out, they're running the risk of not being able to breathe back in next.
- when the car accelerates, there is a tendency for the person to weigh back into the seat. That causes the belt to contract.
Repeat the 2 latter points enough, and it becomes practically impossible to breathe or even to undo the buckle so you get trapped quite literally dying of asphyxia and in pain from the belt pressing hard against you.
Maybe I’m missing something, but seems like your dad might want to take the car in for service. Malfunctioning safety equipment is a weird thing to let persist like this.
I recall seeing a video of a car who managed to drive in a blocked zone and stop on rails, and eventually got hit by a train as it was unable to drive out - arguably because the driver opened the door, and afterwards the car automatically switched to parking or something.
About a year ago, was driving on a mountain road, the lane markets were eroded and the auto-pilot / driver assist thing stopped turning, If I wasn't paying attention it was straight into the barriers at speed.
I just prefer to drive, I actually don't mind driving either.
Had the same experience while driving through Tahoe in a Toyota Camry rental. Immediately turned that "feature" off because the last thing I want is my car fighting me if I have to make a split second decision.
Lane assist is a product of the pervasive "what works in urban San Francisco/other coastal cities is the right solution for everyone" mentality. Unexpected obstacles are far from the worst of its problems on rural roads. Lane assist doesn't give a fig about the fully loaded dump truck coming around the inside of the curve at a relative speed of 100mph within inches of your door, because that situation just doesn't exist in urban environments. (You are supposed to be all over your white line in this case, in case it's not obvious). Conversely, headlight tech which matters much more in rural environments (oncoming headlights are too dazzling on dark roads, and your own headlights are too dim - current legal tech fails both ways simultaneously) is irrelevant in urban environments, so it is literally illegal to improve.
The greatest trick Musk played was marketing the lane-keeping assist feature of Tesla as “Autopilot” or FSD. It is the best lane-keeping assist I’ll give it that but FSD not in five years and not until commodity LIDAR and map-sensing. Mobileye deserves credit for jumpstarting Tesla’s lane-keeping assist early on.
I think most people would agree that the middle zone is not ideal for most drivers.
However, there may be no other way to someday reach level 5 for general driving, other than to live with that middle zone for a while (keeping in mind that it’s already safer than humans driving alone) in order to conduct the training needed to reach level 5.
You don’t just wish, or regulate, level 5 into existence. You need to build toward it by having real systems deployed in the real world. Those real systems are not going to be level 5 at first or even level 3. This is where we are today, for general driving we only have level 2 and some level 3 if that.
Training is needed not only for the systems but also for drivers (including other drivers in traditional vehicles, they are also being trained unwittingly) and pedestrians, and regulators. Everyone is learning here. It’s not just an overnight, snap of the fingers kind of thing.
I believe the article speaks to the lack of data: we don’t know if L3 is safer. Safer than a drunk driver or a tired long haul, maybe. But a typical driver that would also use L3 responsibly, it may not be safer.
One other huge disadvantage of L3. As driver physically drive less and less while models learn, their skills will wither and when the car gives up control it will be to out of practice drivers.
This training in the wild sounds scary at first, until i remembered that we've done something similar with air travel, basically "training" with live people and fatalities. But over the years we did reduce those from 40 crashes/mil to 0.1 crashes per million flights. I would not want to fly on those early jets in the 50's, but enough people actually took that chance!
40 crashes per million flights has still got to be better than current road safety, at a very rough calculation the US for example had 128 road deaths per million people, which I'm admittedly not sure how to convert to make a fair comparison.
> However, there may be no other way to someday reach level 5 for general driving, other than to live with that middle zone for a while (keeping in mind that it’s already safer than humans driving alone) in order to conduct the training needed to reach level 5.
Gotta call bullshit on this. If human-piloted cars were bedecked in sensors, recording drivers' inputs and outputs, we could collect all the training we could need for L5 driving. It would take a ton of stringent curation, but that's going to be necessary anyway. It's not like Tesla has magically learned not to hit emergency vehicles stopped in traffic yet.
Or, alternatively, we can slowly make roads safe for any AI driven cars.
Use appropriate marking on the roads everywhere, clearly visible signs, clearly visible crosswalks, physically separated lanes for bicycles and vehicle similar to bicycles etc. etc. etc.
Funny thing that we will simultaneously make roads safer for all cars and pedestrians and cyclists too.
Yesterday I was on a two lane country road where I saw a man standing on the opposite side of the road holding up a bright orange baseball cap in his hand and his other hand in a "stop" motion. I slowed down enough to where I didn't hit the crashed car which was just around the bend, but I may have hit it without the advance notice because the turn had very poor visibility. If I were an AI driver, I probably would have considered the man "not-an-obstacle" and promptly ignored him and hit the other car. After all, I'm trained to slow down when I see people in hi-vis with proper "slow traffic" signs. Some random person whose neighbor crashed and is waving the brightest color thing they have is not in the training data.
Or in other words, there are signals humans can send to other humans which can be understood even if you haven't seen that particular signal before, and those will be the biggest challenge for an AI driver, at least if it resembles the current generation of AI.
I did read the article. It’s ironic given her anti-Tesla stance how Tesla exactly follows what she recommends… “No hands-free L2” is pretty much exactly what Tesla urges and enforces its drivers to follow. Is the enforcement perfect? No, of course not; humans are simultaneously stupid and creative.
Try reading it again. She thinks that either the driver should control the car or the car should control the car. Hands on a wheel while the car steers is not a driver controlling the car. Tesla does exactly what she objects to.
Wayne did a study several years ago IIRC and determined that even their trained, professional drivers being paid for this exact task could not reliably, effectively and quickly take over driving when a self-driving car made a mistake. It’s a task that humans are really quite bad at.
I know I am. Steering a car on an average road: not a big deal. Taking over on time when a new, borrowed Tesla tries to mess up on that same road: not so easy. Also, it takes more concentration than just steering the car normally.
This is one of the things that really annoys me. We have so much data and case studies of the effects of automations (of different levels of sophistication) and user interfaces and how those interact with other things including distractions through FAA investigations.
To say nothing of other safety critical industries.
We know what works ok. We know what’s dangerous. We know what’s lethal. There are lessons to look at.
And yet Tesla and a lot of people seem content to re-learn the same lessons through blood over and over again. Like this is 100% incomparable to anything that’s ever happened before.
Sounds like a very good person to have in that position. Being a single pilot in an aircraft, she would not only have years of experience relying on automation, but also risk-reward decision making involving life or death. Self-driving cars are not a "restart the container" situation.
‘Former pilot’ seems like a fairly irrelevant part of the resume of a human-autonomous vehicle interaction researcher with a PhD in systems engineering and a series of professorships either side of an NHTSA advisory position.
For most people "former navy pilot" is more likely to establish credibility by association. Not that this kind of journalism is worthy of praise, but it is pretty prevalent.
Her being a major stockholder and board member of a LIDAR company is also important context, but was left out for some reason. She is pushing a technology approach Elon has called “a fool’s errand” which explains a lot of her incongruous hostility toward Tesla.
How it is important context? The article isn't about LIDAR.
And if LIDAR is the wrong way to go for technical reasons, then if she pushes for better autonomous driving technology, she is working against her supposed interests.
When they leave things out it’s up to you to connect the dots, or not. Her association with a LIDAR company suggests she is not a neutral voice here when it comes to evaluating solutions.
Flippant. By that logic he wouldn’t work at Twitter or SpaceX. The point is she has an agenda to make money off her stake in a competitor, so her opinions should be seen in that light.
“Has an agenda to make money”implies an evidence based assumption of knowledge of likely motives, and that she did not divest her stake before taking a governance role.
She did divest preemptively, as described elsewhere in this thread.
Her position will be kept open for her so she still has an interest. And we don’t really know what deal she may have with the company. She certainly has a close relationship with them. My understanding is that she had stock options, which can be granted again.
What company, Veoneer? The company where she resigned from the board and sold all of her shares prior to accepting her appointment as a senior safety advisor to NHTSA?
We should believe Elon Musk who has the largest current conflict of interest in the world over a woman who has no current conflict of interest and deliberately took action to terminate any potential conflict of interest she may have had (to be fair that is legally required to hold such a post)?
On the one hand, she's been writing about this sort of thing for years--including at a time when when a lot of people here and elsewhere had bought into the fully autonomous vehicles next year Koolaid. And there are probably a lot fewer people running human factors labs that relate to this topic than there are former fighter pilots. That said, being a researcher who also has first-hand experience as one of the US Navy's first female fighter pilots does bring additional real world credibility--and, yes, probably makes people more likely to click on the headline.
Other former pilot here… I’d say that HCI and understanding how automation works in the user environment is probably more useful than her PhD in systems engineering given the context.
Switching/using correct/monitoring autopilot modes is (somehow) still a major problem in aviation.
It’s not surprising that drivers (who basically get no training) have a hard time with this too, nor is it surprising that manufacturers are blinded by their own BS (see 737 Max for further reference).
Comically the GA airplanes with Garmin units in them were wildly better designed than the “professional FMS” boxes Honeywell sold. In particular, the flag ship “Honeywell Apex” units were terrible in comparison to the touch screen Garmins.
That’s wild, because the Honeywell box is fully integrated with the airplane and an order of magnitude more expensive.
I can see it. The other day a self driving Cruise car was extremely aggressive with me, almost feeling aggressive as I tried to cross an intersection on foot. It was stop/starting quickly towards me to try and get me to walk faster it feels like.
I think that these self driving cars need some kind of feedback system for pedestrians and cyclists. Even an emoji face that shows facial expressions would help. When I see a real driver at an intersection, I feel like we come to some agreement about what is going to happen next based on our facial expressions. Or I can at least guess what may happen and take an action. With the self driving car there is 0 feed back. It's just an empty car that may or may not run me over.
Additionally, I also almost got run over by a dude head down driving in the Mission the other day, typing on his phone with both hands as his he just felt comfy having his Tesla do all the work.
At the moment all of this feels so sketchy. I hate my life being used for beta testing software lol.
My concern would be the potential for an alignment issue. The agent directs it's eyes to look at you - because it's communicating something accurate about the plan it's formed about how to cross the intersection? Or because it's been trained to look at people when it crosses intersections, but communicating isn't it's true objective?
(I'm not trying to imply the agent may fool you into crossing so it can mow you down, the case of "AI incompetence" so to speak is more concerning to me then "AI malice".)
100% agree with this attitude. A human whose attention is directed at you who uses their eyes for input will turn their eyes to you. The eyes are not just signalers, but the person actually needs to turn them to you to direct their attention to you, in some sense, so this is a good indicator that they have done so.
> With the self driving car there is 0 feed back. It's just an empty car that may or may not run me over.
As someone who struggles with nonverbal communication sometimes, I frequently feel this way about all vehicles both as a pedestrian and also as a driver myself.
There are something like 230 million licensed drivers.
So if I was the non-existent "average driver" with no control over risk, it seems to me I should expect a fatal accident roughly once every 100 lifetimes.
Or every 100 million miles.
I don't perceive the state of the art in autonomous driving as anywhere near that kind of reliability.
I’m not in favor of how Tesla has used BS marketing to make their product seem more capable than it is, nor am I excited about how all these companies Over promise and under deliver.
However… driving is one of the most dangerous things we do. 1/100 people (roughly) die in a traffic related accident in their lives. That’s bonkers and unacceptably high.
My estimate of my risk of death is 100/100, so 1/100 does not seem that high to me.
What I think could legitimately be called "bonkers and unacceptably high" is the prevalence of medical errors as a cause of death in the US.
"Analyzing medical death rate data over an eight-year period, Johns Hopkins patient safety experts have calculated that more than 250,000 deaths per year are due to medical error in the U.S."
Since I don't drive drunk, and usually miss rush hour, I do think it's reasonable for me to consider going to the doctor the most dangerous thing I do.
While I agree that the rate of medical mistakes is unacceptably high and the profession is just ridiculous in terms of how they behave from a risk management standpoint… this appears some wild base rate fallacy here, in a big way. The people going to the doctor’s office because they think something is wrong are by definition the more fragile patents. I say this as someone who was misdiagnosed for some time and has dealt with the inertia and bureaucracy of the system.
I spent most of my adult life flying bush planes around Alaska and working to try to make things safer in that (horrifically unsafe) industry. Trust me when I say 1/100 is unacceptable and absurd in the extreme and humans are terrible at assessing risk objectively. Just your statement: “I don’t drive drunk and I usually miss rush hour” is indicative of misunderstanding how this works. That’s ok! Everyone does that, I did too until I was trained on specific ways to think about risk and risk mitigation strategies.
The risk often doesn’t come down to individual action - it’s links in the chain, or the Swiss cheese model of you prefer and when all the factors line up an accident is more likely.
It’s great that you don’t drink and drive, it is also great that you avoid driving at peak hours. But do you actively do risk mitigation before driving? Probably not, more power to you if you do, but probably not.
i guess this is a long way to say "something you are exposed to every day multiple times" is way different than a (albeit terrible) system you are exposed to when you are old and infirm and already prone to death.
still, everyone thinks the 1/100 won't be them so they feel it isn't a problem… that is a poor way to figure out "what should we do."
I see a lot of people arguing from their intuition (or experience in other fields) that what Tesla is doing with their driving assistance features is dangerous. There are millions of Teslas on the road, shouldn't the first step be to find out check whether they crash more often in practice? If not, then what's the big deal?
The data analysis is not straightforward. You would expect a driver's decision to use Autopilot to be correlated with the ability of the Autopilot to handle the conditions. So just comparing Autopilot accidents per mile to human accidents per mile won't cut it.
How do you conclude that? GP’s premise amounts to “if the overall rate of human crashes per million miles is H and the autopilot rate is A, even if A is half of H, that doesn’t tell you anything about the relative safety if autopilot is engaged during a particularly safe subset of overall driving.”
My suggestion is to compare Tesla accidents per mile to non-Tesla accidents per mile. If people are being lulled into a false sense of security by AP then this should show up in the data.
A similar problem exists with that analysis. Tesla ownership is probably correlated to the amount of time/miles spent in driving conditions where Autopilot is effective.
You also add a whole bunch of other confounding factors if you're not comparing same-driver accident rates. Accident rates vary by location, city/highway driving mix, driver attributes (age, gender, credit score, etc), and probably many other things that may be correlated with Tesla ownership.
“ Cummings: There is no evidence of mitigation. At NHTSA we couldn’t answer the question that you’re less likely to get in a crash—no data.”
Compared to what? I find this very hard to believe. Surely forward collision avoidance systems have stopped a number accidents over the years. To say that there is no evidence for it, no data even, seems to suggest that you might have a data problem. Hard to take them seriously when they don’t have good data, yet maintain their very negative position.
The reasoning you are using seems to be popular - I found something similar on a page about Waymo.
If an autonomous driving system clearly prevented or would have prevented a crash that happened to a human, then that's one life saved, is the philosophy. The minimum benefit from everyone using the system is at least one life.
But some people do not see it that way. So you are probably at cross purposes with them. I'm sure the skeptics understand there is evidence for what you think proves mitigation - just not for what they think would prove it.
I haven't got what it takes right now to describe at length the alternative way of looking at things, plus I could have it (or phrase it) wrong. Maybe someone else will comment.
I've tried 'self driving' and driving assist. Driving assist is intolerable for me because it can jerk you around even when you have a reason to be close to the line. I don't like fighting for control of the car.
I am a huge proponent for good self driving but experiencing driving assist and being a passenger in Tesla cars has made me much more cautious of the current capability.
Driving assist is so annoying. It has to be the exact same distance behind the car in front of it, regardless if cars are slowing down ahead or speeding up. No sense of the flow of traffic.
Then it has to be exactly in the middle of the road, even in a two lane one with another car coming towards you.
128 comments
[ 3.4 ms ] story [ 172 ms ] threadWho is if a fully autonomous driving car has an accident.
What about software or sensor errors?
The insurance industry expects the average insured driver to file a claim every 18 years. That's covering all conditions, all of the time. No "hand off to someone else when I get scared" because humans can't do that.
That's honestly pretty good.
Wake me when autonomous driving can get within a factor of two of that.
this is a false goal that doesn't exist in the real world
cars, planes, trains, all automation of any kind, all the code you ever wrote, nuclear power plants...all this and more operates with some fallibility that is deemed acceptable given the benefits
no one is excusing AV accidents that seem to defy common sense (randomly driving into barriers etc)...but humans fail in these ways too, every day
Also you’re trying to discredit a well credentialed individual (who worked in automotive safety regulation) based upon age/gender and other irrelevant factors. I feel the need to call that out as toxic.
I appreciate her credentials, but I don't see why her statements are somehow unassailable
and I stick by my point that her history as a fighter pilot has no bearing
not sure where your "gender" point comes from? other than using the pronoun, I made no claims based on gender
On the other hand she has a point. As long as the tech won't be 99.99xxxx safe, it will introduce moral hazard (she calls it "risk-homeostasis" but I don't think that kind of neologism is particularly expressive).
At a meta-level, coining some new term could make someone raise an eyebrow. Perhaps she just didn't know about it.
The comment about her appeal to authority on the basis of being a fighter pilot is also more a counterpoint to crediting her additionally on that basis above any non-expert, since that skillset isn't actually particularly related to autonomous driving.
She doesn't claim that. You claim she claimed that. Why make things up?
In the interview she talks about existing autonomous driving applications that are operating today. How can you go from reading that part of her statements to your gross misinterpretation?
https://en.m.wikipedia.org/wiki/Existential_risk_from_artifi...
Many of the top AI experts are now coming out sounding warnings. AI Experts have predicted a chance of 10-20% human level extinction from AI, have a history of underestimating AI, but companies are rushing to the precipice, and it is just the early days.
Everyone should listen to this warning: https://m.youtube.com/watch?v=xoVJKj8lcNQ
But Elon's income on the other hand depends on the success of Tesla's FSD.
Well, his income and net worth have done pretty well without delivering FSD, so maybe not?
What would happen if he cancels FSD?
Not something that can be told about the bulk of the drivers on the road.
Although I see the point of and agree with the necessity of 'crytal clear' situation, knowing what exactly can be expected from the assitance, I believe that the average driver should have as much help as possible. The roads are pretty unsafe yet, because of the human drivers.
I think roads and cars should be made to look less safe than they are, so the average driver will pay more attention.
But she clearly shows how we are currently in the most dangerous phase “car drives except handing off to human drivers in a crisis”.
“ The policy should be that either the computer is driving or you are driving. And by driving I mean steering—people do fine with regular cruise control. The act of keeping your hands on the wheel and guiding the car’s lateral motion is enough to keep your brain engaged. So, no L3 [full self-driving, but the driver must be ready to take the wheel], which is too confusing, and no hands-free L2 [partial self-driving]. I am not against the passing of control per se, but there should just be two modes of operation, with crystal clear feedback about which mode you are in.”
FSD has been 5 years away for… 15 years now?
this was only Elon-time hallucinations that made for great marketing
you won't get to Mars on Elon-time either, but that doesn't mean you won't get there
Doesn't mean you will either.
FSD is going to be enabled on HW2 anyday now. Oh look at that HW4 just rolled out.
the past few years it has been "later this year"
[1] https://amp.cnn.com/cnn/2021/10/28/cars/tesla-ntsb-cummings/...
Where'd he do this? Have a link? Curious.
Out of context, the tweet is largely innocuous as plenty of people might tweet such things. However, immediately following that tweet Missy Cummings was harassed ferociously by the Tesla shareholder base as seen here [1].
One of the leaders of the petition immediately following the tweet was Omar Qazi of WholeMarsBlog who is such a rabid fan and shareholder that he sees no issue with endangering children to support Tesla’s stock price as seen here [2] and the corresponding article. He also advocates for harassing any critics with vicious, baseless accusations [3].
Even if we assume that Elon is not directly organizing the attacks or by proxy, he has deliberately shaped a shareholding fanbase who will at the slightest mention. And, even if we assume that such a virulent base was purely accidental, which it never is, public figures have a moral duty to consider the likely consequences of their actions or speech.
His followers, many of which he engages with and responds to on Twitter such as the aforementioned Onar Qazi, are well known to viciously attack people he identifies as critics. These are not fringe elements, but the core of his fanbase. His actions are all but guaranteed to result in harassment of his critics, which is to his benefit, and he makes no effort to stop such interpretation, but rather harnesses it to suppress his opponents.
[1] https://twitter.com/MoodyHikmet/status/1451332679597625344
[2] https://twitter.com/WholeMarsBlog/status/1558122848601812993
[3] https://dawnproject.com/what-is-the-credibility-and-characte...
Cummings had a seat on the board of a LIDAR company and was paid by them [1].
She has had research funding from competitors in the industry Toyota, Uber, and Volkswagen [2].
She had associations with Tesla short-sellers, which raises questions about her impartiality [1].
Notably, she made public remarks suggesting an inclination to punch Musk [1].
Moreover, her legal team tried to suppress public discourse by attempting to get the petition on Change.org removed [1].
Also, should someone like this really hold an impartial position like the senior advisor of NHTSA? [3][4].
Let's not overlook these potential conflicts of interest just because they are inconvenient to acknowledge.
[1] https://wholemars.net/2021/11/21/missy-cummings-is-fighting-... [2] https://twitter.com/wholemarsblog/status/1450710771399200769 [3] https://twitter.com/heydave7/status/1450691154299731970 [4] https://twitter.com/TitterDaily/status/1450832510267629570
It sure is convenient how the blog post by Omar Qazi, a well known shareholder and propagandist who literally spends all of his time marketing for Tesla, made on November 21 fails to mention that event that occurred on October 25. Almost as if he has a massive conflict of interest and will say and do anything to support the value of his Tesla stock, such as endangering children [2]. I can not think of a more unreliable source other than Elon Musk himself (Elon Musk being a more unreliable source due to his even bigger conflict of interest).
And yes, a expert with decades of experience in safety engineering having a informed opinion supported by actual research they have done [3] on products is exactly someone who should have a position like senior advisor to NHTSA [4].
[1] https://www.nasdaq.com/articles/veoneer-says-cummings-to-res...
[2] https://twitter.com/WholeMarsBlog/status/1558122848601812993
[3] http://hal.pratt.duke.edu/sites/hal.pratt.duke.edu/files/u39...
[4] https://www.curbed.com/2021/10/elon-musk-missy-cummings-tran...
I am really skeptical at this point if it can ever master something to the point where we can safely trust our lives with it.
Another baffling feature was that the car refused to move after stopping at stop sign. Turns out there’s a “safety feature” to disable the engine if the seatbelt is not buckled. Thankfully I was just looking for parking and the only consequence was some inconvenienced cars behind me but that could get really bad in the wrong situation.
- if the person sitting in the middle has failed to buckle up or at least stretch the belt enough before the car started moving, now they can't do it until the car stops again;
- whenever the person in that place breathes out, they're running the risk of not being able to breathe back in next.
- when the car accelerates, there is a tendency for the person to weigh back into the seat. That causes the belt to contract.
Repeat the 2 latter points enough, and it becomes practically impossible to breathe or even to undo the buckle so you get trapped quite literally dying of asphyxia and in pain from the belt pressing hard against you.
* The fact that other cars of the same make and model do the same
* The fact that (at least) some other models of the same make do the same
* The fact that apparently I was the only person surprised at it when I mentioned to a few people who have cars of other makes as well
* The fact that it was like this when it was new
Perhaps none of these in isolation mean it's as designed but I reckon taken together they're a very good indication that it is as designed.
I just prefer to drive, I actually don't mind driving either.
However, there may be no other way to someday reach level 5 for general driving, other than to live with that middle zone for a while (keeping in mind that it’s already safer than humans driving alone) in order to conduct the training needed to reach level 5.
You don’t just wish, or regulate, level 5 into existence. You need to build toward it by having real systems deployed in the real world. Those real systems are not going to be level 5 at first or even level 3. This is where we are today, for general driving we only have level 2 and some level 3 if that.
Training is needed not only for the systems but also for drivers (including other drivers in traditional vehicles, they are also being trained unwittingly) and pedestrians, and regulators. Everyone is learning here. It’s not just an overnight, snap of the fingers kind of thing.
Gotta call bullshit on this. If human-piloted cars were bedecked in sensors, recording drivers' inputs and outputs, we could collect all the training we could need for L5 driving. It would take a ton of stringent curation, but that's going to be necessary anyway. It's not like Tesla has magically learned not to hit emergency vehicles stopped in traffic yet.
Funny thing that we will simultaneously make roads safer for all cars and pedestrians and cyclists too.
Or in other words, there are signals humans can send to other humans which can be understood even if you haven't seen that particular signal before, and those will be the biggest challenge for an AI driver, at least if it resembles the current generation of AI.
I did read the article. It’s ironic given her anti-Tesla stance how Tesla exactly follows what she recommends… “No hands-free L2” is pretty much exactly what Tesla urges and enforces its drivers to follow. Is the enforcement perfect? No, of course not; humans are simultaneously stupid and creative.
Wayne did a study several years ago IIRC and determined that even their trained, professional drivers being paid for this exact task could not reliably, effectively and quickly take over driving when a self-driving car made a mistake. It’s a task that humans are really quite bad at.
I know I am. Steering a car on an average road: not a big deal. Taking over on time when a new, borrowed Tesla tries to mess up on that same road: not so easy. Also, it takes more concentration than just steering the car normally.
"So, no L3 [full self-driving, but the driver must be ready to take the wheel], which is too confusing, and no hands-free L2 [partial self-driving]."
Let's zoom in on the part so you can read it, too:
Tesla requires its users to keep their hands on the wheel at all times, so they agree.You have your opinion about this, which you explain, but it differs from her opinion as expressed in the article.
Key point (read the quote):
She puts hands-on L2 inside the bounds of what she accepts.
That is also exactly what Tesla currently provides.
To say nothing of other safety critical industries.
We know what works ok. We know what’s dangerous. We know what’s lethal. There are lessons to look at.
And yet Tesla and a lot of people seem content to re-learn the same lessons through blood over and over again. Like this is 100% incomparable to anything that’s ever happened before.
Tech is so powerful now, it controls narratives, influences politics, huge revenues.
I feel tech is beyond control at the moment.
Does it?
"Tesla's Autopilot AI team drives the future of autonomy of current and new generations of vehicles"
And if LIDAR is the wrong way to go for technical reasons, then if she pushes for better autonomous driving technology, she is working against her supposed interests.
She did divest preemptively, as described elsewhere in this thread.
We should believe Elon Musk who has the largest current conflict of interest in the world over a woman who has no current conflict of interest and deliberately took action to terminate any potential conflict of interest she may have had (to be fair that is legally required to hold such a post)?
Switching/using correct/monitoring autopilot modes is (somehow) still a major problem in aviation.
It’s not surprising that drivers (who basically get no training) have a hard time with this too, nor is it surprising that manufacturers are blinded by their own BS (see 737 Max for further reference).
Comically the GA airplanes with Garmin units in them were wildly better designed than the “professional FMS” boxes Honeywell sold. In particular, the flag ship “Honeywell Apex” units were terrible in comparison to the touch screen Garmins.
That’s wild, because the Honeywell box is fully integrated with the airplane and an order of magnitude more expensive.
I think that these self driving cars need some kind of feedback system for pedestrians and cyclists. Even an emoji face that shows facial expressions would help. When I see a real driver at an intersection, I feel like we come to some agreement about what is going to happen next based on our facial expressions. Or I can at least guess what may happen and take an action. With the self driving car there is 0 feed back. It's just an empty car that may or may not run me over.
Additionally, I also almost got run over by a dude head down driving in the Mission the other day, typing on his phone with both hands as his he just felt comfy having his Tesla do all the work.
At the moment all of this feels so sketchy. I hate my life being used for beta testing software lol.
At one point I saw a proposal to put cartoon eyes on self driving cars that could turn and “look” at you to achieve this effect.
Actually seems like a good idea to me!
(I'm not trying to imply the agent may fool you into crossing so it can mow you down, the case of "AI incompetence" so to speak is more concerning to me then "AI malice".)
https://www.youtube.com/watch?v=1Za8BtLgKv8
Maybe a carton of eggs would achieve the same message? Don't want to do what those kids in Aurora did.
As someone who struggles with nonverbal communication sometimes, I frequently feel this way about all vehicles both as a pedestrian and also as a driver myself.
There are something like 230 million licensed drivers.
So if I was the non-existent "average driver" with no control over risk, it seems to me I should expect a fatal accident roughly once every 100 lifetimes.
Or every 100 million miles.
I don't perceive the state of the art in autonomous driving as anywhere near that kind of reliability.
However… driving is one of the most dangerous things we do. 1/100 people (roughly) die in a traffic related accident in their lives. That’s bonkers and unacceptably high.
Well, that's your opinion.
My estimate of my risk of death is 100/100, so 1/100 does not seem that high to me.
What I think could legitimately be called "bonkers and unacceptably high" is the prevalence of medical errors as a cause of death in the US.
"Analyzing medical death rate data over an eight-year period, Johns Hopkins patient safety experts have calculated that more than 250,000 deaths per year are due to medical error in the U.S."
https://hub.jhu.edu/2016/05/03/medical-errors-third-leading-...
Since I don't drive drunk, and usually miss rush hour, I do think it's reasonable for me to consider going to the doctor the most dangerous thing I do.
I spent most of my adult life flying bush planes around Alaska and working to try to make things safer in that (horrifically unsafe) industry. Trust me when I say 1/100 is unacceptable and absurd in the extreme and humans are terrible at assessing risk objectively. Just your statement: “I don’t drive drunk and I usually miss rush hour” is indicative of misunderstanding how this works. That’s ok! Everyone does that, I did too until I was trained on specific ways to think about risk and risk mitigation strategies.
The risk often doesn’t come down to individual action - it’s links in the chain, or the Swiss cheese model of you prefer and when all the factors line up an accident is more likely.
It’s great that you don’t drink and drive, it is also great that you avoid driving at peak hours. But do you actively do risk mitigation before driving? Probably not, more power to you if you do, but probably not.
i guess this is a long way to say "something you are exposed to every day multiple times" is way different than a (albeit terrible) system you are exposed to when you are old and infirm and already prone to death.
still, everyone thinks the 1/100 won't be them so they feel it isn't a problem… that is a poor way to figure out "what should we do."
You also add a whole bunch of other confounding factors if you're not comparing same-driver accident rates. Accident rates vary by location, city/highway driving mix, driver attributes (age, gender, credit score, etc), and probably many other things that may be correlated with Tesla ownership.
Compared to what? I find this very hard to believe. Surely forward collision avoidance systems have stopped a number accidents over the years. To say that there is no evidence for it, no data even, seems to suggest that you might have a data problem. Hard to take them seriously when they don’t have good data, yet maintain their very negative position.
If an autonomous driving system clearly prevented or would have prevented a crash that happened to a human, then that's one life saved, is the philosophy. The minimum benefit from everyone using the system is at least one life.
But some people do not see it that way. So you are probably at cross purposes with them. I'm sure the skeptics understand there is evidence for what you think proves mitigation - just not for what they think would prove it.
I haven't got what it takes right now to describe at length the alternative way of looking at things, plus I could have it (or phrase it) wrong. Maybe someone else will comment.
I am a huge proponent for good self driving but experiencing driving assist and being a passenger in Tesla cars has made me much more cautious of the current capability.
Then it has to be exactly in the middle of the road, even in a two lane one with another car coming towards you.