Between this and deliberately disabling Volvo's own emergency braking features, I think some decision-makers at Uber should be charged with negligent homicide.
Maybe a few executives going to jail will put an end to this cowboy attitude where lives are on the line.
IANAL, but given that the disabling of the security system was an act of commission, not omission, it could be argued that this was more than just "negligent". On the other hand, car manufacturers have no legal duty to include collision avoidance systems, so disabling one is probably not illegal in itself.
And now today (Nov 6) there are news articles about the Tesla "Smart Summon" which was quietly activated. There are videos of dangerous near collisions of "summoned" driver-less Tesla vehicles very slowly driving among other vehicles.
That’s very blatant sensationalist journalism that conveniently does not mention that there is a person holding down a button the entire time that the car is being summoned.
If that’s “dangerous”, well, so is piloting it manually.
A person who does always what is in front of the car and what is coming behind or on the sides, and who will probably be looking at the button rather than the car while it moves.
That is just as dangerous as using your smartphone while driving in a parking lot, which is forbidden for good reasons.
The video shows the app with a radar-ish view of what the car is seeing of its surroundings and its planned route. It's more than just a button, and I could see why someone would be focused on the screen display instead of the car.
Agree, a licensed human driver is in charge of a "summoned" vehicle. Any operation you might expect from such a vehicle is on the person holding the button.
So long as everything works perfectly, sure. But there's nobody in place to hit the brakes in an emergency scenario. Releasing a button as an emergency measure seems a bit unreliable, since humans tend to clench their muscles when there's an emergency.
Unless the car brakes the very instant the connection drops (and not when the timeout propagates through the networking stack), now you have two problems.
The linked video talks about updating laws regarding the legality of operating the smart summon feature on public roads. I don't see how that can be called "blatant sensationalism".
Someone inside the car has a better sense of the surroundings than someone operating it from a distance. They also don't have a skin in the game, which can impact the judgement
It is the same 'skin in the game' as being in the car. They are not concerned about their own safety if driving the car in a parking lot, they are concerned about their car getting into a collision. No different whether in the car or not.
That’s a red herring though. It shouldn’t matter if the person is paying attention if the self driving tech is working. (As Musk likes to say, the person is just there to tick off a legal checkbox.)
At what point do Musk’s statements represent a moderate legal risk?
For the sake of argument, could Musk’s statement, along with his very public insistence of being actively engaged in the design process, demonstrate a cavalier disregard for proper safety engineering at the management level (outside acceptable industry practice) and so a defective process?
It wasn't that the sensors were confused or something - the AI had no idea how to cope with pedestrians in the road for whatever reason, so threw them in the "Other" category.
At least 98% of objects are categorised, the rest don't matter. /s
TBH this does sound exactly like something you'd expect to happen if you took some Javascript ninja used to moving fast and breaking things and asked them to build a car. But surely Uber has more sense than that...
Coming from the defense (go ahead, clutch your pearls, I won't be offended) industry I just can't understand why people ship things that fail into in potentially dangerous states on some known non-trivial fraction of inputs. If the purpose of the software were entertainment or some other low stakes thing I'd understand but software controlling things that can kill people or lose lots of money need to be able to handle unexpected inputs gracefully.
> I just can't understand why people ship things that fail into in potentially dangerous states on some known non-trivial fraction of inputs
Because that kind of safety-conscious culture needs to be deliberately enforced, from the top all the way down, by a system that takes a long view.
If we're making an analogy to the defense industry, then building self-driving cars as a tech startup is closer to the Anarchist's Cookbook than Northrop Grumman.
Isn't the bigger problem that it did not classify the object as being on a collision path. It doesn't matter what the object is, brake if you are going to hit it.
From what I read in some of the earlier reports on this, the car didn't have the ability to emergency brake in autonomous mode. It was disabled at that time, so it could only brake in regular traffic, not for obstacles that appear suddenly.
The justification made for disabling the emergency braking - that it would interfere with data gathering - might appear reasonable at first sight, but it does not stand up to scrutiny, for if the emergency braking is triggered, the driving system has already made a mistake, and you already have the data on that malfunction.
> Yes, but accidental emergency breaking can cause just as bad of an accident as this, it all just depends on the scenario.
If Volvo's system is that dangerous, then it should not be on the road at all - but there is no evidence that it is, you are just making a speculative argument.
> They had hired a driver to sit behind the wheel to monitor the road and the car for exactly this reason.
That is no reason to disable a safety feature that would add safety in depth.
> The driver they hired decided to watch a movie on their phone instead of paying attention to the road.
That was a major error - a crime, in fact - but, unfortunately, also an entirely predictable scenario that cannot be dismissed on the grounds that dealing with it would make testing more difficult or expensive. So now we have three errors.
> It doesn't matter what the object is, brake if you are going to hit it.
The built-in emergency braking system from Volvo does this [0] but Uber deliberately disabled it (presumably because it conflicted with their self-driving rig).
Emergency brakes were disabled because they behaved erratically. So I'm not sure what it would have done if it had ID'd her correctly, it was still relying on witless behind the wheel. What a shit show.
I doubt that the emergency braking system was behaving erratically in itself, to any significant extent - if that were the case, it would do so for human drivers, and so should not be on the road at all. What I suspect is more likely is that Uber's system was behaving erratically, triggering the emergency response with some frequency.
Yea, should have put "behaving erratically" in quotes, I think you were probably on the money that it was in response to the AI or AD in this case. Probably was behaving like antilock brakes as the classification flipped every few milliseconds.
It's astonishing to me that emergency brakes were off. The investigation report clearly shows that the pedestrian was classified incorrectly, alternating multiple times between wrong classifications, and the path prediction was off. Despite all of those failures, it was still correctly determined that a crash was imminent 1.2 seconds before impact, and a second later realized that avoidance had failed - at some point around 1.2 seconds before the crash, AEB should have engaged.
Instead, as the report says:
> The vehicle was factory-equipped by Volvo with several advanced driver assistance systems (ADAS), including forward collision warning (FCW) system and automatic emergency braking (AEB) system. However, Volvo collision avoidance ADAS were not active at the time of the crash; the interaction of the Volvo ADAS and ATG ADS is further explored in section 1.9.
It was apparently disabled by Uber because it was difficult to run the Volvo systems alongside their own, which strikes me as highly irresponsible. Autonomous driving software relies on multiple redundancy, and AEB is the last-resort system for when everything else, from automatic maneuvers to alerting the driver, has already failed.
Consistently hitting the "other" category would be a massive improvement over what happened. According to the timeline for the pedestrian was initially classified as a vehicle, later as a bicycle, with short periods in between where she's classified as "other" or "unknown". And of course when an object changes classification it loses all history and is initially predicted to be static. Despite detecting her for 5.6 seconds the system never had more than 1.5 seconds of tracking history because it couldn't make up it's mind and kept throwing everything away.
Well I hope you are going to be that egg not me.
Cuz multi billion corp is too cheap to properly test their product and just deploy into production and see what happens and patch it later.
We can either have fast technological progress, or we can be completely tied up in bureaucratic red tape, with every potential innovation having to go through endless "oversight" committees staffed by non-technical airheads.
> We can either have fast technological progress, or we can be completely tied up in bureaucratic red tape, with every potential innovation having to go through endless "oversight" committees staffed by non-technical airheads.
That is a false-dichotomy. We can have something more in the middle, or at least less an extreme as "let cars just drive wherever without any tests and when it kills someone just say o well, that's progress".
In this particular case, Uber got a deal with the mayor (Governor? I can't remember) of the city to do testing, and it was clear the government bureaucrat allowed them to do whatever, and then coordinated with the police after Uber killed a pedestrian to fabricate evidence that looked like the woman came out of nowhere.
I would say that is really far from "endless oversight" and much more into the "just mow them down one at a time until you get it right" territory.
It's how we've arrived at where we are now, and why you have all the things you take for granted. Buildings, planes, borders, they all had a human expense.
A classic "tragedy of the commons" problem. Inarguably having only self-driving shared vehicles on the street will be massively positive to society (public space now reserved for parking will be public space, accidents due to drivers being tired/under the influence will not be a thing any more, resource waste for cars standing around 99.99% of their lifetime will not be a thing any more), but there will be a not-so-small amount of sacrifices on the way (be it people dying or being injured by prototypes, or massive job losses because the demand for cars will drop).
Air travel was horribly accident prone too, for decades, and now it is one of the safest forms of travel in developed countries - and for electric/autonomous travel, it will be the same.
Yeh agree with both of your comments but is this the way to justify the error that caused someone’s life ? Almost makes it sound like “oh well it happens”. Comments like these are showing why software like that was rushed to test against live human. What about showing some empathy?
All progress involves a certain amount of irreducible risk and I'm sure that any self driving car program, even if well run, will at some point end up killing someone by mistake - though hopefully after far more than the normal number of miles driven per fatality average.
But in this case it looks like Uber was really cutting corners that they shouldn't have cut and I think that Uber does deserve sanction for that.
"If the collision cannot be avoided with the application of the maximum allowed braking,
the system is designed to provide an auditory warning to the vehicle operator while
simultaneously initiating gradual vehicle slowdown. In such circumstance, ADS would not
apply the maximum braking to only mitigate the collision."
"Certain object classifications—
other—are not assigned goals. For such objects, their currently detected location is viewed as a
static location; unless that location is directly on the path of the automated vehicle, that object is
not considered as a possible obstacle. Additionally, pedestrians outside a vicinity of a crosswalk
are also not assigned an explicit goal. However, they may be predicted a trajectory based on the
observed velocities, when continually detected as a pedestrian."
Interesting, from the article and this, it sounds like the system can’t maintain position tracking of an object if it’s classification changes. So even if it could detect a pedestrian, something that could be ambiguous like a pedestrian pushing a bike might have no motion tracking data from one moment to the next, so the car would have no ability to predict its trajectory.
I had the same impression from reading the table.
Looking at the map and trajectories though, how could the human not see the car coming? Or was she thinking that the car would stop/slow down? Same question holds for the vehicle operator.
edit after reading the other reports: the victim apparently was under the influence of methamphetamine and the vehicle operator was busy watching Hulu.
As far as I recall the driver was looking at the car's status screen which they'd been instructed by Uber to monitor rather than at the road at the time of the crash, so didn't notice the pedestrian until too late. I can't say much about what the pedestrian saw, the Uber car killed them so we can't ask.
The pedestrian was about 3/4 of the way across the road when she was struck, and was walking a bicycle that was partially laden with goods. That suggests that quick evasion on the pedestrian's part would have been somewhat difficult, but given that the road was empty of other vehicles, there was a long clear sight distance to the pedestrian, and there was ample space to maneuver, any reasonable driver would have been able to stop or switch lanes to evade the pedestrian.
The driver was not paying attention to the road and was incapable of performing a timely emergency maneuver (be it a stop or lane change).
Yeah but when you cross the street you generally keep an eye out for traffic, and the sight-line is such that if the car had headlights on it should have been visible for more than 6 seconds before the collision. The victim might well have expected the car to yield somewhat.
However, I don't know if the car did have any external lights on, so it might have been hard to see until it was closer than 6 seconds away.
I think this is key. Had the system been able to retrieve tracking history as the classification was changing, it would have realized that she was moving perpendicular to the vehicle's path (not, for example, an object traveling alongside the vehicle).
I found the "action suppression" system very interesting, and I think that is new information. Essentially, for 1 whole second after an emergency situation is detected, the system does nothing at all by design. It even suppresses planned breaking maneuvers. If the operator doesn't take over control after one second, the car will slow down slowly, not even using the full allowed braking power for regular driving (not to speak of actual emergency braking, which the system wasn't allowed to do at all).
To me it indicates that at the time of the crash the software was still in a state of development where false positives massively outnumbered situations where action was warranted.
The insane and arguably negligent part of this is that in a situation where false positives result in braking and false negatives result in death, that as an engineer you don't go with braking
With a disabled emergency braking system and an inability to handle a rather common situations like people jaywalking, these cars really shouldn't have been driving in public. That is the kind of stuff that I'd expect to be tested and implemented on private grounds.
Additionally, reducing the number of people in the car to one when the car is pretty much by design not capable of handling emergency situations by itself is quite reckless.
They shouldn't have been driving in public with someone who wasn't paying attention 100% of the time as any regular driver would. The longer quote:
> Also, don't forget: the SUV's emergency braking system was deliberately disabled because when it was switched on, the vehicle would act erratically, according to Uber. The software biz previously said “the vehicle operator is relied on to intervene and take action," in an emergency.
The question then was there proper training and communication to the test drivers that it's never okay to look down at your phone. Or whether that was simply unrealistic expectations. Or if the hours were too long, or testing at night, etc.
It said there was 5 seconds which should have been more than enough for a human test driver to hit the brakes, which was their stated job.
In this case the safety drive was watching their phone which was playing a movie so a bit beyond the normal level of attention wandering you'd expect. But even for people trying their best having to pay attention to a road for hours without having to provide any input isn't something you can reasonably expect people to do. NHTSA level 3 autonomy is just a bad idea, we need to go straight from 2 to 4.
> They shouldn't have been driving in public with someone who wasn't paying attention 100% of the time as any regular driver would.
It's pretty well known that humans' minds wander, that it happens more when monitoring reliable systems for rare problems, and that it makes operators less responsive and lowers their error detection rate [1] - as anyone who's attended a boring meeting or lecture can attest!
I'm not sure that anyone informed would imagine a worker spending 40 hours a week monitoring a self-driving car would be able to watch it with 100% attention.
The truth is nobody realistic expects the safety driver to respond to reliably prevent an accident like this - they're there for slower-developing problems, resetting false alarm stops, and taking the blame.
That's why two tests drivers makes sense in the early days and not keeping extended hours.
Companies like https://comma.ai/ are taking a much better approach IMO by keeping it simple by first perfecting lane assist/highway driving + building a driver watching device which alerts them when they stop paying attention for x amount of time. Which Uber should be investing in for their test drivers.
Another important thing is being realistic about expectations, of course 100% paying attention is unrealistic even for normal drivers, accidents will happen regardless. Hitting jaywalkers on a dark multi-lane high speed road is a lot less bad than other possible scenarios and there really hasn't been that many accidents yet.
Additionally, permitting a test plan that prominently involved requiring all operators to break the law by operating a computer when they were supposed to be keeping their eyes on the road is quite reckless on the part of the State of Arizona.
I really don't think we should be so eager to damn Uber that we forget that there were more than just Uber employees asleep at the wheel here: Arizona has a duty to protect public safety. By permitting a self-driving car test program on the public roadways without doing even basic due diligence in vetting the program first, the State was grossly negligent in that duty.
> there were more than just Uber employees asleep at the wheel here
You couldn't be more right. The zeitgeist in 2015 was full-steam ahead on the autonomous future and woe to anything that stood in its way.
The framing of Arizona's embrace of autonomous testing four years ago [0,1] contrasted with California's caution [2,3,4] couldn't have been starker. One was branded enabling innovation, the other was seen as bureaucratic red tape holding up progress.
It's too easy to get wrapped up in these narratives.
This makes me wonder how Uber got clearance in the first place. Did a regulatory body, like the NTSB, need to sign off on these self-driving cars before they could be driven on public roadways? If so, how did they miss this during that certification process? Surely emergency situations are at least discussed, tested, and reviewed?
So uber fails to predict even completely linear paths traveled by objects if the system isn't trained on or expecting that specific type of object? That sounds like an even bigger issue than "we didn't think pedestrians could exist outside designated crossings"
What's crazy to me is that an "Other" object isn't flagged for a complete stop or a major alert to the "safety" driver. This seems like a major case for when the software should stop being autonomous, when it isn't sure.
Likely the story is that their software flags a lot of things "Other" without figuring it out upon closer inspection.
For those who think "well, pedestrians don't belong on the road so this wasn't much of a bug" (because I'm sure there are some of those): just imagine the "other" was a large chunk of debris, or cargo fallen off a truck. The car would have crashed right into it.
There can be many obstacles on the road, stationary or non-stationary.
AI can get away with requiring humans to take control in emergency/unusual situations, but not when it is supposed to be "self-driving". The software at least needs to recognise problematic situations and request a safety override or come to a safe stop, it can't just shrug it off and move on.
Debris tends to be stationary most of the time. Sounds like the car could have handled that.
I agree with that ADS being shit. Objects, once identified, should be accounted for in later updates in some form. E.g. in the way of "pedestrian now behind truck next to me". How the hell can an object switch status back and forth, loosing all its history? This should make the system be extra cautious at the very least. But yeah, safety doesn't seem to have been that big of a priority.
Most of the time, yes. But if you've ever been in a mountainous region, you'll know that it won't be stationary all the time and that it's most dangerous when it's not.
There was an emergency braking feature that would engage when a collision was imminent. It fired 1.3 seconds before hitting Herzberg. But Uber had disabled that function because there were too many false positives, so the car just alerted the driver (who was not paying attention) instead. https://www.washingtonpost.com/news/dr-gridlock/wp/2018/05/2...
1.3 seconds is definitely not enough for a human to engage brain. I agree the human should have been paying attention and might have spotted the problem sooner, but that alert is completely useless.
Alerting a human (who wasn't even in the driver's feedback loop at the time) 1.3 seconds before the collision? That is equal to "hey, look at this unavoidable crash that we're driving into!"
It is ridiculous to expect that the driver can be alerted and react in any kind of reasonable time frame unless they were already paying attention, in which case what use is a self driving vehicle?
They are supposed to be paying attention in these self-driving car tests. If the driver was negligent then they should be held at least partially accountable.
FWIW, we now know Uber's technology didn't do this, but we don't necessarily know the nitty-gritty details of how other self-driving systems work.
For me, the most important take-away isn't something about Uber being sloppy. It's that the distinction between "machine learning" and "artificial intelligence" really matters, and the efforts of marketing teams to over-hype ML by selling it as AI may be hindering our society's ability to make cogent public policy decisions around these technologies.
(No, I'm not really interested in haggling over ideas like "hard AI" and "soft AI" here, either. Normal people don't split those hairs. Nerds generally don't, either, except when they're trying to score easy points in a debate.)
The machine learning part could have gone better: it didn't expect a pedestrian and jumped between vehicle, other, bicylce and unknown. But that would have been good enough, the failure is in the code around that. When your classification is unstable or low confidence you can't trust it, which in a safety critical system would mean falling back to something else. That could be using a different classifier (e.g. linear interpolation of the object's position to find out if we come anywhere close to it) or just giving up and stopping the vehicle. Just continuing as if the fluctuating results were reliable wouldn't be acceptable in any industry.
What kind of tech do you expect them to use? While I hope it wasn't, I'd not be surprised with them just running a classifier over the current scan, have it spit out "vehicle" at coordinate A, search the previous scan's analysis for a matching vehicle close by. How do you handle uncertain/flip classifications? Or objects appearing to suddenly "spawn"/"despawn"? A proper "what object was that in the past" would likely require its own AI. Reliable linear interpolation doesn't seem that simple to me.
From what I remember, the pedestrian was detected early enough but classification jumped between two or three classes and it never got a prediction vector, basically being a standstill object (at different locations) that was never predicted to cross the trajectory of the car.
If the safety driver had been paying attention to the road instead of his phone, they might have been able to break in time so that this woman did not get hit.
Human minds don't work that way. Paying continuous attention to a vehicle you're not controlling is quite simply impossible, and if safety relies on it then your safety system is broken.
Human minds don't work that way. Paying continuous attention to an airplane you're not controlling is quite simply impossible, and if the function of a copilot relies on it then airlines are broken.
People are tasked with monitoring things all the time. It's not some brand new impossible task.
There's a critical difference between airplanes and cars. When flying an airplane, very few emergencies ever require action faster than "Couple of seconds" -- and any procedure which does require such will be thrown out at first opportunity.
Possible, but should not be relied upon. Even in cases where the driver knows that a handoff to manual control will occur, it takes a significant amount of time to mentally switch from being a passenger to being a driver. For cases where the AI knows it isn't performing well, such as heavy rain or snow, it makes sense to handoff. For anything that happens on shorter than a 10 second timeframe, the AI must be able to handle it, because the human cannot switch modes that quickly.
You can also see the detailed report from the NTSB which will go through about everything you would want to know about the actual driver and her reaction:
* Uber had previously staffed each car with two operators, allowing one to keep their eyes on the road while the other took notes.
* Forthcoming performance metrics for operators would include "the VOs ability to keep the SDV in autonomous mode as long as possible unless there is a Fleet Desk support issue (software issue)."
Obviously not as relevant to this particular case as the Hulu streaming, but to me these details suggest safety isn't their highest priority.
One of the most terrifying lines in those documents to me is this:
> According to Uber ATG, the SDS did not have the capability to classify an object as a pedestrian unless that object was near a crosswalk.
Jaywalking is extremely common. I've seen pedestrians jaywalk a 45mph 6-lane road, with no median to speak of. Anyone with driving experience should know that pedestrians can occur even with no crosswalk, so it boggles my mind that any engineer would sign off on such a decision.
As a UK-resident, I find the omission of considering a pedestrian in the road quite unexpectedly, and unfortunately, overly car-centric.
There isn’t really the term “jaywalking” here. It’s just “crossing the road”. I’m not sure on exactly who has what legal responsibility, but it certainly feels like pedestrians should look out for cars when crossing, and drivers should look out for pedestrians.
I posted about Harry Dunn who died while "riding his motorcycle when a woman emerged from the airbase on the wrong side of the road and there was a head-on collision." I mistakenly recalled that case as involving a pedestrian.
Interesting segue: in The Netherlands, that would not legally be a pedestrian, although you would be classified as one for most purposes.
A pedestrian ("voetganger") is someone who travels by foot only. If you guide another vehicle by hand, or even a horse or cattle, you are legally considered a driver. Therefore, most laws concerning pedestrians have to make explicit allowances for walking a personal vehicle. For example, the primary Dutch Traffic Regulations [1]:
The rules in this resolution concerning pedestrians apply equally to persons guiding by hand a motorbike, moped or bicycle
When I trained for my driver's license in Germany, the scenario "people may run onto the street" was one of the most frequently recurring scenarios in the theory part.
Anything from "children run onto the street to chase after a ball they were playing with" to "pedestrian jumping out between cars because they didn't see you" or "tram stops in the middle of the road and people exit onto the street".
Legally pedestrians must use marked crossings when available and are told to check both ways, but if a driver hits a pedestrian, it's always the driver's fault (though the pedestrian might share some blame in some circumstances but never the full liability).
> Legally pedestrians must use marked crossings when available and are told to check both ways, but if a driver hits a pedestrian, it's always the driver's fault (though the pedestrian might share some blame in some circumstances but never the full liability).
It's theoretically the same in most US jurisdictions. However, it's not what the law says that matters; it's what the police will actually do, and in practice, there is a very strong tendency to blame the pedestrian for any accident.
I've seen that happen (on a 4-lane hwy): a drunk who apparently tried to jump under a bus, but reacted so late he nearly missed it; as it were, he stumbled into the side (and rebounded to the wayside). Quite improbable, yet there we were.
> Anything from "children run onto the street to chase after a ball they were playing with" to "pedestrian jumping out between cars because they didn't see you" or "tram stops in the middle of the road and people exit onto the street".
What about "drug impaired people going onto the autobahn at night wearing dark clothes in a section unilluminated by street lights" which is what I believe happened here?
It's near impossible to enter autobahn without a car or other motorized vehicle. I am not sure what kind of road the victim was following but it surely wouldn't compare to autobahn. Was it a highway with free entrance for bicycles and scooters?
I find extremely hard to believe it wasn't top of mind for the Uber engineers. I think it's more likely they just fucked it up or deliberately ignored it and are letting the NTSB think it was negligence because it's a less harmful outcome for them.
> but it certainly feels like pedestrians should look out for cars when crossing, and drivers should look out for pedestrians.
The problem in this case was that the pedestrian was impaired by drugs, if I recall correctly.
In my opinion, this happened because of 3 simultaneous failures: failure of the pedestrian to responsibly cross street, failure of the computer system to recognize the obstacle, failure of the computer babysitter to be on the lookout for obstacles and override the computer.
Hopefully we learn something from this triple failure and take measures to reduce their likelihood in the future.
According to the preliminary report of the collision released by the NTSB, Herzberg had tested positive for methamphetamine and marijuana in a toxicology test carried out after the collision. Residual toxicology itself does not establish if or when she was under their influence, and hence an actual factor.
Soooo...it's now okay to run over people (5 seconds is a LOOOOOOOOOOOOOOON-crash! time), as long as you have an excuse? Go watch the scapecam footage: even with the horrible picture quality, you can see the victim walking (in a straight line across the road, no less!) a very long time before impact.
Perhaps you could also argue "if she were wearing a helmet, it would have conferred +1 to CON and perhaps saved her", for what it's worth - makes sense in the crudest of simulations, but not in RL.
Of course not! It's never okay to kill people. But she is at least partially to blame for her own death. If I take a nap on some train tracks and get run over, is my death 100% on the train engineer for not stopping even though I was in plain sight for 30+ seconds?
You are taking this into ridiculous extremes. She wasn't taking a nap in the road, and a car is stoppable at a far smaller distance than a train. Your argument boils down to "it's her own fault for existing outside a car, shit happens."
When I jaywalk (or cross anywhere) I consider it my responsibility to cross safely. If I got hit by a car, it is 100% my fault for not looking. Maybe it is also 100% the driver's fault. Responsibility and blame is not something that has to get divided up such that it adds to 100%.
In this case the pedestrian and driver are both fully responsible. The pedestrian was lollygagging across the road like a complete idiot. The driver wasn't paying attention.
I'm not saying responsibility + blame > 100%, if that's what you think I said. Read it as two statements: "Responsibility is not something...," and "Blame is not something..."
If doesn't clear things up, I don't know how it could be incoherent.
In the UK pedestrians have the "right of way" if they were crossing the road and a car comes up on them on all roads except on Motorways and slip roads where they are banned. I put "right of way" in quotes because there's no set law - but almost all the time because a car will hurt a pedestrian - the law will side with the pedestrian. In terms of law, the written one says if a walker is crossing a side road that the driver wants to enter into, then the driver must give way.
One example would be if you were leading a walking group of 20 people and wanted to cross the road and it was safe to do so at the beginning but because the group is large it takes a long time. Any approaching cars would not have any right of way until all the group has passed.
A right of way means that they have priority. In the UK pedestrians only have a right of way in specific cases.
But drivers have a duty of care, and if a pedestrian is crossing a road then obviously they must do what is safe, i.e. slow down and let the pedestrian cross if (s)he is in the way.
At least some US states don't have a legal concept of "right of way", specifically to avoid the tacit implication that the person who has right of way is under no responsibility to avoid a collision if someone in their path doesn't.
e.g., where I live, the traffic laws are defined in terms of who has the responsibility to yield, and undergirded by a catch-all requirement that everybody who is able to act to avoid a collision must do so.
It's perfectly fine to give pedestrians the right of way, because assuming they also have some sense of self preservation isn't really so far fetched.
For instance in Germany they drill into you in driver's ed that when you hit someone with your car, it's pretty much always your fault. I'd say we don't have a jaywalking epidemic.
It's reasonable to give priority to pedestrians once they are on the road because it makes sense to allow them to leave the road as quickly as possible instead of being stuck there because cars don't let them, which is both very dangerous and bad for traffic.
It's equally reasonable for pedestrians NOT to have priority to start crossing the road unless there is a clear, marked pedestrian crossing.
Sure, you may get a ticket for not using a pedestrian crossing if there is one nearby, but if you get run over it's still probably the driver's fault. You can have both.
There is a difference between having priority and having the right to be safe.
Priority means that cars must give way, which is the case at pedestrian crossings: Cars must give way BEFORE a pedestrian has started to cross in order to let him cross.
But in any case once a pedestrian is on the road cars have a duty of care not to hit him.
The reason some US states choose to specify who should yield instead of who gets to go first is because framing things in those terms hopefully alters the ways that drivers think about things in a way that will improve safety overall.
TBH, I suspect that it's an empty gesture that is being used as an alternative to implementing more rigorous driver education and licensure requirements; the US tends to view driving as more of a right than a privilege.
Legally the pedestrian has no responsibility except that they're prohibited from entering certain areas specifically legislatively set aside for motor vehicles like motorways (approximately "freeways").
Drivers are required to give "due care and attention" to driving which can be demonstrated by following the "highway code" and that code tells them pedestrians might do things they don't expect and to assume if it's not clear what's going to happen then yeah there is a pedestrian behind that obstruction and they are going to run into the road in front of you, whereupon hitting them would be your fault.
For example when I was a child I got off my first bus home from secondary school, and ran straight into the road in front of a car I couldn't see because the bus was in the way. The horrified driver was legally responsible for that, even though she hasn't intended to hit me. I believe she would have been automatically billed by the authorities for the cost of shipping me to a hospital to have my broken leg set and so on.
Clearly it is in some sense my fault that happened, but on the other hand it's not me choosing to drive a huge steel box at 30mph past a bunch of idiot children...
Driving without due care and attention is also an offense in the United States and Canada, and should apply in almost all cases where a pedestrian is hit, but usually isn't.
I presume that's the case in every U.S. state. Part of the bus stop protocol I got taught is that kids cross the street in front of the bus after they get out, and that we needed to walk forward far enough that the driver could see us.
My school bus had this pole attached to the front bumper area of the bus. It swung out when it stops. This forces kids to walk at least 8 feet (or however wide a bus us) in front of it before crossing.
Yep, they appeared in my school district in the mid-90's. And with it came stop signs that swung out from the side of the bus, overhead escape hatches that could be opened for ventilation, and ejection seats.
In Minnesota, the school buses have stop signs that swing out with lights on them when the bus is boarding or letting off children. It is illegal to go around the bus when the sign is displayed. I hope I never find out how much the ticket is.
Yup, we have those in Texas too. I assumed they were standard; I guess TIL I learned they're not. They very much should be. Children, especially very young children, are quite unpredictable even when they should know better, and the signs are a signal from the bus driver to surrounding cars that the safety of those children (which the driver should be monitoring) is much, much more important than the impatience of the surrounding motorists. You always stop. Always.
Yep, same in WA. In fact, every school bus deploys bright red flashing lights and displays giant stop signals unfolding from the back, with signs that say in very uncertain terms that it is illegal to pass a school bus when it is stopped and the lights are deployed.
Ridiculous those big road hogs get to sit there impeding traffic for literally no reason. Just adjust the route so the kid only gets out on the side facing the house so the rest of us don't have to suffer. Pay taxes for the road then have to sit there not even able to properly use it, ludicrous.
Why didn't your school bus have a stop sign deployed? Those are for exactly this risk. Running a school bus stop sign I believe is a more significant violation than running post-stop signs.
Soon Asia none of those are public school buses, especially China where they are all private buses painted yellow and have no special treatment (no stop sign popping out of the side). Similar differences exist in Europe. Or I guess my point is, none of these systems come even close to resembling what we ache in the USA and Canada.
In London, children take the same double decker buses everyone else takes. There’s no “school bus” like they do in the US. This is also true for Germany and the Netherlands where I’ve lived as well.
I'm glad you weren't hurt worse; an unfortunate college student at my university tried the same thing with a public transit bus stopped to let her off at a green light, and she did not survive.
The car passing the bus had no chance of seeing her, and there was no reason to believe anyone would be crossing against the green light. I think she may have had a pattern-match malfunction and acted as if she'd just disembarked a school bus. It was a tragedy all around.
I was crossing behind the bus. That's why she couldn't see me. Also, this sub-thread is about the UK, and so unsurprisingly my anecdote is also about the UK.
back by the massive diesel exhaust pipe? no thanks. you’re really setting kids up to be healthy! the front the driver can see if the child is far enough away (hence the pole) and it keeps the kid in sight for any other stopped cars next to the bus. Also I hold you understand running around saying things like “it just never ceases to amaze me how car-centric the US is” is offensive and micro aggressive. we have what we have, and that’s a lot of room.
It's not car-centric, it's just extremely bad (though perhaps just incomplete...) design for a car that is tested on public roads.
Anything can get in the way of a moving car. Pedestrians are unpredictable and should be kept an eye on at all times.
In the UK it is legal for pedestrians to cross roads anywhere unless specifically forbidden (that's basically motorways). In general accidents it will be deemed the driver's fault (if you see a pedestrian you're supposed to slow down and avoid, not honk and continue because you have a duty to drive with due care and consideration) unless the pedestrian acted in such a way that the car could not avoid the accident.
All sorts of hazards can come up in the road, such as children running into the street, deer or other animals.
This scenario was bound to happen given that auto-breaking was disabled and the human driver was playing with their phone instead of paying attention to the road, which was the whole reason they were in the car.
Agreed; my thanks to whomever removed the politicizing term "jaywalker" from the submission title, but more importantly shame on the NTSB for using it in this report. Whether or not she was breaking a street-crossing law is entirely irrelevant to the drivers obligation of due diligence.
Rhetoric spewing forthcoming... I talk about this stuff with my peers, and am constantly dismissed by the techno-optimists that believe in the promises of big tech, but I think it is important not to jump the gun and put these self-driving cars into production (i.e. the real world) without a significant paper trail and analysis of real-time incidents being made available to the public. Full stop, all incidents and outcomes, not just the good.
Going further, I wish for more initiative to re-design infrastructure putting the safety of pedestrians, cyclists etc at the forefront. It is simply not safe for alternative transportation to share the road with vehicles, autonomous or otherwise. Coincidentally, if infrastructure were designed to limit or prevent the interaction between cars and non-cars, the problem of detection for autonomous systems becomes much easier. Unfortunately, society would rather prop up the free market with capital than invest in public works which benefit more than corporations and shareholders.
Good points, but we also need to quantify the risks posed by human drivers in similar situations. Getting "similar situations" is hard, but not impossible: aircraft industry builds faithful sims where people can respond to various stimuli (or lack thereof for hours after which trouble comes).
As it is, we bash every crash and failure of such cars and may prohibit those from public roads while statistically they might be making our lives safer, not more dangerous. My 2c.
I see what you're getting at, but there is a certain amount of cognitive dissonance at play here. If a person driving a car strikes a pedestrian, there may be a grey area as to who was at fault. What was the speed of the driver? Was the driver impaired? Was the pedestrian impaired? Was the driver's view obstructed, was the pedestrians view obstructed? Etc, etc.
But when a system that is designed to autonomously detect infinite, non-discrete, random variables fails prevent a collision/incident, you have a failure. There is no grey area, Y or N. The hope is, the system is smart enough to determine exactly 'why' there was a failure (i.e. the system recognizes the input, a pedestrian in this case, but was not able to prevent collision due to the constraints of physics, or without a nominal increase for concurrent accidents). Still a failure, albeit a failure with a greater snapshot of heuristics and causality.
Citation needed. This is my argument. Unless you are working for Uber/Tesla/Waymo where is the data? I'd even be satisfied with seeing incidents where the system successfully avoided collisions, at least as a baseline. How do regression lines trend in comparison to non-autonomous cars considering scale?
I'm pretty sure if they were better than humans by any real statistical margin they would be trumpeting that in press releases. The fact that they work so hard to maintain such secrecy- only revealing data that they are legally mandated to- suggests that they, the people must familiar with the system performance, think it is inadequate.
In essence, because the people who control the data have an interest in revealing data that is superior to human performance, the lack of conclusive data on this leads to an inference that it isn't.
Uber's self-driving car fatality rate: 1 death per 2 million miles[2]
That is, Uber's state-of-the-art is an order of magnitude worse than human drivers.
I know, you'll complain about small sample size. The thing is, assuming Uber's cars match humans, the chances of a collision happening only after 2 million miles are very small, like pulling an ace of spades out of a shuffled deck of cards.
The point is, we do have the data, and the data does not support optimistic beliefs.
I've never understood techno-optimists on this point.
If Uber has such amazing technology that avoids accidents, they could deploy them as accident-avoidance for regular drivers. They could give normal driving control to the humans, and take over only if there is a potential of accident.
If you argue that these driver assistance technologies already exist, then you better not be comparing your magical AI on supersafe cars with sunny climate on well marked highways to the average driver on the average vehicle under average climate on average roads.
That is a good point, especially when considering that modern cars have semi-autonomous controls which are likely to actually be effective in preventing ‘human’ failures, while still maintaining the UX of a non-autonomous vehicle.
Another point to consider, is that the actual experience of autonomous driving may increase the likelihood for ‘manual intervention’ failures due to the system being able to react correctly most of the time!
> without a significant paper trail and analysis of real-time incidents being made available to the public. Full stop, all incidents and outcomes, not just the good
Without simultaneously reporting the identical data for human driven vehicles in a complete, accurate, and globally comprehensive way there is little that can be done with such data except use it to make political attacks on the technology.
As comparison: every single battery electric vehicle that has a fire is widely reported and disclaimed, impugning the technology as unsafe. The 10s of thousands of flammable fueled vehicle fires are completely unreported in the media.
A brief look at https://www.nhtsa.gov/ reveals that more data is available than I thought, along with a roadmap to full automation (https://www.nhtsa.gov/technology-innovation/automated-vehicl...). I'm glad this is being looked at, and it seems that they are in favor of automation. That being said, the data is there for comparison/analysis.
One major benefit of autonomous vehicles is that they are certainly better reporters than humans. They need not rely on the fallible memories of human drivers when data from sensors, cameras, internal system monitoring etc is out there living in the data-center. A complete snapshot of an event for replay removes any uncertainty from the equation. Simply put, these systems are designed to provide optimal data that should be captured for analysis.
I am not a Luddite here, I think that it is possible for machines to be more effective drivers than humans. As a consumer/pedestrian however, I am skeptical of any claims without actionable (open) data being provided.
Yes, exactly! I'm Australian, it's perfectly normal here to walk on the road when there aren't footpaths, ideally walking into the direction of traffic flow so that you can see oncoming cars.
Pedestrian crossings aren't provided on most roads outside of shopping and business areas - you simply have to cross at a safe point and try to be visible.
I really don't know about the US but I can't imagine it's that different: surely people need to be able to cross the road without being run down as an unclassified object.
The US has a very hostile view towards pedestrians, in large part due to PR campaigns by car companies. The term jaywalking was invented to slander pedestrians not crossing “correctly” (jay is a synonym for rube).
I feel like if someone's out walking around they're probably just doing it for fun or exercise and don't have anywhere important to be. People in cars are more likely to be on a schedule. One of the reasons I think it makes more sense to favor cars in most situations.
I have walked around outside in order to get places almost every day since I moved out of Arizona years ago. Even in the car-centric US, huge numbers of people live in places that aren’t quite so car-centric as Phoenix.
No, the view is that Uber (and city officials, and some of the press) are spinning hard to shift the blame to the victim of this manslaughter, by using words such as jaywalker.
FYI, this is an American thing[1]. There's no jaywalker in Russian, for instance. Or French. If you know any other language that has a word for that, let me know.
It didn't even spread to the UK:
"In other countries such as the United Kingdom, the word is not generally used and there are no laws limiting how pedestrians can use public highways."
In reasonably dense U.S. cities, people walk into the road outside of crosswalks all the time. Even in suburban areas, it's normal to cross like that in residential areas. But suburban areas are often crisscrossed by 30-50 mph streets where crossing the street outside of crosswalks (or sadly even in crosswalks, really) is dangerous. This is partly because of the speed limit. But also partly because there are few pedestrians (points of interest are far apart so people need to drive everywhere), so drivers are relatively reckless.
I'm not convinced that the pedestrian outside crosswalk omission arose because of U.S. car-centric thinking. I think if any software engineer working on the project was told that their autonomous driving system wasn't watching out for these pedestrians, they'd be horrified. In fact, probably many of them were already horrified at how poorly their system was performing in general, and yet management still deployed the software in actual physical cars.
Perhaps they should be the ones who get to decide. In many safety-critical fields they do. That's the point of requiring an engineer's stamp of approval on a design.
The emergency braking system was likelt turned off because it had triggered too many false positives. (Abruptly braking too often also makes you a dangerous driver.)
So out of complete necessity, these cars need to be programmed to actually ignore pedestrians unlikely to cross into the path of the vehicle.
This is a tough problem because the AI has to identify the human (seems as if it did in this case) AND the intent. I've seen two recent videos of Teslas abruptly stopping for humans on the side of the road. One was actually a bus stop poster model and the other clearly had no intent of crossing the street and angry motioned the Tesla on.
Imagine driving by a group of joggers on the side of the road. Who can you pass safely? Who many stumble a bit into your path? Who will prepare to cross but first wait for you to pass by? Who will try to cross in front of your path? The micro-decisions and predictions made are very challenging to get right.
This is where Waymo uses deep learning models that can predict the future behavior of other road users in real-time. They are hiring: https://waymo.com/joinus/1235933/
This is par for the course in AVs. It was probably the case for Uber too: hardcoding an "action suppression" heuristic on top of it is horrifically negligent IMO (I work on ML systems for an AV company)
Well surely not, if its a bird and you'd cause an accident with an emergency stop then you should continue. Dogmatic responses are exactly the problem with self-driving vehicles.
Sometimes it's preferable to hit something in your own vehicle. Cue the classic moral question of which person would you kill if you can only avoid one - with your vehicle - by hitting another.
Generally we should probably have vehicles follow proscriptions of the law when they can't avoid hitting anything and change the law if we want them to act differently. I've never seen a hypothetical self-driving car trolley problem where there wasn't a single option that was clearly what the law required.
By and large, the trolley problem concept is overblown. Most of the time the right/best answer is going to be to stand on the brakes and hope for the best.
But I'm honestly not sure what the law "requires" if you've got a scenario where there are going to be bad outcomes no matter what you do.
For instance if a car has the choice of hitting someone in the road or swerving onto the sidewalk and hitting someone there then clearly the legal thing to do is for the car to stay in its right of way and hit the person in the road.
That's a pretty clear case of taking a deliberate action to leave the road surface. But you can at least imagine scenarios where everyone is within the bounds of the road--say 5 people directly ahead and 1 off to the side.
As I say though, if you can't swerve to avoid people, the most reasonable action that most people would take--to the degree they had time to make a conscious decision at all--would be to brake as hard as they could and let things play out as they will.
Well yeah sure, but a bird is quite different to a person, it's small and usually not on the floor. I just meant it seemed strange to detect an object but not do anything about it because it's not labelled.
Well, if a large bird - say - an emu or ostrich (but probably also a turkey) suddenly crosses your car's path I believe you would brake.
At least here (in the country) it is not uncommon that wild boars or deers cross the road suddenly at night, and I have seen cars literally destroyed by the collision, in some cases with the driver seriously injured.
I think part of it comes down to reliability of the sensors - if you detect an object 200m down the road, 5 metres to the right and you have only a small error in the reading, you could easily get the impression that it's moving into the path of the car. However, if you can identify what it is then you can identify if it's likely that it really is going to move into your path. These sensors really aren't as perfect as you would hope which is why so much effort is being put into machine intelligence.
> If you think Uber’s car is bad can you imagine how bad Lyft’s is or most any company that started years behind?
The amount of time someone spent on a software project isn't the best indicator of its quality... especially for science work where there's a large amount of knowledge transfer within the field.
Lyft hired a top Google engineer to run the project and plenty of other experienced people. They weren't starting from scratch like Waymo and nor did Uber.
The car simply shouldn't have been tested without a driver (or two) constantly paying attention. Clearly Uber didn't trust it to be running by itself yet (especially with the emergency brakes being disabled) but it basically was if the test driver wasn't paying attention.
Maybe works for Uber and GM, but in Europe jaywalking doesn't exist. If they want to sell cars here they will have to deal with being responsible for almost every pedestrian they hit, because that's the standard we already set for human drivers.
But it is a somewhat homogeneous region with similar laws, as supported by that Wikipedia article: in general you can cross roads anywhere unless it's a motorway or you are within 100 meters of a designated crossing (any closer and the laws diverge on what's acceptable). That's just how we talk about the US treating things a certain way despite the different states having varying laws with countless exceptions and special cases.
According to wikipedia your nearby is 110 meters, so slightly out of the range for most European countries. She most likely used a set of crossing paths on the area between both halves of Mill Ave so she started even further south of it.
> your nearby is 110 meters, so slightly out of the range for most European countries.
« Every » rather than « most », 100m seems to be the Eastern European limit and going down the list in western or northern Europe the limits seem far lower, usually 20 to 50. Furthermore pedestrians not following this often does not absolve drivers from any (let alone all) responsibility.
Along with attempting to shift blame to the cloud, there are always suggestions which try to mitigate their inability to deal with sentient things with more rules.
In NYC and LA, pedestrians have the right of way, no matter where they are in the road. The pedestrian might be cited for jaywalking, but the motorist is liable for any injury to a pedestrian.
Not necessarily the programmers. There'll always be some people who are easily enough to manipulate to write this kind of code. But those making the decision to ship this and sign off on it definitely should face criminal negligence charges.
No, this is a video from a shitty dash cam released by Uber (and then happily compressed to shit and forwarded by police, who don't miss a chance to blame whoever died in a crash) to deliberately misrepresent the situation. This garbage pinhole sensor has not one hundredth of the dynamic range of the actual cameras they use for the autonomous driving and is not at all representative of how human eyes see, which would have had no problem spotting her from many many seconds away.
Here's a simple rule that could be enacted for companies seeking to do work in the autonomous driving space: if your car kills a person, your company loses its license to work on autonomous vehicles. Forever. That will dictate the adequate pace to achieve these goals safely. Then we'll see what's really possible with this technology.
Having cars that can drive themselves just doesn't seem like a particularly high priority for society at large in the face of other looming issues. Why allow it to proceed in such a dangerous fashion at all?
What if the rate of deaths for a given company is not zero, but below the rate for human drivers? Is it ok to have additional, unnecessary pedestrian deaths by NOT allowing that company to deploy their technology?
(That said, the negligence in the Uber case makes it pretty clear they are likely far from reaching that level of competency)
So you mean being okay with a multibillion dollar corporation harming people so it can make even more profits, all the while telling you that it's "good for society"
Good for the people once the system is perfected. Not so much for the ones who get run while the system is being perfected.
The "greater good" utilitarian argument has been the basis of some of the worst policies and politics in the world.
I'm not saying that self driving cars fall into the same category, but how many deaths are you okay with until Uber/Waymo perfect their algorithms (and later, charge you for it)? 1? 10? 100?
But human-driven cars kill tens of thousands of people a year in the US. I think improving that is really important for our society.
It is very sad that this pedestrian died and companies that kill people through avoidable accidents like this should be punished. Uber specifically seems like a trash tier self-driving car research program and I wouldn’t mind if they just stopped.
But one day, even a system that’s far safer than human drivers is going to kill someone accidentally. It is going too far to say, a method of driving cars that ever kills somebody should be abandoned.
I'm pretty skeptical we'll see autonomous vehicles--at least outside of limited access highways or other relatively less difficult scenarios--sooner than decades from now. But your suggestion is an impossibly high bar. There will always be failures because of debris on roads, unpredictable actions by human drivers/cyclists/pedestrians, weather problems (e.g. black ice), mechanical failure, etc. that will result in some level of fatalities.
Given how many people die per year in traffic accidents with non-autonomous vehicles, shouldn't this rule apply too? Humans have shown that they can't drive cars safely and should no longer be trusted to drive.
More seriously though, it would be more interesting to compare number of kilometers driven without accidents compared to national statistics. And having something like the NTSB do thorough investigations when accidents do happen. Ultimately there is a need for autonomous driving because it should eventually cause less deaths
If a fault with the vehicle parts itself causes an accident in a human driven car then the manufacturer absolutely should be held accountable.
Also, if you believe Uber or any company is pursuing autonomous with the goal of decreasing the amount overall automobile deaths I daresay you're extremely naive. It's probably not possible to achieve anyway on a road shared with human drivers.
I don't believe that Uber's goal (or any other companies) is to decrease the amount of overall automobile deaths. However I believe that the widespread adoption of automated driving will eventually result in a decrease of automobile deaths.
Some blame, too, belongs to the human driver. His job was to look for exceptional conditions that the car may not pick up. And he failed.
It's quite possible there wasn't time for a good human driver to stop completely or avoid a pedestrian where one shouldn't be, and this accident was all but unavoidable, but we'll never know because no attempt was made. Given the speed of the car, the woman crossing the street either miscalculated, was betting that traffic would see her and slow down, or didn't see the oncoming car.
It was also odd that they disabled the car's stock emergency collision avoidance system.
When the software I write fails, and it does, someone doesn't get to read a web page. Nobody dies or even gets very inconvenienced. This story reminds me of how good a thing that is. After 40 years of practice I'm still not competent to write such high stakes code. I hope the people who do write it are far more competent than me, but still at least as skeptical about their own capability.
An interesting trap is people in the field of social media (FB/ Twitter) not realizing that they are in that high-stakes environment where bad privacy settings and leaks can cause death.
(I don't know how competent you are, but it's likely that) nobody is more competent than you are. We solve high-stakes situations not by installing different people, but by installing better systems and by double-checking things.
In my job we ensure the quality of high-risk things by having multiple layers of review and quality control. It doesn't matter how good the people are who are assigned to the task. We need double-checks. Lower-risk things are not subject to this level of quality control.
Most computer software is low-risk and so it gets very little review. Unfortunately some computer software is much higher risk but does not seem to get review commensurate with its risk.
Yeah, this is so important to understand. The fundamentals of creating safe systems revolve around putting in place good procedures and systems, not just by 'being better'. A great example is one of the ways that surgery was made safer was simply by introducing check-list to ensure all pre-surgery checks were done. Surgeons didn't get smarter or more diligent, they simply had to check off all the basic things before a surgery and that forced them into a situation where they couldn't miss obvious things. Just introducing that basic checklist according to studies reduced in-hospital mortality for surgeries by close to 50%.[1]
You are right, if this was a code issue. But it's not a code issue. It's a fundamental misunderstanding of the situation. It's not a problem that can be solved with code. People vastly underestimate the power of brains due to lifetimes of science fiction anthropomorphizing machines.
When I learned to drive they stressed unanticipated objects coming into the street - especially balls: don't look at the trajectory of the ball and judge whether it will be in your way, instead look for the kid who will be running after the ball while not paying attention to traffic. Also in my state if you hit a kid (a minor actually) on a bicycle you are guilty until proven innocent. That's the law. You have to prove there was no reasonable way for you to avoid the accident.
It’s mind boggling that tracking history wasn’t maintained between object classifications. Surely the best thing to do would be to detect "object" -> track it -> classify -> predict path -> back to classify
593 comments
[ 6.4 ms ] story [ 342 ms ] threadMaybe a few executives going to jail will put an end to this cowboy attitude where lives are on the line.
https://www.ctvnews.ca/video?playlistId=1.4658613
If that’s “dangerous”, well, so is piloting it manually.
That is just as dangerous as using your smartphone while driving in a parking lot, which is forbidden for good reasons.
AT+RELEA
NO CARRIER
Unless the car brakes the very instant the connection drops (and not when the timeout propagates through the networking stack), now you have two problems.
For the sake of argument, could Musk’s statement, along with his very public insistence of being actively engaged in the design process, demonstrate a cavalier disregard for proper safety engineering at the management level (outside acceptable industry practice) and so a defective process?
At least 98% of objects are categorised, the rest don't matter. /s
Because that kind of safety-conscious culture needs to be deliberately enforced, from the top all the way down, by a system that takes a long view.
If we're making an analogy to the defense industry, then building self-driving cars as a tech startup is closer to the Anarchist's Cookbook than Northrop Grumman.
The justification made for disabling the emergency braking - that it would interfere with data gathering - might appear reasonable at first sight, but it does not stand up to scrutiny, for if the emergency braking is triggered, the driving system has already made a mistake, and you already have the data on that malfunction.
They had hired a driver to sit behind the wheel to monitor the road and the car for exactly this reason.
The driver they hired decided to watch a movie on their phone instead of paying attention to the road.
If Volvo's system is that dangerous, then it should not be on the road at all - but there is no evidence that it is, you are just making a speculative argument.
> They had hired a driver to sit behind the wheel to monitor the road and the car for exactly this reason.
That is no reason to disable a safety feature that would add safety in depth.
> The driver they hired decided to watch a movie on their phone instead of paying attention to the road.
That was a major error - a crime, in fact - but, unfortunately, also an entirely predictable scenario that cannot be dismissed on the grounds that dealing with it would make testing more difficult or expensive. So now we have three errors.
The built-in emergency braking system from Volvo does this [0] but Uber deliberately disabled it (presumably because it conflicted with their self-driving rig).
[0] https://www.media.volvocars.com/global/en-gb/media/pressrele...
If you hit the accelerator harder when the Volvo brakes you override it. It should be fairly easy to integrate as a backup.
However I guess that the Volvo system might not be far looking enough for those speeds?
Instead, as the report says:
> The vehicle was factory-equipped by Volvo with several advanced driver assistance systems (ADAS), including forward collision warning (FCW) system and automatic emergency braking (AEB) system. However, Volvo collision avoidance ADAS were not active at the time of the crash; the interaction of the Volvo ADAS and ATG ADS is further explored in section 1.9.
It was apparently disabled by Uber because it was difficult to run the Volvo systems alongside their own, which strikes me as highly irresponsible. Autonomous driving software relies on multiple redundancy, and AEB is the last-resort system for when everything else, from automatic maneuvers to alerting the driver, has already failed.
The deadly assumption here is "doesn't clearly match a category" => "safe to ignore"
I know which I prefer.
https://en.wikipedia.org/wiki/False_dilemma
In this particular case, Uber got a deal with the mayor (Governor? I can't remember) of the city to do testing, and it was clear the government bureaucrat allowed them to do whatever, and then coordinated with the police after Uber killed a pedestrian to fabricate evidence that looked like the woman came out of nowhere.
I would say that is really far from "endless oversight" and much more into the "just mow them down one at a time until you get it right" territory.
Air travel was horribly accident prone too, for decades, and now it is one of the safest forms of travel in developed countries - and for electric/autonomous travel, it will be the same.
But in this case it looks like Uber was really cutting corners that they shouldn't have cut and I think that Uber does deserve sanction for that.
"If the collision cannot be avoided with the application of the maximum allowed braking, the system is designed to provide an auditory warning to the vehicle operator while simultaneously initiating gradual vehicle slowdown. In such circumstance, ADS would not apply the maximum braking to only mitigate the collision."
"Certain object classifications— other—are not assigned goals. For such objects, their currently detected location is viewed as a static location; unless that location is directly on the path of the automated vehicle, that object is not considered as a possible obstacle. Additionally, pedestrians outside a vicinity of a crosswalk are also not assigned an explicit goal. However, they may be predicted a trajectory based on the observed velocities, when continually detected as a pedestrian."
Interesting, from the article and this, it sounds like the system can’t maintain position tracking of an object if it’s classification changes. So even if it could detect a pedestrian, something that could be ambiguous like a pedestrian pushing a bike might have no motion tracking data from one moment to the next, so the car would have no ability to predict its trajectory.
edit after reading the other reports: the victim apparently was under the influence of methamphetamine and the vehicle operator was busy watching Hulu.
The driver was not paying attention to the road and was incapable of performing a timely emergency maneuver (be it a stop or lane change).
However, I don't know if the car did have any external lights on, so it might have been hard to see until it was closer than 6 seconds away.
Why the victim didn't pay attention isn't known as far as I can tell, but from the released video they do appear inattentive.
Page 10-11 of the NTSB report make this very clear: https://dms.ntsb.gov/public/62500-62999/62978/629713.pdf
Additionally, reducing the number of people in the car to one when the car is pretty much by design not capable of handling emergency situations by itself is quite reckless.
> Also, don't forget: the SUV's emergency braking system was deliberately disabled because when it was switched on, the vehicle would act erratically, according to Uber. The software biz previously said “the vehicle operator is relied on to intervene and take action," in an emergency.
The question then was there proper training and communication to the test drivers that it's never okay to look down at your phone. Or whether that was simply unrealistic expectations. Or if the hours were too long, or testing at night, etc.
It said there was 5 seconds which should have been more than enough for a human test driver to hit the brakes, which was their stated job.
If I remember right, Uber did cost-savings by removing the safety engineer and expected the driver to fill both roles simultaneously.
It's pretty well known that humans' minds wander, that it happens more when monitoring reliable systems for rare problems, and that it makes operators less responsive and lowers their error detection rate [1] - as anyone who's attended a boring meeting or lecture can attest!
I'm not sure that anyone informed would imagine a worker spending 40 hours a week monitoring a self-driving car would be able to watch it with 100% attention.
The truth is nobody realistic expects the safety driver to respond to reliably prevent an accident like this - they're there for slower-developing problems, resetting false alarm stops, and taking the blame.
[1] https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5633607/
Companies like https://comma.ai/ are taking a much better approach IMO by keeping it simple by first perfecting lane assist/highway driving + building a driver watching device which alerts them when they stop paying attention for x amount of time. Which Uber should be investing in for their test drivers.
Another important thing is being realistic about expectations, of course 100% paying attention is unrealistic even for normal drivers, accidents will happen regardless. Hitting jaywalkers on a dark multi-lane high speed road is a lot less bad than other possible scenarios and there really hasn't been that many accidents yet.
I would be surprised if it involved anything more than an easily-slept-through safety video. Should have come with periodic checks for distraction.
I really don't think we should be so eager to damn Uber that we forget that there were more than just Uber employees asleep at the wheel here: Arizona has a duty to protect public safety. By permitting a self-driving car test program on the public roadways without doing even basic due diligence in vetting the program first, the State was grossly negligent in that duty.
You couldn't be more right. The zeitgeist in 2015 was full-steam ahead on the autonomous future and woe to anything that stood in its way.
The framing of Arizona's embrace of autonomous testing four years ago [0,1] contrasted with California's caution [2,3,4] couldn't have been starker. One was branded enabling innovation, the other was seen as bureaucratic red tape holding up progress.
It's too easy to get wrapped up in these narratives.
[0] https://azgovernor.gov/governor/news/2015/08/governor-doug-d... [1] https://www.uber.com/blog/tucson/driving-innovation-in-arizo...
[2] https://fortune.com/2015/12/16/google-california-rules-self-... [3] https://www.theverge.com/2015/12/16/10325672/california-dmv-... [4] https://www.nytimes.com/2015/12/17/technology/california-dmv...
Likely the story is that their software flags a lot of things "Other" without figuring it out upon closer inspection.
There can be many obstacles on the road, stationary or non-stationary.
AI can get away with requiring humans to take control in emergency/unusual situations, but not when it is supposed to be "self-driving". The software at least needs to recognise problematic situations and request a safety override or come to a safe stop, it can't just shrug it off and move on.
I agree with that ADS being shit. Objects, once identified, should be accounted for in later updates in some form. E.g. in the way of "pedestrian now behind truck next to me". How the hell can an object switch status back and forth, loosing all its history? This should make the system be extra cautious at the very least. But yeah, safety doesn't seem to have been that big of a priority.
For me, the most important take-away isn't something about Uber being sloppy. It's that the distinction between "machine learning" and "artificial intelligence" really matters, and the efforts of marketing teams to over-hype ML by selling it as AI may be hindering our society's ability to make cogent public policy decisions around these technologies.
(No, I'm not really interested in haggling over ideas like "hard AI" and "soft AI" here, either. Normal people don't split those hairs. Nerds generally don't, either, except when they're trying to score easy points in a debate.)
If you make it stop on each unforeseen event, it'll keep stopping. If you don't, it'll keep killing.
People are tasked with monitoring things all the time. It's not some brand new impossible task.
https://dms.ntsb.gov/pubdms/search/document.cfm?docID=477745...
You can also see the detailed report from the NTSB which will go through about everything you would want to know about the actual driver and her reaction:
https://dms.ntsb.gov/pubdms/search/document.cfm?docID=477743...
Particularly of note is that Hulu was streaming video on her phone until about 1 minute after the crash (page 11 of the report).
* Uber had previously staffed each car with two operators, allowing one to keep their eyes on the road while the other took notes.
* Forthcoming performance metrics for operators would include "the VOs ability to keep the SDV in autonomous mode as long as possible unless there is a Fleet Desk support issue (software issue)."
Obviously not as relevant to this particular case as the Hulu streaming, but to me these details suggest safety isn't their highest priority.
> According to Uber ATG, the SDS did not have the capability to classify an object as a pedestrian unless that object was near a crosswalk.
Jaywalking is extremely common. I've seen pedestrians jaywalk a 45mph 6-lane road, with no median to speak of. Anyone with driving experience should know that pedestrians can occur even with no crosswalk, so it boggles my mind that any engineer would sign off on such a decision.
There isn’t really the term “jaywalking” here. It’s just “crossing the road”. I’m not sure on exactly who has what legal responsibility, but it certainly feels like pedestrians should look out for cars when crossing, and drivers should look out for pedestrians.
A pedestrian ("voetganger") is someone who travels by foot only. If you guide another vehicle by hand, or even a horse or cattle, you are legally considered a driver. Therefore, most laws concerning pedestrians have to make explicit allowances for walking a personal vehicle. For example, the primary Dutch Traffic Regulations [1]:
The rules in this resolution concerning pedestrians apply equally to persons guiding by hand a motorbike, moped or bicycle
[1] https://wetten.overheid.nl/jci1.3:c:BWBR0004825&hoofdstuk=I&...
Anything from "children run onto the street to chase after a ball they were playing with" to "pedestrian jumping out between cars because they didn't see you" or "tram stops in the middle of the road and people exit onto the street".
Legally pedestrians must use marked crossings when available and are told to check both ways, but if a driver hits a pedestrian, it's always the driver's fault (though the pedestrian might share some blame in some circumstances but never the full liability).
It's theoretically the same in most US jurisdictions. However, it's not what the law says that matters; it's what the police will actually do, and in practice, there is a very strong tendency to blame the pedestrian for any accident.
The German law still has some notion of whether a driver could have prevented an accident or not. Not all situations are the fault of the driver.
Extreme example: If somebody runs into your car on the Autobahn that's not your fault.
What about "drug impaired people going onto the autobahn at night wearing dark clothes in a section unilluminated by street lights" which is what I believe happened here?
This implies the woman was not easily spotted. She was, at > 5 seconds to impact - plenty of time for a braking or avoidance maneuver.
The problem in this case was that the pedestrian was impaired by drugs, if I recall correctly.
In my opinion, this happened because of 3 simultaneous failures: failure of the pedestrian to responsibly cross street, failure of the computer system to recognize the obstacle, failure of the computer babysitter to be on the lookout for obstacles and override the computer.
Hopefully we learn something from this triple failure and take measures to reduce their likelihood in the future.
That's overstating it. I think it would be decent under the circumstances to spend 30 seconds Googling it before slandering the victim:
Wikipedia: https://en.wikipedia.org/wiki/Death_of_Elaine_Herzberg
According to the preliminary report of the collision released by the NTSB, Herzberg had tested positive for methamphetamine and marijuana in a toxicology test carried out after the collision. Residual toxicology itself does not establish if or when she was under their influence, and hence an actual factor.
Perhaps you could also argue "if she were wearing a helmet, it would have conferred +1 to CON and perhaps saved her", for what it's worth - makes sense in the crudest of simulations, but not in RL.
In this case the pedestrian and driver are both fully responsible. The pedestrian was lollygagging across the road like a complete idiot. The driver wasn't paying attention.
I think I might agree with something you're trying to say, but this point is completely incoherent.
If doesn't clear things up, I don't know how it could be incoherent.
One example would be if you were leading a walking group of 20 people and wanted to cross the road and it was safe to do so at the beginning but because the group is large it takes a long time. Any approaching cars would not have any right of way until all the group has passed.
But drivers have a duty of care, and if a pedestrian is crossing a road then obviously they must do what is safe, i.e. slow down and let the pedestrian cross if (s)he is in the way.
e.g., where I live, the traffic laws are defined in terms of who has the responsibility to yield, and undergirded by a catch-all requirement that everybody who is able to act to avoid a collision must do so.
For instance in Germany they drill into you in driver's ed that when you hit someone with your car, it's pretty much always your fault. I'd say we don't have a jaywalking epidemic.
It's equally reasonable for pedestrians NOT to have priority to start crossing the road unless there is a clear, marked pedestrian crossing.
Priority means that cars must give way, which is the case at pedestrian crossings: Cars must give way BEFORE a pedestrian has started to cross in order to let him cross.
But in any case once a pedestrian is on the road cars have a duty of care not to hit him.
The reason some US states choose to specify who should yield instead of who gets to go first is because framing things in those terms hopefully alters the ways that drivers think about things in a way that will improve safety overall.
TBH, I suspect that it's an empty gesture that is being used as an alternative to implementing more rigorous driver education and licensure requirements; the US tends to view driving as more of a right than a privilege.
Drivers are required to give "due care and attention" to driving which can be demonstrated by following the "highway code" and that code tells them pedestrians might do things they don't expect and to assume if it's not clear what's going to happen then yeah there is a pedestrian behind that obstruction and they are going to run into the road in front of you, whereupon hitting them would be your fault.
For example when I was a child I got off my first bus home from secondary school, and ran straight into the road in front of a car I couldn't see because the bus was in the way. The horrified driver was legally responsible for that, even though she hasn't intended to hit me. I believe she would have been automatically billed by the authorities for the cost of shipping me to a hospital to have my broken leg set and so on.
Clearly it is in some sense my fault that happened, but on the other hand it's not me choosing to drive a huge steel box at 30mph past a bunch of idiot children...
https://en.wikipedia.org/wiki/School_bus_by_country
It was simply a bus. Specifically it was the hourly bus from the nearest large town (where my new school was) to the village I grew up in.
The car passing the bus had no chance of seeing her, and there was no reason to believe anyone would be crossing against the green light. I think she may have had a pattern-match malfunction and acted as if she'd just disembarked a school bus. It was a tragedy all around.
I'm not saying it was your fault, it just never ceases to amaze me how car-centric the US is (that's after over a decade of living here)
Anything can get in the way of a moving car. Pedestrians are unpredictable and should be kept an eye on at all times.
In the UK it is legal for pedestrians to cross roads anywhere unless specifically forbidden (that's basically motorways). In general accidents it will be deemed the driver's fault (if you see a pedestrian you're supposed to slow down and avoid, not honk and continue because you have a duty to drive with due care and consideration) unless the pedestrian acted in such a way that the car could not avoid the accident.
All sorts of hazards can come up in the road, such as children running into the street, deer or other animals.
This scenario was bound to happen given that auto-breaking was disabled and the human driver was playing with their phone instead of paying attention to the road, which was the whole reason they were in the car.
Only the Volvo auto-breaking was disabled; the Uber software/hardware itself should have handled the situation fine.
Going further, I wish for more initiative to re-design infrastructure putting the safety of pedestrians, cyclists etc at the forefront. It is simply not safe for alternative transportation to share the road with vehicles, autonomous or otherwise. Coincidentally, if infrastructure were designed to limit or prevent the interaction between cars and non-cars, the problem of detection for autonomous systems becomes much easier. Unfortunately, society would rather prop up the free market with capital than invest in public works which benefit more than corporations and shareholders.
As it is, we bash every crash and failure of such cars and may prohibit those from public roads while statistically they might be making our lives safer, not more dangerous. My 2c.
But when a system that is designed to autonomously detect infinite, non-discrete, random variables fails prevent a collision/incident, you have a failure. There is no grey area, Y or N. The hope is, the system is smart enough to determine exactly 'why' there was a failure (i.e. the system recognizes the input, a pedestrian in this case, but was not able to prevent collision due to the constraints of physics, or without a nominal increase for concurrent accidents). Still a failure, albeit a failure with a greater snapshot of heuristics and causality.
In essence, because the people who control the data have an interest in revealing data that is superior to human performance, the lack of conclusive data on this leads to an inference that it isn't.
Sure!
Traffic fatalities in the US, human drivers: 1.25 deaths per 100 million vehicle miles [1]https://en.wikipedia.org/wiki/Transportation_safety_in_the_U...
Uber's self-driving car fatality rate: 1 death per 2 million miles[2]
That is, Uber's state-of-the-art is an order of magnitude worse than human drivers.
I know, you'll complain about small sample size. The thing is, assuming Uber's cars match humans, the chances of a collision happening only after 2 million miles are very small, like pulling an ace of spades out of a shuffled deck of cards.
The point is, we do have the data, and the data does not support optimistic beliefs.
[1]https://en.wikipedia.org/wiki/Transportation_safety_in_the_U...
[2]https://arstechnica.com/cars/2018/10/waymo-has-driven-10-mil...
Say, you claim your product works, on the average, for 80 years before breaking.
I believe you, buy it, and it promptly breaks within 2 years of operation.
Maybe I was unlucky. But high more likely, you were bullshitting me.
That's what we have here.
Human death rate: 1 death / 80 million miles
Uber's death rate: 1 death after 2 million miles
So yes, we can make judgement calls.
If Uber has such amazing technology that avoids accidents, they could deploy them as accident-avoidance for regular drivers. They could give normal driving control to the humans, and take over only if there is a potential of accident.
If you argue that these driver assistance technologies already exist, then you better not be comparing your magical AI on supersafe cars with sunny climate on well marked highways to the average driver on the average vehicle under average climate on average roads.
Another point to consider, is that the actual experience of autonomous driving may increase the likelihood for ‘manual intervention’ failures due to the system being able to react correctly most of the time!
Without simultaneously reporting the identical data for human driven vehicles in a complete, accurate, and globally comprehensive way there is little that can be done with such data except use it to make political attacks on the technology.
As comparison: every single battery electric vehicle that has a fire is widely reported and disclaimed, impugning the technology as unsafe. The 10s of thousands of flammable fueled vehicle fires are completely unreported in the media.
*edit whoops don't know HN link formatting :)
Pedestrian crossings aren't provided on most roads outside of shopping and business areas - you simply have to cross at a safe point and try to be visible.
I really don't know about the US but I can't imagine it's that different: surely people need to be able to cross the road without being run down as an unclassified object.
FYI, this is an American thing[1]. There's no jaywalker in Russian, for instance. Or French. If you know any other language that has a word for that, let me know.
It didn't even spread to the UK:
"In other countries such as the United Kingdom, the word is not generally used and there are no laws limiting how pedestrians can use public highways."
[1] https://en.wikipedia.org/wiki/Jaywalking#Origin_of_the_term
I'm not convinced that the pedestrian outside crosswalk omission arose because of U.S. car-centric thinking. I think if any software engineer working on the project was told that their autonomous driving system wasn't watching out for these pedestrians, they'd be horrified. In fact, probably many of them were already horrified at how poorly their system was performing in general, and yet management still deployed the software in actual physical cars.
So out of complete necessity, these cars need to be programmed to actually ignore pedestrians unlikely to cross into the path of the vehicle.
This is a tough problem because the AI has to identify the human (seems as if it did in this case) AND the intent. I've seen two recent videos of Teslas abruptly stopping for humans on the side of the road. One was actually a bus stop poster model and the other clearly had no intent of crossing the street and angry motioned the Tesla on.
Imagine driving by a group of joggers on the side of the road. Who can you pass safely? Who many stumble a bit into your path? Who will prepare to cross but first wait for you to pass by? Who will try to cross in front of your path? The micro-decisions and predictions made are very challenging to get right.
Yeah, they really are. So... maybe they just shouldn't have self-driving cars until they can figure that stuff out?
Sometimes it's preferable to hit something in your own vehicle. Cue the classic moral question of which person would you kill if you can only avoid one - with your vehicle - by hitting another.
But I'm honestly not sure what the law "requires" if you've got a scenario where there are going to be bad outcomes no matter what you do.
As I say though, if you can't swerve to avoid people, the most reasonable action that most people would take--to the degree they had time to make a conscious decision at all--would be to brake as hard as they could and let things play out as they will.
At least here (in the country) it is not uncommon that wild boars or deers cross the road suddenly at night, and I have seen cars literally destroyed by the collision, in some cases with the driver seriously injured.
I worked in the SDC industry. It’s mostly a science project.
Between its economics and its car I don’t understand why Lyft isn’t heavily shorted.
The amount of time someone spent on a software project isn't the best indicator of its quality... especially for science work where there's a large amount of knowledge transfer within the field.
Lyft hired a top Google engineer to run the project and plenty of other experienced people. They weren't starting from scratch like Waymo and nor did Uber.
The car simply shouldn't have been tested without a driver (or two) constantly paying attention. Clearly Uber didn't trust it to be running by itself yet (especially with the emergency brakes being disabled) but it basically was if the test driver wasn't paying attention.
Europe is not a country and each European country has different laws: https://en.wikipedia.org/wiki/Jaywalking#Europe
However, it is not the role of a driver (or their car) to serve death penalties for that.
« Every » rather than « most », 100m seems to be the Eastern European limit and going down the list in western or northern Europe the limits seem far lower, usually 20 to 50. Furthermore pedestrians not following this often does not absolve drivers from any (let alone all) responsibility.
https://news.ycombinator.com/item?id=20882763 https://news.ycombinator.com/item?id=20520124
This crash seems to be the result of several cases not being handled correctly, or at all.
1. Pedestrian crossing -not- at a crosswalk, aka jaywalking
2. Objects tagged as “Unknown” don’t have their path tracked.
By the time the car saw an “unknown object” (the woman) directly in its trajectory it was too late (1.2s before impact).
Why was there no system for a person jaywalking? That’s extremely common.
Why don’t “Unknown” objects get stored and individually have their trajectories tracked? Hello?
Programmers need to start going to jail.
But yes, managers and the like involved should also be going to jail. This cavalier, negligent attitude needs to be dealt with yesterday.
Because what if that "unknown" object is just a plastic bag blowing in the wind?
People would get mighty annoyed very quickly if emergency braking kicked in for every piece of garbage blowing onto the road.
Yeah, cases like that might make people wonder about the correctness of the self-driving computer.
No, this is part of the crime here.
Having cars that can drive themselves just doesn't seem like a particularly high priority for society at large in the face of other looming issues. Why allow it to proceed in such a dangerous fashion at all?
(That said, the negligence in the Uber case makes it pretty clear they are likely far from reaching that level of competency)
Gee, where have I heard that script before?
The "greater good" utilitarian argument has been the basis of some of the worst policies and politics in the world.
I'm not saying that self driving cars fall into the same category, but how many deaths are you okay with until Uber/Waymo perfect their algorithms (and later, charge you for it)? 1? 10? 100?
It is very sad that this pedestrian died and companies that kill people through avoidable accidents like this should be punished. Uber specifically seems like a trash tier self-driving car research program and I wouldn’t mind if they just stopped.
But one day, even a system that’s far safer than human drivers is going to kill someone accidentally. It is going too far to say, a method of driving cars that ever kills somebody should be abandoned.
More seriously though, it would be more interesting to compare number of kilometers driven without accidents compared to national statistics. And having something like the NTSB do thorough investigations when accidents do happen. Ultimately there is a need for autonomous driving because it should eventually cause less deaths
Also, if you believe Uber or any company is pursuing autonomous with the goal of decreasing the amount overall automobile deaths I daresay you're extremely naive. It's probably not possible to achieve anyway on a road shared with human drivers.
It's quite possible there wasn't time for a good human driver to stop completely or avoid a pedestrian where one shouldn't be, and this accident was all but unavoidable, but we'll never know because no attempt was made. Given the speed of the car, the woman crossing the street either miscalculated, was betting that traffic would see her and slow down, or didn't see the oncoming car.
It was also odd that they disabled the car's stock emergency collision avoidance system.
https://www.bbc.co.uk/news/blogs-trending-45449938
Unfortunately, there still aren't enough of us. We're trying to change that.
In my job we ensure the quality of high-risk things by having multiple layers of review and quality control. It doesn't matter how good the people are who are assigned to the task. We need double-checks. Lower-risk things are not subject to this level of quality control.
Most computer software is low-risk and so it gets very little review. Unfortunately some computer software is much higher risk but does not seem to get review commensurate with its risk.
[1]:https://www.who.int/patientsafety/safesurgery/faq_introducti...
https://news.ycombinator.com/item?id=21250424