It seems a little weird to focus on this. The article itself admits it may be standard practice for makers of self-driving tech to disable any built-in automated driving features in the car. This makes sense: presumably there's no good way to talk to the built-in system; having a second system running that can't coordinate with the self-driving system would likely make the combined system less safe. Even if communication between the two were possible, I'd still imagine that just having a single system operating the care would be much more desirable.
>having a second system running that can't coordinate with the self-driving system would likely make the combined system less safe. Even if communication between the two were possible, I'd still imagine that just having a single system operating the care would be much more desirable.
Multiple redundant systems that do not need to talk to each other to coordinate is generally safest.
And if the existing systems can not be coordinated with and can make cars less safe because of this, then how are drivers managing it?
I'm not sure I'd say that this is being "focused" on. It's an interesting bit of info, the other players in the space (Intel, Volvo, Aptiv) have an interest in getting their spin out there, readers are interested, so we get this article and everyone is happy. It's not a super critical key to the story, but it does help explain what happened in some small way.
> having a second system running that can't coordinate with the self-driving system would likely make the combined system less safe.
That seems like a bit of a leap to me. The Volvo system is not super-advanced; the relevant bit here would probably be their "Collision Warning with Auto Brake" feature.
Multiple independent systems with the ability to sound an alarm and slam on the brakes if they detect an imminent pedestrian collision...seem like they probably would be safer overall? They'd certainly be safer for the pedestrians, and we're currently at 1 pedestrian fatality and 0 fatalities for "people in cars killed due to overzealous/conflicting automated safety systems".
I rather suspect Uber disabled it because they wanted better data, not because they thought it was safer for other people on the road. That's certainly a valid choice, but one which might have been wiser if their tech was a lot more reliable.
I was already aware of that one, but sure, that's another example of the opposite of what I asked for. Rather than someone killed by conflicting or over-cautious safety systems, that's an example of someone killed by a single ineffective safety system, just like the pedestrian in Arizona.
Disabling a backup system like that strikes me as very negligent from an engineering perspective. They had reason to expect their system to fail because it's a prototype, but then they go and disable another system that is actively marketed and that could have prevented this death.
I'm so glad they're not testing their cars in my state. This makes me wonder if they even do basic safety engineering like FMEA or if they just wing it.
But it was never built as a backup system. It's a whole feature that was designed from the ground up assuming it's the only thing on the car doing anything like the job it does.
Actually, those systems are designed as backup. They are designed as backup, when the driver does not react. That is the reason why they warn first and initiate an emergency brake only, and only if an accident becomes unavoidable.
So that would, if the automated tech from Uber would have detected in time, it would have reacted like a human driver and the pre-installed system would may have not noticed it. But in case, it would have reacted as a backup.
On the two occasions my Volvo has decided to take over (both admittedly due to overhanging bushes in a parking space), it has taken over control immediately and entirely.
No warning necessary because self driving cars are completely capable of being programmed to deal with varying ability to control the car. It can still have an intention whether or not the car is allowing it to proceed. It needs to make a best effort, allowing failure when a backup system takes control.
The collision avoidance system? The fail-safe that's meant to ensure the car doesn't hit something when the human driver screws up? The one that works with an entirely separate autonomous intelligence (aka, the human) controlling the car? The one explicitly designed to trigger emergency brakeing even if the primary controller is trying to do something else? The kind of system installed on many other AI controlled cars to avoid this exact type of disaster?
That backup system?
Look, it's entirely possible that for some reason this specific safety system conflicted with the current version of Uber's system, but we have zero data to suggest that, and if so, it's something specific to those two systems in that particular vehicle (and suggests a severe issue with Uber's system to boot). There's no general issue here.
> There's 2 chefs.
Yes, and sometimes one of them takes over from the other one. That's how this works.
Have you actually driven with one of these systems?
There are plenty of intentional maneuvers required to integrate with real-world (ie. non perfect) traffic that trigger these systems.
They’re designed to take hints from human input (so for example, LKA responds less strongly to forceful unexpected inputs than a lack of expected input).
There is 0 chance the system wouldn’t interfere with a self driving.
Not to mention many of those safety systems include lockouts on the CAN bus which make removing their safety module a first step to even add your own inputs. I know that’s how it works on my Volt with a Mobileye sensor
Problem is, that greatly complicates the problem-solviing process for SD vehicles---you're now trying to build atop a model of the world provided by the vendor, with its own faults and quirks (and you probably need an agreement with the vendor to see their code / dissect their model, and now your release schedule depends on their release schedule, etc.). It also makes Uber dependent upon Volvo's buy-in on the project, and that arrangement is fraught in a world where we don't yet know if SD cars are going to damage the business of traditional auto manufacturers 50 years out---Uber wants to be in a place where if a vendor relationship sours, they can pack up and change chassis without a complete overhaul of the SD logic.
If your SD system is supposed to be solving 100% of the same problem anyway, it's best to gut the software you don't control and work with the hardware as directly as you can. The notion of building atop a lower-level abstraction makes for great academia but doesn't work with industrial realities.
Thanks for the explanation and I agree that it won't be as easy to build SD for auto braking systems that change from chassis to chassis, but this is what I think people want. People want to have fail safes from different vendors.
Automatic emergency braking is quite definitely a backup system. If it ever trips in my car it is because the primary system, that is me, has failed to brake in time.
> Multiple independent systems with the ability to sound an alarm and slam on the brakes if they detect an imminent pedestrian collision...seem like they probably would be safer overall?
While yes - it would also force Uber to permanently partner with Volvo on technology they can't control. 'Uhuh, my Ubers a Chevy not a Ford - it could kill me because the Chevies don't built-in X is worse!'.
You do not want an arbitrary third party system controlling the car while the Uber system is in unknown ways; if the systems conflicted things could go out of control and be more dangerous.
> it would also force Uber to permanently partner with Volvo on technology they can't control.
That sounds like a win-win to me.
>You do not want an arbitrary third party system controlling the car while the Uber system is in unknown ways; if the systems conflicted things could go out of control and be more dangerous.
I find it interesting that, of these two systems, it is Volvo's widely-deployed system that is described as 'arbitrary'. If this argument had any merit, it would be unsafe for this system to be on the road, 'conflicting' with the car's human driver. On the contrary, the statistics clearly show that these systems reduce the frequency and severity of accidents.
I know this sounds bad, but the arbitrary is that the Ubers aren't designed to be as safe as possible currently - they're designed to be testbeds of the new fully autonomous technology uber is developing.
Maybe it makes sense to rely on arbitrary vehicle specific fallbacks, but long-term it doesn't as I'm sure They don't want to be dependent on a specific model. When you request a ride, do you want to care about whether your getting a Honda or a Volvo?
No, redundancy means making a backup of your primary system so it can take over, not having a completely different uncooperative system active at the same time.
This is just one of many kinds of redundancy. Please see [1] for several ways in which redundancy is manifested (e.g. triple modular redundancy).
Specifically, uncooperative systems running at the same time was exactly how fault tolerance was achieved on the space shuttle[2]:
>For true redundancy to take place, all computers must listen to all traffic on all buses, even though they might be commanding just a few. That way they know about all the data generated in the current phase. They must also be processing that data at the same time the other computers do. If there is a failure [...], since the Shuttle's other three computers are sending apparently correct commands to their actuators, the failed computer's commands are physically out-voted
That's just one example. There's no need for the definition of the word "redundancy" to be as narrow as you said.
Why not, I have 2 different systems that detect an imminent impact in a wall, then if the systems do not agree you stop, if your main system is experimental then having the backup system that is using different tech then it seems obvious that is safer to stop, then analyze the cause of the conflict.
If you walk into a store and don't see that it is being robbed at gunpoint, it would be very nice to hear the assailant ordering the cashier to empty the til. You have two chances to steer clear.
Either your eyes or ears may fail you now and again, but you're better off with both.
Nothing wrong with having a chance to steer clear, but it strikes me as an odd reaction unless you live in a society where only the criminals can defend themselves.
In engineering we don't rely on our feeling. We have to actually have reason and evidence. You're approaching it as if the car should be moving at all costs, but the responsible engineering task is to keep the car moving unless the automated systems indicate that it's hazardous to others.
Yes, it's redundantly _safe_, which is the thing we actually care about. Safety is job #1. Well, not at Uber apparently.
There are trade-offs here, sometimes we might decide to take a very small risk, after weighing it up. A famous example is ETOPS. Unlike propeller engines, jet engines for airliners are ridiculously reliable once running. ETOPS (jokingly "Engines Turn Or Passengers Swim") is a scheme which over the past few decades has gradually allowed twin engine jets (which obviously just become over-engineered gliders if both their engines fail) to fly over oceans and other large distances with no airports, on the rationale that although total engine failure would probably be deadly without anywhere to put the plane down, two unrelated sequential engine failures are now so unlikely as to be overall negligible risk to the passengers.
But that needs a risk we understand. If you have no idea what the risk is, you can't make that trade, so redundantly safe systems are the obvious choice.
Did you skip the part where one of the system detects hitting a wall? Why is not the normal thing to slow down and stop? You will trust the experimental system that is in testing over a safer system?
But they aren't conflicting: they have the same goal (not running into stuff).
In fact, there's yet another different, redundant, potentially conflicting safety system: the human safety driver.
If only one were to trigger, it still wouldn't be a conflict.
And yeah, redundancy through using non-identical systems is very much a thing, and considered much safer than having two identical things (which might make the same mistakes).
Source: I've worked on ASIL D safety-critical automotive systems, where diverse redundant subsystems were actually a design criterion.
No, in safety-critical systems redundancy means having a backup system or systems that can support the safety-critical aspects of the system if the main control system fails.
Redundancy is when the primary system fails and the backup system takes over. Since the primary and backup in this case are unrelated, created by different entities, and the entity building the primary system may not be able to fully understand conditions that will cause the backup system to engage, this isn't so much a "backup" as a "second system that could interfere in unpredictable ways".
The code exists! The collision detection and emergency braking feature which many new cars already have, and which (I assume and hope) has been heavily road-tested.
But my point is, Uber’s self-driving car has lots of goals -- drive safely, drive legally, drive quickly, drive comfortably, go in the right direction.
A low-level safety system has a much simpler requirement: if it looks like the car is about to hit something, hit the brakes.
And this second category of code is something that already exists, deployed in production today by multiple car manufacturers.
That’s the bit Uber turned off. “It’s a hard problem!” doesn’t seem like a good explanation.
The human driver can't coordinate with the self-driving system either, he just makes overriding inputs. Treat the built-in safety system as another overriding actor.
I have an XC90. The collision avoidance system has been helpful in a couple of instances (for sudden car stops, not pedestrians). However it also “misfires” a few times per month. Typically this occurs when I’m driving on a curving stretch of road with cars parked along the side: due to the curve, the system will sometimes interpret the vehicles on the side of the road as being obstacles in my path. When it does, I get an alarm and sudden unexpected braking. The first time(s) this happened I was genuinely terrified.
I have no idea how this input would be interpreted by an autonomous driving system. However if I was designing such a system I would be pretty concerned about the potential safety impact. At very least I would want to add code to recognize these special braking events and then I would need to test this code... which of course would be impossible to do in simulation because Volvo’s system is probably closed source and likely exists as a hardware black box. At the end of the day I would probably be more comfortable designing my own collision avoidance system than relying on one that I couldn’t test easily, and that might spit unexpected inputs into my already-complex system.
So no, I don’t see this is a simple question of “adding redundancy”. I think what Uber did here probably makes good sense. The fact that their own collision-avoidance systems didn’t fire is, however, a problem.
You should file a bug on that. Someone coming up behind you might end up getting hurt due to the chain reaction of your car braking unexpectedly (and without a good reason to do so). There's a typical domino effect from one car braking where the reaction times compound and cars further back end up having to brake harder and harder to avoid a collision until one ends up being out of room and out of time.
As has been said elsewhere false positives of stopping suddenly are bad, but are better than false negatives of not stopping when there is something there.
Also the collision-detection systems should be getting tested so that they improve too. You want your fall back systems to be more reliable than the regular system. So if you turn off the collision-detection systems never fix the false positives and false negatives then you'll always have these problems.
I would argue that AI systems should be able to handle sudden and unexpected events like system malfunction or failure.
> At very least I would want to add code to recognize these special braking events and then I would need to test this code... which of course would be impossible to do in simulation
Why? Does Volvos system raise events differently in context? I would imagine it's pretty simple to simulate an event by driving the car into a wall and capturing a dump of the CAN bus.
The autonomous system could take the Volvo system firing as an event like anything else that it needs to reconcile in the system. If anything the autonomous system could be used to improve Volvo's system by capturing extensive data dumps of the events leading up to misfire events.
Having them disable a standard feature isn't per se a problem. However, it is an issue if the car is demonstrably less safe with it removed. It's possible that this car was going too fast for this feature to engage anyway (depends on the age of the XC90: if it was post-2014, it should have kicked in).
The Volvo system uses a LIDAR just like the main Uber system. In principle they should have been able to replicate this safety feature pretty much precisely.
The question then arises; if they're making modifications on systems like that, what other modifications have they made? Are the modifications correct?
The issue with the headlamp angle implies there could be other problems. Modern cars are incredibly complex; pulling out single features might be OK when they are add-ons, but the XC90 ships with a lot of this stuff in every model.
I wonder why they bought volvo's when they don't wanted to use there radar system or were not able to use the radar system. And is it that easy to disable such an important feature which Volvo brag about ?
>>having a second system running that can't coordinate with the self-driving system would likely make the combined system less safe.
I think this case clearly demonstrates otherwise. Think of the factory fitted as fail-safe - if their own autonomous tech fails(like it did here) you have the one last system that will prevent collision(and it looks almost certain that it would have done that here).
The problem is that the standard, "dumb", security system is still more effective that "self"-driving solution from Uber.
How is a "self"-driving solution allowed to drive in real traffic when it can't break in a simple situation where relatively simple, existing systems can already reliably break.
> presumably there's no good way to talk to the built-in system; having a second system running that can't coordinate with the self-driving system would likely make the combined system less safe.
Honestly, this is a cop-out. A substantial amount of information is available from the vehicle's CAN networks - including speed and braking action!
If the autonomous driving system detects sudden and unplanned changes in speed and/or braking (perhaps caused either by the driver, an external safety system or some kind of impact with something on the road) then the obvious choice there is for the autonomous driving system to release control of the vehicle and let whatever existing systems in the car, or even the driver, handle what is happening.
The car's built-in safety systems are a) simpler, b) less likely to malfunction and c) substantially better road-tested. There was no reason whatsoever to disable them.
If the volvo system were to kick in and brake, the uber system shouldn't relinquish all control. It should only give up the brakes, if you are in the middle of a corner, you'd want the self-driving system to keep steering.
The only reason I could imagine if the built-in system interferes with the system-under-test. For example, if the built-in LIDAR messes up the LIDAR of the AI.
Except for emergency systems. If you have two or more independent systems that are able to stop, should any of them should detect an anomaly, you are far more likely to survive a partial system failure. The end result of all systems identifying the problem is the same.
In a modern car there are plenty of systems that independently will start. An example is some cars that are fixed to prepare your breaks if you take your foot off the accelerator quickly, more or less giving you instant breaking. At the same time, completely independent of this, emergency breaking might already have been started by the collision detection system.
> there won't be 2 breaking systems competing with one-another. Different systems controlling the same end-points is never a good idea.
Aerospace engineer here. Different systems controlling the same end-points is literally a basic engineering safety technique. Not only you have the human pilot as a competing system controlling the same end-points as the autopilot, you may have even different competing autopilots.
I had a professor at the Uni who was part of the engineering team of the flight computer of a certain jet fighter. Two engineering teams were given the specs of the system and told to produce a flight computer. The two teams weren't allowed to talk to each other to prevent bugs cross-pollinating the two systems. After all, what good is to have two identical system for redundancy if they can be affected by the same bug? (That's also why pilots eat different meals in-flight!)
Then, the two completely different systems are installed, duplicated, into the plane and put to the test. The four computers must perform equally as per the specs. If the recommended output of a single computer diverges from the other three, that computer is faulty and disregarded. If two computers of the same design have a bug and start giving out-of-envelope readings but the other two are consistent, control is handled to them.
I'm certainly very appalled to see in this thread how safety engineering techniques that every engineer learns in University is completely disregarded or even reversed. At least in the aerospace community, development and implementation of these safety techniques has been paid in blood. These are lessons that shouldn't be forgotten when developing other transport technologies IMHO.
Most of the commenters did not study engineering. They studied computer science. So the only exposure they would have had is working with someone like you.
I wonder if those in self-driving development at Uber et al. share the same non-engineering background. I think it would be ultimately in the interest of these companies that stringent regulation is enacted, so the public can feel as safe and confident riding self-driven vehicles as they feel riding a thin aluminium tube with two gigantic kerosene hairdryers flying at 35,000 ft.
Software people as a rule absolutely suck at dealing with exceptions and in general tend to make a ton of assumptions about the underlying hardware and sensors that may not hold up in practice. The 'happy path' is where 90% of the attention goes, when actually 90% of the attention should go to the exceptional bits. But as an industry we're slowly learning and the habits and practices of other engineering disciplines are slowly seeping into to software.
Aerospace (and medical) is where things are much better.
> to disable any built-in automated driving features in the car
Automatic Emergency Braking (AEB), which is the system that could have prevented this accident, is not an automated driving function though; it is an active safety function. As such, it is more common to stack such functions. For instance, ABS > AEB > automated driving, with the left most being the function with the highest priority, i.e., it can override the functions to the right. This is how most safety systems in cars work in production.
> This makes sense: presumably there's no good way to talk to the built-in system; having a second system running that can't coordinate with the self-driving system would likely make the combined system less safe. Even if communication between the two were possible, I'd still imagine that just having a single system operating the care would be much more desirable.
That's the exact argument I make for never buying a car with that technology because I'm driving it and I have no way to communicate with it...
I'd be super happy to be driven around in an autonomous car, but have zero interest in auto-lane-keeping-pilot-collision avoidance-parking etc
While it could make it less safe in very specific circumstances, note, it's the same as with a human driver
The system kicks in when there is a high probability of a collision ahead. Think of it as the instructor in the driving school that slams the brakes when you're doing something wrong.
So basically, the driver (be either human or another system) did something wrong already if that system kicks in
I can see it from the perspective of Volvo pushing this - one of their cars is now widely shown in the media killing someone, and their whole reputation is safety. I'd also want people to know that Uber was tinkering with the standard safety features, even if it was for good reason.
“We don’t want people to be confused or think it was a failure of the technology that we supply for Volvo, because that’s not the case,” Zach Peterson, a spokesman for Aptiv Plc, said by phone. The Volvo XC90’s standard advanced driver-assistance system “has nothing to do” with the Uber test vehicle’s autonomous driving system, he said. Aptiv is speaking up for its technology to avoid being tainted by the fatality involving Uber ...
No. This is Volvo pressuring Aptiv to make a statement on its behalf, to avoid being tainted by this PR disaster and to shift investigators' focus away from both companies.
Also, in all of the discussions about this incident, let's forget that Elaine Herzberg, a mother of two children, lost her life due to an engineering experiment and publicity stunt ("Come to Arizona! We don't have safety regs!") being conducted on public roads with great risk to members of the public. Volvo, Aptiv, and other participants in these trials should have never signed off ... yet they willingly participated, and an innocent member of the public has paid with her life.
There were multiple failures here. Let's not put all of the blame on the victim.
This particular location has the sidewalk end on one side, and there is a paved section in the median which looks like it is intended for crossing. There is a small sign that indicates that you should not cross here.
From the video, we can see that while there were street lights on the road, the area in which the apparent crossing existed was completely dark. A pedestrian may not have been able to see the sign indicating that you shouldn't cross here; or even if she did, there are so many roads that are built in pedestrian unfriendly ways that sometimes you just have to cross at unmarked crossings to avoid having to go substantially out of your way.
Additionally, the video shows that the driver was distracted immediately before the crash. While the dash cam video seems to indicate that you can't see the pedestrian until it was probably already too late, human eyes are a lot more sensitive over a greater dynamic range than most video cameras, so there's a good chance that if the driver were paying attention, she would have seen the pedestrian before the car did.
The thing about safety systems and standards are that there should be some redundancy in them. People make mistakes, so you shouldn't have to rely on every actor acting perfectly to remain safe. In this case, there was a failure on many parts leading to a tragedy.
It's definitely worth investigating whether Uber had done enough to ensure that operators will be staying alert during the entire test, and whether Uber had done enough testing before being on open roads of small, possibly moving obstacles on dark parts of the road, like pedestrians, deer, and so on.
I'd also put a good amount of blame on whatever department designed that portion of road and sidewalk.
The question isn't if Volvo is in any way culpable (they almost certainly aren't) but if there's a risk that Volvo might appear in any way culpable to the general public. The very last thing Volvo wants is the headline "Self-driving Volvo kills pedestrian".
To be clear, I was specifically responding to the parent who somehow acted like Volvo was somehow complicit in this crash and thus wrong to distance itself.
Critical thought would agree. Unfortunately for Volvo, the vehicle was a Volvo and without proper context (e.g. only looking at an image) it'd be easy to miss that Uber's self-driving technology was being used - not Volvo's.
Are you saying that Volvo and Aptiv should have stopped Uber from running these tests in Arizona, because Uber disabled Volvo's proven safety features?
> let's forget that Elaine Herzberg, a mother of two children, lost her life due to an engineering experiment
I'll sound harsh, but to be honest it seems that Elaine Herzberg's behaviour also played a role in this drama.
The video is a perfect example of jaywalking.
Dressed in black, crossing the street at night in a dark place, no lights, not reflective jacket or zebra crossing area, not paying enough attention to traffic. In the video she's not even looking at the car. There is not signs of the woman stopping or changing their trajectory to avoid the crash.
Most industrial controls have safety interlocks which are independent systems with hard limits. Why is it unrealistic to think that an AI system couldn't be trained to work with-in the limits imposed by these safety systems? If the AI system is being interrupted by the independent safety systems in the car the the AI is doing something wrong.
The point is that Uber should only be disabling these safety features on public roads if their technology is superior.
If their technology is less safe, and in this case it appears to be that it is, they shouldn't be driving around on public roads. Keep it to test tracks.
Volvo's system would have either stopped or slowed enough to make this crash non-fatal. That's worth pointing out. There is no point in Uber testing technology that is worse than what you can buy any number of cars today.
...playing Devil's Advocate, as GP said, it's common practice when testing any self driving tech to disable these features on the cars in question so your automated system isn't fighting with another set of automated systems trying to control the same vehicle. That sounds far more unsafe and unpredictable than just disabling the built in ones.
Personally I think it's naive and unreasonable to think we're going to get totally safe and rock solid self-driving tech without at least a few coffins worth of people who gave their lives to show us how it could go wrong. There isn't one safety system in existence that wasn't built on top of a pile of corpses.
I've had a car with automatic breaking system for a while now and my experience is that they do not kick in easily. Had to test it with a friend that throw an inflatable mattress in front of the car to make it react.
It should not be too hard to make these systems co-exist. If one of the systems says break then the car will break. The few times the car breaks when it really didn't need to should be an acceptable side-effect during testing.
Except this is a prototype, not a final product. Were a self driving car being made for public use, there would not be a pre-existing safety system to bypass or modify; it would be built in, or at the very least, the car would be delivered for modification with that system already removed.
If you had a firewall in your network, for example, that had both a firewall system that you configured, and alongside it, a firewall configured by a third party with "best practices" in mind, that would be frustrating to configure and test because you'd never know for sure if it was a configuration you didn't setup correctly, or if it was this other firewall that you couldn't operate interfering with whatever it was you were doing.
When designing things like this, step 1 is you remove variables that could change the behavior of the device in question without you knowing why it did. I'm sure once completed, a self driving car system could well be made to interface with the safety systems otherwise present, or could be designed in such a way where they wouldn't conflict. When you're in the thick of building that system, however, an additional (and by your comment, not exactly predictable) system to contend with, that you have no access to, is going to be a pain in the rear.
No, you do NOT remove security systems when trying out an untested system. If anything you should add MORE.
Again, the stock auto break systems rarely engage unless there is real danger. By that I mean last second breaking, absolute last second. Most people can drive a lifetime without ever having them engaged. If you cant program a self driving car that can handle that then you're in the wrong business.
Only yesterday there was this discussion [1], in which the general consensus was that the safe way to do things is to put the 'smart' goal-oriented driving stuff on top of the 'dumber-but-reliable' technology to handle specific dangers. It is the latter that has produced most of the statistical evidence justifying the claim that automation makes us safer.
I am very skeptical of your claim that this is a less safe way of doing things. There is, in fact, communication between the two systems: the safety system 'says' to the driving system, "no you don't", and that should not be up for negotiation any more than it is when a human is driving. As was pointed out in the other discussion, if the driving system cannot find a way to work within the restrictions of the safety systems, it is doing it wrong.
You could also record the decisions of the self-driving system and replay the scenario in a simulation environment (either virtual or closed course). Keep the stock safety features on for public roads and still get useful data.
Isn't this how the human control systems are? Without reading the referenced convo, I can see many parallels.
Like we have reflexes that we don't have full control over, that often don't even interact with our brain but rather are initiated from the spinal cord, which I would classify as dumb in this instance. And layered on top are emotional responses, memories and finally consiousness as, arguably, the control layer.
Having a dumber system underneath the smarter system is a great idea. I'm not arguing against that. I'm arguing against having two separate systems that are unrelated and are built (likely) without a deep understanding of how the other behaves.
Uber themselves should have a dumber system built underneath the smarter system (and maybe they do!); that way they understand all the variables that govern the operation of both and can be reasonably sure that the two systems won't fight each other. They can also build an interlock that causes the smarter system to completely disengage if the dumber system has to take over. I of course don't know this for a fact, but I wouldn't be surprised if Uber doesn't and can't have a complete understanding of how the Volvo system works. Disabling it makes sense. Again, they should replace it with their own dumber system, but that's a separate issue.
> Even if communication between the two were possible, I'd still imagine that just having a single system operating the care would be much more desirable.
If I had a Volvo XC90, I would want to operate it with two systems: the collision avoidance features plus an advanced system of muscles, eyeballs and brain. The whole point of collision avoidance is to operate in conjunction with a driver. Self-driving should be more or less just another driver.
Aren't humans glorified self-driving systems? The second system is a backup for when the human makes a terrible mistake, similar to how this could've been a backup for the computerized self-driving system.
It reads like a story press-released by Volvo to make sure Volvo is CYA'd in this in the public eye (lest uninformed consumers find themselves wondering how good Volvo's collision avoidance is if it couldn't stop that Volvo-branded car from hitting that woman in Arizona).
There you go. Was seeing the videos of other drivers passing same road. Even the Google XL is better quality than Uber cam.
Also, all the lidar gimmick looks like not worked well there.
Like the Intel statement from article.
Hope there will be some sort of requirements for this equipment defined. Or technically benchmarked before allowed on road.
>
Intel Corp.’s Mobileye, which makes chips and sensors used in collision-avoidance systems and is a supplier to Aptiv, said Monday that it tested its own software after the crash by playing a video of the Uber incident on a television monitor. Mobileye said it was able to detect Herzberg one second before impact in its internal tests, despite the poor second-hand quality of the video relative to a direct connection to cameras equipped to the car.
Tesla, in their immediate response (same article), basically tried to imply they have magic abilities to do much better, which, based on later crashes, we know is not true.
Mobileye, in response, chose to end their partnership with Tesla (and yes, it was Mobileye who did it. The letter they sent ending the contract can be found if you browse around, after Tesla explicitly asked them to continue and signed a multi-year contract earlier in 2016)
This seems like an obvious thing to do: if you’re trying to test the AI, remove complicating, possibly contradictory OEM systems.
But compare with the other discussion currently on the front page, where users point out system subsumption is a basic safety principle. You need a safety system to fall back on if the AI fails (as it will, at least at this stage). It seems grossly negligent if Uber didn’t re-implement collision avoidance at a lower level.
Does not seem obvious to me at all. First thought I had when hearing of the accident was "how come Volvo didn't break, they advertised the tech quite a bit few years back?"
Why is "car's built-in system took control" harder to handle than "human safety driver took control"? Seems like they are exactly the same: "external system took control".
I would imagine it's easier to hook into human operated controls than to car's internal systems. Just put some sensors on steering wheel, pedals, handbrake and couple of others and you will know when someone uses them. If they are used, give up control of the car (or the specific system that was controlled).
Getting information that some automated system initiated some operation is almost certainly harder. System only knows that the car is not behaving as expected, but it doesn't know what caused it. You could probably figure out how to determine if something like collision avoidance was triggered (large shifts in expected vs actual outcomes), but in case of something like ABS you could end up with two systems fighting for the control over brakes in emergency situation.
Integrating the two systems must be fiddly, sure, but fundamentally not hard and fairly easy to test.
Compared to all the other aspects of making a self-driving car, which are genuinely very, very hard and require groundbreaking technology, it seems incredibly lazy and negligent not to get this bit right.
The auto-brake systems in all cars will activate the same brake-is-on switch as the human operator. Often, it's just a switch on the brake pedal assembly. Trivial to tap into it.
If the switch state changes from closed to open, [driver|AEB] have intervened and Uber's system should relinquish control of that sub-system.
> If the switch state changes from closed to open, [driver|AEB] have intervened and Uber's system should relinquish control of that sub-system.
No, both cases are fundamentally different. If a human driver has intervened, the system should expect the human wants to take over all subsystems: braking, steering, accelerating. If the AEB has intervened, the system should expect it will take over only the braking, and only for a limited amount of time; it shouldn't relinquish control (and doesn't have to, since a properly-designed AEB will be an override of the braking control), and should be ready to take over immediately after it releases (and probably bring the car to a safe stop as soon as it has control back).
That is, the self-driving system must know whether the command came from the human driver, or from another system.
Steering input sensed, give up steering control.
Brake input sensed, give up brake control.
I'm not convinced the two should be linked in any way, human driver or AEB or whatever else. If the human is taking evasive action, the auto-drive is getting input from all the available sensors.
This shouldn't be that hard (relative to the other challenges being faced).
I'd say because night time visibility in Arizona on a clear day is way, way more than 50 feet, and the low-quality dashcam footage you see here is highly misleading.
Low visibility on a clear day is simply lack of illumination, and it is a horrible excuse, because:
1. The car is equipped with headlights, which solve the "limited night time visibility" problem;
2. The car is equipped with multiple types of high-quality cameras;
3. The car is equipped with a LIDAR, which is an active sensing system.
Finally, if one accepts that the visibility was limited to 50 feet, as the video misleadingly suggests, THEN one has to concede that the car was going too fast for how much it could see.
People are taught to drive so that at least three seconds worth of road ahead can be seen. If you believe that either the car or the driver were seeing the equivalent of what the webcam footage was showing, the car was going unsafely fast.
Hanlon's razor would suggest, though, that the car simply wasn't able to process the data available to it. We'll know for sure if the investigation is successful.
It doesn't even need to be edited to be misleading.
Simply taking a crappy dashcam video with low dynamic range suffices; it will make an illusion that anything not directly in the headlights of the vehicle is invisible.
I don't know how people can believe a single word of the early official statements which claimed that the victim "suddenly" appeared in front of the vehicle. The problem of "it's too dark to drive" has been solved a hundred years ago with the invention of headlights.
Here's an experiment anyone can try: turn on the headlights on a dark road, and see if you really see nothing in the lane to your left (driver side). Whatever you do see (which, under normal circumstances would be at least three seconds worth of road ahead) is the way in which the released video is misleading.
I was wondering why it is necessary for self driving tech to be tested in "live" mode, instead of having the software passively log all event data. Then analyze it to see what the software "would have done" compared to a human driver.
Then a lot more testing data could be gathered by outfitting random vehicles (taxis, etc) with the tech, and analyze (and further refine the software) around every event where the software and human driver differed in opinion (i.e., whenever the vehicle abruptly changes speed, did the software detect that it should have hit the brakes at / before the same time).
Any feedback from the response of the vehicle to the commands would be absent. So would whatever happens next. If the computer says "steer left a little" but the video keeps going straight, what's the computer supposed to do next?
With friends like this, why would Uber need enemies? We now have public statements from both Velodyne (makes the LiDAR that Uber uses) and Aptiv (makes the hardware used in the Volvos that Uber uses) saying “it wasn’t us”.
Volvo is the gentleman here, with its “can’t speculate on the cause of the incident” statement.
The pedestrian airbag was introduced before the Automatic Emergency Braking (AEB) safety function was available. Volvo no longer offers the pedestrian airbag as they found most pedestrian accidents were avoided by the AEB.
The V40 is also a rather old car and is based on an old platform. The XC90 on the other hand was their newest car at the time the Uber deal was made and is based on their latest platform. So it is not unusual that both Uber and Volvo would prefer the XC90 over the V40. Besides, given the newer technology in the XC90 it is quite possible it is better in pedestrian accidents than the V40 (with the safety systems enabled on both).
That's quite opposite to how autonomous systems should be built in the first place: start with isolated modules that do not depend on each other, and stack them on top of each other as the abstraction level goes up.
If you have an automatic but a brutal, on-off style emergency-braking module based on, for example, lidar then it's much easier to develop another, maybe less aggressive but much smarter, auto-braking module based on the regular camera inputs. You can count on the failsafe system to act if higher-level modules fail. With braking in particular, maybe even a couple of redundant low-level emergency brake systems would be a good idea before you even consider any of the more high-level systems. You really don't want to hit jaywalking pedestrians or wandering drunks because both kinds do exist in reality.
Similarly, once the car has a couple of obstacle-detection and auto-braking systems up and running, it's easier to work on autonomous driving because you can be confident that your computer won't be able to drive the car into anything as these isolated and separate systems will take care to stop the car.
Consequently, when you're developing the navigation and routing system it has to be able to rely on the driving functions that deals with the present-time traffic situations, like keeping up with the flow of traffic, slowing down, changing lanes, etc. Again, same thing as with emergency brakes: to develop a smarter higher-level system you need to have lower-level failsafes in place.
In the case of autonomous driving a failure should mean a harsh but unnecessary full stop or the inability to continue, not a collision or the inability to stop. They should definitely have left Volvo's own system on as an additional failsafe mechanism.
Maybe disabling Volvo's built in autonomous braking capability is needed for operating an autonomous driving system. It seems intuitive that multiple autonomous systems could clash.
But it also seems like "It works better than the stock production autonomous braking" would be a gating factor in putting these vehicles on the road.
141 comments
[ 3.5 ms ] story [ 70.6 ms ] threadMultiple redundant systems that do not need to talk to each other to coordinate is generally safest.
And if the existing systems can not be coordinated with and can make cars less safe because of this, then how are drivers managing it?
I'm not sure I'd say that this is being "focused" on. It's an interesting bit of info, the other players in the space (Intel, Volvo, Aptiv) have an interest in getting their spin out there, readers are interested, so we get this article and everyone is happy. It's not a super critical key to the story, but it does help explain what happened in some small way.
> having a second system running that can't coordinate with the self-driving system would likely make the combined system less safe.
That seems like a bit of a leap to me. The Volvo system is not super-advanced; the relevant bit here would probably be their "Collision Warning with Auto Brake" feature.
Multiple independent systems with the ability to sound an alarm and slam on the brakes if they detect an imminent pedestrian collision...seem like they probably would be safer overall? They'd certainly be safer for the pedestrians, and we're currently at 1 pedestrian fatality and 0 fatalities for "people in cars killed due to overzealous/conflicting automated safety systems".
I rather suspect Uber disabled it because they wanted better data, not because they thought it was safer for other people on the road. That's certainly a valid choice, but one which might have been wiser if their tech was a lot more reliable.
Not trying to take sides here, but the last part strikes me as "citation needed".
Do you know of anyone who's died that way?
This further reinforces my point, I think. :)
I'm so glad they're not testing their cars in my state. This makes me wonder if they even do basic safety engineering like FMEA or if they just wing it.
There's 2 chefs.
So that would, if the automated tech from Uber would have detected in time, it would have reacted like a human driver and the pre-installed system would may have not noticed it. But in case, it would have reacted as a backup.
There is no warning.
The collision avoidance system? The fail-safe that's meant to ensure the car doesn't hit something when the human driver screws up? The one that works with an entirely separate autonomous intelligence (aka, the human) controlling the car? The one explicitly designed to trigger emergency brakeing even if the primary controller is trying to do something else? The kind of system installed on many other AI controlled cars to avoid this exact type of disaster?
That backup system?
Look, it's entirely possible that for some reason this specific safety system conflicted with the current version of Uber's system, but we have zero data to suggest that, and if so, it's something specific to those two systems in that particular vehicle (and suggests a severe issue with Uber's system to boot). There's no general issue here.
> There's 2 chefs.
Yes, and sometimes one of them takes over from the other one. That's how this works.
There are plenty of intentional maneuvers required to integrate with real-world (ie. non perfect) traffic that trigger these systems.
They’re designed to take hints from human input (so for example, LKA responds less strongly to forceful unexpected inputs than a lack of expected input).
There is 0 chance the system wouldn’t interfere with a self driving.
Not to mention many of those safety systems include lockouts on the CAN bus which make removing their safety module a first step to even add your own inputs. I know that’s how it works on my Volt with a Mobileye sensor
If your SD system is supposed to be solving 100% of the same problem anyway, it's best to gut the software you don't control and work with the hardware as directly as you can. The notion of building atop a lower-level abstraction makes for great academia but doesn't work with industrial realities.
While yes - it would also force Uber to permanently partner with Volvo on technology they can't control. 'Uhuh, my Ubers a Chevy not a Ford - it could kill me because the Chevies don't built-in X is worse!'.
You do not want an arbitrary third party system controlling the car while the Uber system is in unknown ways; if the systems conflicted things could go out of control and be more dangerous.
That sounds like a win-win to me.
>You do not want an arbitrary third party system controlling the car while the Uber system is in unknown ways; if the systems conflicted things could go out of control and be more dangerous.
I find it interesting that, of these two systems, it is Volvo's widely-deployed system that is described as 'arbitrary'. If this argument had any merit, it would be unsafe for this system to be on the road, 'conflicting' with the car's human driver. On the contrary, the statistics clearly show that these systems reduce the frequency and severity of accidents.
Maybe it makes sense to rely on arbitrary vehicle specific fallbacks, but long-term it doesn't as I'm sure They don't want to be dependent on a specific model. When you request a ride, do you want to care about whether your getting a Honda or a Volvo?
... is called redundancy and is the basis of building mission-critical, safe systems.
Specifically, uncooperative systems running at the same time was exactly how fault tolerance was achieved on the space shuttle[2]:
>For true redundancy to take place, all computers must listen to all traffic on all buses, even though they might be commanding just a few. That way they know about all the data generated in the current phase. They must also be processing that data at the same time the other computers do. If there is a failure [...], since the Shuttle's other three computers are sending apparently correct commands to their actuators, the failed computer's commands are physically out-voted
That's just one example. There's no need for the definition of the word "redundancy" to be as narrow as you said.
[1]https://en.wikipedia.org/wiki/Redundancy_(engineering)
[2]https://history.nasa.gov/computers/Ch4-4.html
Either your eyes or ears may fail you now and again, but you're better off with both.
No?
If one system detects an obstacle, stop the car. Even if this system was failing and the other wasn't detecting anything.
And if one system fail, yes the car should stop if it's not safe to drive with only one of them.
There are trade-offs here, sometimes we might decide to take a very small risk, after weighing it up. A famous example is ETOPS. Unlike propeller engines, jet engines for airliners are ridiculously reliable once running. ETOPS (jokingly "Engines Turn Or Passengers Swim") is a scheme which over the past few decades has gradually allowed twin engine jets (which obviously just become over-engineered gliders if both their engines fail) to fly over oceans and other large distances with no airports, on the rationale that although total engine failure would probably be deadly without anywhere to put the plane down, two unrelated sequential engine failures are now so unlikely as to be overall negligible risk to the passengers.
But that needs a risk we understand. If you have no idea what the risk is, you can't make that trade, so redundantly safe systems are the obvious choice.
In fact, there's yet another different, redundant, potentially conflicting safety system: the human safety driver.
If only one were to trigger, it still wouldn't be a conflict.
And yeah, redundancy through using non-identical systems is very much a thing, and considered much safer than having two identical things (which might make the same mistakes).
Source: I've worked on ASIL D safety-critical automotive systems, where diverse redundant subsystems were actually a design criterion.
It seems like "not running into stuff" is about the only level of coordination necessary here.
A low-level safety system has a much simpler requirement: if it looks like the car is about to hit something, hit the brakes.
And this second category of code is something that already exists, deployed in production today by multiple car manufacturers.
That’s the bit Uber turned off. “It’s a hard problem!” doesn’t seem like a good explanation.
But -- here's the crux of the thing -- it's not a new goal, it's the same goal they (should) already have.
I have no idea how this input would be interpreted by an autonomous driving system. However if I was designing such a system I would be pretty concerned about the potential safety impact. At very least I would want to add code to recognize these special braking events and then I would need to test this code... which of course would be impossible to do in simulation because Volvo’s system is probably closed source and likely exists as a hardware black box. At the end of the day I would probably be more comfortable designing my own collision avoidance system than relying on one that I couldn’t test easily, and that might spit unexpected inputs into my already-complex system.
So no, I don’t see this is a simple question of “adding redundancy”. I think what Uber did here probably makes good sense. The fact that their own collision-avoidance systems didn’t fire is, however, a problem.
Also the collision-detection systems should be getting tested so that they improve too. You want your fall back systems to be more reliable than the regular system. So if you turn off the collision-detection systems never fix the false positives and false negatives then you'll always have these problems.
> At very least I would want to add code to recognize these special braking events and then I would need to test this code... which of course would be impossible to do in simulation
Why? Does Volvos system raise events differently in context? I would imagine it's pretty simple to simulate an event by driving the car into a wall and capturing a dump of the CAN bus.
The autonomous system could take the Volvo system firing as an event like anything else that it needs to reconcile in the system. If anything the autonomous system could be used to improve Volvo's system by capturing extensive data dumps of the events leading up to misfire events.
https://news.ycombinator.com/item?id=16658566
Having them disable a standard feature isn't per se a problem. However, it is an issue if the car is demonstrably less safe with it removed. It's possible that this car was going too fast for this feature to engage anyway (depends on the age of the XC90: if it was post-2014, it should have kicked in).
The Volvo system uses a LIDAR just like the main Uber system. In principle they should have been able to replicate this safety feature pretty much precisely.
The question then arises; if they're making modifications on systems like that, what other modifications have they made? Are the modifications correct?
The issue with the headlamp angle implies there could be other problems. Modern cars are incredibly complex; pulling out single features might be OK when they are add-ons, but the XC90 ships with a lot of this stuff in every model.
I think this case clearly demonstrates otherwise. Think of the factory fitted as fail-safe - if their own autonomous tech fails(like it did here) you have the one last system that will prevent collision(and it looks almost certain that it would have done that here).
How is a "self"-driving solution allowed to drive in real traffic when it can't break in a simple situation where relatively simple, existing systems can already reliably break.
Honestly, this is a cop-out. A substantial amount of information is available from the vehicle's CAN networks - including speed and braking action!
If the autonomous driving system detects sudden and unplanned changes in speed and/or braking (perhaps caused either by the driver, an external safety system or some kind of impact with something on the road) then the obvious choice there is for the autonomous driving system to release control of the vehicle and let whatever existing systems in the car, or even the driver, handle what is happening.
The car's built-in safety systems are a) simpler, b) less likely to malfunction and c) substantially better road-tested. There was no reason whatsoever to disable them.
This way there won't be 2 breaking systems competing with one-another. Different systems controlling the same end-points is never a good idea.
In a modern car there are plenty of systems that independently will start. An example is some cars that are fixed to prepare your breaks if you take your foot off the accelerator quickly, more or less giving you instant breaking. At the same time, completely independent of this, emergency breaking might already have been started by the collision detection system.
Aerospace engineer here. Different systems controlling the same end-points is literally a basic engineering safety technique. Not only you have the human pilot as a competing system controlling the same end-points as the autopilot, you may have even different competing autopilots.
I had a professor at the Uni who was part of the engineering team of the flight computer of a certain jet fighter. Two engineering teams were given the specs of the system and told to produce a flight computer. The two teams weren't allowed to talk to each other to prevent bugs cross-pollinating the two systems. After all, what good is to have two identical system for redundancy if they can be affected by the same bug? (That's also why pilots eat different meals in-flight!)
Then, the two completely different systems are installed, duplicated, into the plane and put to the test. The four computers must perform equally as per the specs. If the recommended output of a single computer diverges from the other three, that computer is faulty and disregarded. If two computers of the same design have a bug and start giving out-of-envelope readings but the other two are consistent, control is handled to them.
I'm certainly very appalled to see in this thread how safety engineering techniques that every engineer learns in University is completely disregarded or even reversed. At least in the aerospace community, development and implementation of these safety techniques has been paid in blood. These are lessons that shouldn't be forgotten when developing other transport technologies IMHO.
Aerospace (and medical) is where things are much better.
Exactly the same deal if there's an AI driver.
Automatic Emergency Braking (AEB), which is the system that could have prevented this accident, is not an automated driving function though; it is an active safety function. As such, it is more common to stack such functions. For instance, ABS > AEB > automated driving, with the left most being the function with the highest priority, i.e., it can override the functions to the right. This is how most safety systems in cars work in production.
That's the exact argument I make for never buying a car with that technology because I'm driving it and I have no way to communicate with it...
I'd be super happy to be driven around in an autonomous car, but have zero interest in auto-lane-keeping-pilot-collision avoidance-parking etc
The system kicks in when there is a high probability of a collision ahead. Think of it as the instructor in the driving school that slams the brakes when you're doing something wrong.
So basically, the driver (be either human or another system) did something wrong already if that system kicks in
No. This is Volvo pressuring Aptiv to make a statement on its behalf, to avoid being tainted by this PR disaster and to shift investigators' focus away from both companies.
Also, in all of the discussions about this incident, let's forget that Elaine Herzberg, a mother of two children, lost her life due to an engineering experiment and publicity stunt ("Come to Arizona! We don't have safety regs!") being conducted on public roads with great risk to members of the public. Volvo, Aptiv, and other participants in these trials should have never signed off ... yet they willingly participated, and an innocent member of the public has paid with her life.
This particular location has the sidewalk end on one side, and there is a paved section in the median which looks like it is intended for crossing. There is a small sign that indicates that you should not cross here.
From the video, we can see that while there were street lights on the road, the area in which the apparent crossing existed was completely dark. A pedestrian may not have been able to see the sign indicating that you shouldn't cross here; or even if she did, there are so many roads that are built in pedestrian unfriendly ways that sometimes you just have to cross at unmarked crossings to avoid having to go substantially out of your way.
Additionally, the video shows that the driver was distracted immediately before the crash. While the dash cam video seems to indicate that you can't see the pedestrian until it was probably already too late, human eyes are a lot more sensitive over a greater dynamic range than most video cameras, so there's a good chance that if the driver were paying attention, she would have seen the pedestrian before the car did.
The thing about safety systems and standards are that there should be some redundancy in them. People make mistakes, so you shouldn't have to rely on every actor acting perfectly to remain safe. In this case, there was a failure on many parts leading to a tragedy.
It's definitely worth investigating whether Uber had done enough to ensure that operators will be staying alert during the entire test, and whether Uber had done enough testing before being on open roads of small, possibly moving obstacles on dark parts of the road, like pedestrians, deer, and so on.
I'd also put a good amount of blame on whatever department designed that portion of road and sidewalk.
https://www.youtube.com/watch?v=WbW2UgmjJUA
IIRC it was discovered this happened because the owner hadn't purchased the "Pedestrian Avoidance Package".
And shortly before that, a Volvo plowed into a parked truck (in a demo):
https://youtu.be/aNi17YLnZpg
This was "was the result of an earlier flat battery that temporarily disabled the system due to low voltage" [0]
The pattern here is not only due to the collision systems themselves, but to the system around the collision avoidance system.
[0] https://jalopnik.com/5648126/volvo-pedestrian-avoidance-cras...
I'll sound harsh, but to be honest it seems that Elaine Herzberg's behaviour also played a role in this drama.
The video is a perfect example of jaywalking.
Dressed in black, crossing the street at night in a dark place, no lights, not reflective jacket or zebra crossing area, not paying enough attention to traffic. In the video she's not even looking at the car. There is not signs of the woman stopping or changing their trajectory to avoid the crash.
The point is that Uber should only be disabling these safety features on public roads if their technology is superior.
If their technology is less safe, and in this case it appears to be that it is, they shouldn't be driving around on public roads. Keep it to test tracks.
Volvo's system would have either stopped or slowed enough to make this crash non-fatal. That's worth pointing out. There is no point in Uber testing technology that is worse than what you can buy any number of cars today.
Personally I think it's naive and unreasonable to think we're going to get totally safe and rock solid self-driving tech without at least a few coffins worth of people who gave their lives to show us how it could go wrong. There isn't one safety system in existence that wasn't built on top of a pile of corpses.
It should not be too hard to make these systems co-exist. If one of the systems says break then the car will break. The few times the car breaks when it really didn't need to should be an acceptable side-effect during testing.
If you had a firewall in your network, for example, that had both a firewall system that you configured, and alongside it, a firewall configured by a third party with "best practices" in mind, that would be frustrating to configure and test because you'd never know for sure if it was a configuration you didn't setup correctly, or if it was this other firewall that you couldn't operate interfering with whatever it was you were doing.
When designing things like this, step 1 is you remove variables that could change the behavior of the device in question without you knowing why it did. I'm sure once completed, a self driving car system could well be made to interface with the safety systems otherwise present, or could be designed in such a way where they wouldn't conflict. When you're in the thick of building that system, however, an additional (and by your comment, not exactly predictable) system to contend with, that you have no access to, is going to be a pain in the rear.
Again, the stock auto break systems rarely engage unless there is real danger. By that I mean last second breaking, absolute last second. Most people can drive a lifetime without ever having them engaged. If you cant program a self driving car that can handle that then you're in the wrong business.
I am very skeptical of your claim that this is a less safe way of doing things. There is, in fact, communication between the two systems: the safety system 'says' to the driving system, "no you don't", and that should not be up for negotiation any more than it is when a human is driving. As was pointed out in the other discussion, if the driving system cannot find a way to work within the restrictions of the safety systems, it is doing it wrong.
[1] https://news.ycombinator.com/item?id=16681611
Like we have reflexes that we don't have full control over, that often don't even interact with our brain but rather are initiated from the spinal cord, which I would classify as dumb in this instance. And layered on top are emotional responses, memories and finally consiousness as, arguably, the control layer.
Uber themselves should have a dumber system built underneath the smarter system (and maybe they do!); that way they understand all the variables that govern the operation of both and can be reasonably sure that the two systems won't fight each other. They can also build an interlock that causes the smarter system to completely disengage if the dumber system has to take over. I of course don't know this for a fact, but I wouldn't be surprised if Uber doesn't and can't have a complete understanding of how the Volvo system works. Disabling it makes sense. Again, they should replace it with their own dumber system, but that's a separate issue.
If I had a Volvo XC90, I would want to operate it with two systems: the collision avoidance features plus an advanced system of muscles, eyeballs and brain. The whole point of collision avoidance is to operate in conjunction with a driver. Self-driving should be more or less just another driver.
Like the Intel statement from article. Hope there will be some sort of requirements for this equipment defined. Or technically benchmarked before allowed on road.
> Intel Corp.’s Mobileye, which makes chips and sensors used in collision-avoidance systems and is a supplier to Aptiv, said Monday that it tested its own software after the crash by playing a video of the Uber incident on a television monitor. Mobileye said it was able to detect Herzberg one second before impact in its internal tests, despite the poor second-hand quality of the video relative to a direct connection to cameras equipped to the car.
https://www.google.com/search?q=mobileye+sends+tesla+a+lette...
This came about because Mobileye was honest about their capabilities: https://electrek.co/2016/07/01/tesla-autopilot-mobileye-fata...
Tesla, in their immediate response (same article), basically tried to imply they have magic abilities to do much better, which, based on later crashes, we know is not true.
Mobileye, in response, chose to end their partnership with Tesla (and yes, it was Mobileye who did it. The letter they sent ending the contract can be found if you browse around, after Tesla explicitly asked them to continue and signed a multi-year contract earlier in 2016)
But compare with the other discussion currently on the front page, where users point out system subsumption is a basic safety principle. You need a safety system to fall back on if the AI fails (as it will, at least at this stage). It seems grossly negligent if Uber didn’t re-implement collision avoidance at a lower level.
Edit: link to discussion I’m thinking of https://news.ycombinator.com/item?id=16681611
Getting information that some automated system initiated some operation is almost certainly harder. System only knows that the car is not behaving as expected, but it doesn't know what caused it. You could probably figure out how to determine if something like collision avoidance was triggered (large shifts in expected vs actual outcomes), but in case of something like ABS you could end up with two systems fighting for the control over brakes in emergency situation.
Compared to all the other aspects of making a self-driving car, which are genuinely very, very hard and require groundbreaking technology, it seems incredibly lazy and negligent not to get this bit right.
If the switch state changes from closed to open, [driver|AEB] have intervened and Uber's system should relinquish control of that sub-system.
No, both cases are fundamentally different. If a human driver has intervened, the system should expect the human wants to take over all subsystems: braking, steering, accelerating. If the AEB has intervened, the system should expect it will take over only the braking, and only for a limited amount of time; it shouldn't relinquish control (and doesn't have to, since a properly-designed AEB will be an override of the braking control), and should be ready to take over immediately after it releases (and probably bring the car to a safe stop as soon as it has control back).
That is, the self-driving system must know whether the command came from the human driver, or from another system.
I'm not convinced the two should be linked in any way, human driver or AEB or whatever else. If the human is taking evasive action, the auto-drive is getting input from all the available sensors.
This shouldn't be that hard (relative to the other challenges being faced).
The auto braking system doesn't do steering, nor indicators. So, you can't treat those the same.
Low visibility on a clear day is simply lack of illumination, and it is a horrible excuse, because:
1. The car is equipped with headlights, which solve the "limited night time visibility" problem;
2. The car is equipped with multiple types of high-quality cameras;
3. The car is equipped with a LIDAR, which is an active sensing system.
Finally, if one accepts that the visibility was limited to 50 feet, as the video misleadingly suggests, THEN one has to concede that the car was going too fast for how much it could see.
People are taught to drive so that at least three seconds worth of road ahead can be seen. If you believe that either the car or the driver were seeing the equivalent of what the webcam footage was showing, the car was going unsafely fast.
Hanlon's razor would suggest, though, that the car simply wasn't able to process the data available to it. We'll know for sure if the investigation is successful.
Simply taking a crappy dashcam video with low dynamic range suffices; it will make an illusion that anything not directly in the headlights of the vehicle is invisible.
I don't know how people can believe a single word of the early official statements which claimed that the victim "suddenly" appeared in front of the vehicle. The problem of "it's too dark to drive" has been solved a hundred years ago with the invention of headlights.
Here's an experiment anyone can try: turn on the headlights on a dark road, and see if you really see nothing in the lane to your left (driver side). Whatever you do see (which, under normal circumstances would be at least three seconds worth of road ahead) is the way in which the released video is misleading.
The speed limit is the max limit, conditions permitting.
Then a lot more testing data could be gathered by outfitting random vehicles (taxis, etc) with the tech, and analyze (and further refine the software) around every event where the software and human driver differed in opinion (i.e., whenever the vehicle abruptly changes speed, did the software detect that it should have hit the brakes at / before the same time).
Volvo is the gentleman here, with its “can’t speculate on the cause of the incident” statement.
Both Volvo and Uber could have opted to use this car instead. Even choosing a SUV is question-worthy, as they're known to be more dangerous for pedestrians due to higher bumper heights: https://en.wikipedia.org/wiki/Criticism_of_sport_utility_veh...
IMO there is at least some negligence on their part for not choosing a car that is more likely to protect pedestrians.
The V40 is also a rather old car and is based on an old platform. The XC90 on the other hand was their newest car at the time the Uber deal was made and is based on their latest platform. So it is not unusual that both Uber and Volvo would prefer the XC90 over the V40. Besides, given the newer technology in the XC90 it is quite possible it is better in pedestrian accidents than the V40 (with the safety systems enabled on both).
If you have an automatic but a brutal, on-off style emergency-braking module based on, for example, lidar then it's much easier to develop another, maybe less aggressive but much smarter, auto-braking module based on the regular camera inputs. You can count on the failsafe system to act if higher-level modules fail. With braking in particular, maybe even a couple of redundant low-level emergency brake systems would be a good idea before you even consider any of the more high-level systems. You really don't want to hit jaywalking pedestrians or wandering drunks because both kinds do exist in reality.
Similarly, once the car has a couple of obstacle-detection and auto-braking systems up and running, it's easier to work on autonomous driving because you can be confident that your computer won't be able to drive the car into anything as these isolated and separate systems will take care to stop the car.
Consequently, when you're developing the navigation and routing system it has to be able to rely on the driving functions that deals with the present-time traffic situations, like keeping up with the flow of traffic, slowing down, changing lanes, etc. Again, same thing as with emergency brakes: to develop a smarter higher-level system you need to have lower-level failsafes in place.
In the case of autonomous driving a failure should mean a harsh but unnecessary full stop or the inability to continue, not a collision or the inability to stop. They should definitely have left Volvo's own system on as an additional failsafe mechanism.
But it also seems like "It works better than the stock production autonomous braking" would be a gating factor in putting these vehicles on the road.