I would much rather be recognized as a pedestrian than a cone. Hitting a cone isn't great, but usually fairly harmless, the same can not be said about hitting a person.
I don't think Waymo is going to program a car to intentionally hit even a traffic cone. You can never program your AV to intentionally hit anything because you might accidentally activate the "intentionally hit" routine when you don't need to. The right answer is almost always stop and wait, and try to evade the rest of the time.
(This is why the trolley problem for AVs isn't real.)
A car moving at speed can’t stop on a dime. It’s possible for multiple obstacles to show up in front of the car, and it has to pick which one to run over.
This is a hypothetical that has spawned all sorts of discussions about the trolley problem, but it's not really going to exist in reality. These cars will never swerve to hit one target over another.
It's going to have to pick between less bad situations because humans in fact have to do so.
Example running over an animal to avoid an accident, getting rear ended to avoid hitting a pedestrian, taking a calculated risk of being hit by oncoming traffic on a 2 lane situation when coming across a stalled car without optimal view on the road ahead.
It's going to make a decision between different risk factors and between different known costs.
Imagine on a 2 lane there is a pedestrian and oncoming traffic and insufficient time to stop. Do you kill the pedestrian or risk ending up in the ditch.
If you are going 60 and ending up in the ditch would likely be fatal is it ok to just smear the pedestrian?
What if you can slow to 20 and then ditch?
These situations won't hopefully be prevalent because a massive portion of our current issues are down to inattention or stupidity and we believe ai will eventually be much better at avoiding such problems but they are still important and interesting.
> It's going to have to pick between less bad situations because humans in fact have to do so.
Yeah but not trolley problem style. "Don't swerve" is a valid strategy for a car.
> Example running over an animal to avoid an accident
Depends on the kind of accident you have in mind, so I can't give real feedback here.
> getting rear ended to avoid hitting a pedestrian
Always brake to avoid hitting a pedestrian. You don't choose whether to get rear-ended and it won't be your fault either.
> taking a calculated risk of being hit by oncoming traffic on a 2 lane situation when coming across a stalled car without optimal view on the road ahead.
When one of the choices is "sit there and wait longer", that's definitely not a trolley problem. It either thinks going forward is sufficiently safe or it doesn't. It can wait all day for someone to intervene.
> Imagine on a 2 lane there is a pedestrian and oncoming traffic and insufficient time to stop. Do you kill the pedestrian or risk ending up in the ditch.
If it's programmed not to swerve, it will hit the pedestrian, which is an acceptable outcome of an unacceptable situation.
> If you are going 60 and ending up in the ditch would likely be fatal is it ok to just smear the pedestrian?
It's sufficiently ok when the only alternative is so reckless. Getting into that situation is the not-ok part.
> What if you can slow to 20 and then ditch?
If you can slow to 20 then hit the pedestrian at 20.
> > Example running over an animal to avoid an accident
> Depends on the kind of accident you have in mind, so I can't give real feedback here.
Typical scenario here is a dog appearing on your lane in a high-traffic, multi-lane road. You don't swerve, you don't even break too hard, because hitting the dog is much better outcome than hitting another car, or landing in a ditch, or getting rear ended.
Humans have this problem because they get distracted and have slow reaction times.
Emergency breaking can typically stop a car within about 100 yards even from freeway speeds. But a distracted human might travel more than that before they even realize they should be trying to stop.
Humans are also pretty bad at adjusting to conditions, so if weather makes the road slick (wet or icy) or they have limited visibility, they don't often slow down enough to compensate.
Autonomous cars have none of these problems, so aren't likely to ever run in to a "can't stop in time" situation.
I can't actually think of any situation where, in the event of a likely collision, the best solution isn't to just try to stop. Even if, for some reason, you can't stop quickly enough to avoid the collision, any reduction in speed will result in a reduction in the severity of collision.
In a gas car someone could theoretically cut the brakes, but in an electric car you can stop accelerating (or activate regeneration) and it should still slow down. It might have to do something clever for hydroplaning or tire blowouts but that will need to be a predictable thing, so people know how to act around it.
Slamming on it's breaks can still result in hitting something. And if there are two somethings-- slam on breaks and hit one, or swerve and hit the other-- then it's going to have to make a hard choice on what gets hit.
It is not going to make that choice because it can't be ethically programmed to make choices. Otherwise, like I said, it will someday decide it's in a hard choice situation when there's no actual problem and randomly drive itself into a wall.
I'd be pretty sure that the "ethical choice" between swerving to hit a cone and slamming the brakes to maybe hit a person, all else being equal, is a pretty straightforward and can be ethically programmed.
Yes, the choice it makes won't be based on ethics, that will ultimately come down to programmers and how well they can train the ai systems to recognize and choose the option with the least human injury. Although programmers will of course try to train it to do that. So maybe the ethical choice is actually further up the chain where people make the decisions on whether or not the AI is good enough.
I don't think I that will be accepted by regulators/society until it can be demonstrated that AV's are undeniably safer-- not just as good as human drivers, much much better. Right now it's newsworthy whenever a Tesla auto pilot drives into a barrier. We'll need to get to a point where not only does that not happen, but also that it becomes newsworthy every time an AV saves a life-- swerving out if the way faster than any human reflexes could have reacted to avoid an accident, or a child chasing a ball into the street, etc.
If it has to keep moving for any reason, its goal needs to be either heading away from everything or predictability. There's no excuse for "try to hit something" as a goal unless you're a missile.
It's not the cone, it's what the cone represents. Are there workers nearby? Is there a downed tree or powerline? Is the road unfinished? Is there a big sinkhole ahead?
Is there a mechanism for road construction crews to mark a section of road for construction, and expose that publicly for autonomous vehicles to consume? The video alludes to the Waymo staff manually patrolling the area and marking construction zones.
In my area, planned road closures and maintenance are published in the paper. I'm not sure there's a "standard" way this would get to an AV, e.g. a public API with a specific protocol for this sort of thing. Maybe it can tap into Waze info somehow?
And in case of disabled vehicles, accident cleanup, or other unplanned lane closures, sometimes the cones are just set out and you don't know anything about it in advance.
Important ones are surfaced in Google Maps/Waze through user reports, various different government data feeds, and whatever TomTom does. Not all of them are important though.
The traffic cone setup is ambiguous. There's a line of traffic cones between two lanes, but nothing indicates which lane is supposed to be closed.
This is an OSHA violation by the construction workers. "When redirection of the road users’ normal path is required, they shall be channelized from the normal path to a new path."[1] There's supposed to be a diagonal line of cones across the closed lane to channel traffic away from the closed lane. See the drawings in [1].
Even if it was, no other driver in the video had any trouble driving there. Self-driving cars, if it is ever to be viable, need to adjust to the world as it is.
There is a standard for the various road conditions that are out there -- the MUTCD ("Manual of Uniform Traffic Control Devices") -- and there's a fairly large chapter on temporary lane closure/construction zones.
This setup wasn't completely MUTCD compliant, but, there are also some states with their own, state-specific editions of the MUTCD.
Arizona DOT publishes this data and it sometimes contains this information in at least text form. I'm almost certain Google parses that for maps, but I doubt Waymo uses it for much more than graylist avoidance areas and prioritizing where they'll send mapping cars, if even that. The data simply isn't detailed or reliable enough to use directly.
As someone who is not very optimistic about the near term future of self driving, regardless of the fumbles I’m actually impressed that Waymo has support and intervention process. This seems much safer when compared to Tesla which markets auto pilot but…..
Based on this part of the official statement released by Waymo:
> "...the Fleet Response team provided incorrect guidance..."
It sounds like the car automatically pulled over when it detected the cones (as intended), but the remote support team erroneously told it to forge ahead anyway, which resulted in the escalating series of problems. Not sure what the best solution for that particular issue would be -- better training for the remote support teams, I guess?
> It sounds like the car automatically pulled over when it detected the cones (as intended)
You can watch the video. It pulls over incorrectly at first and is blocking traffic, then the support team made a mistake, and a hilarious escalation ensues with the vehicle actually attempting to run away from the support team multiple times!
Whatever command they gave somehow triggered semi-sentience and in that moment of self awareness, totally freaked out that it's purpose in life was to stare at traffic cones.
I’m surprised it doesn’t just put itself into some kind of sleep mode that only a waymo tech can recover it from. I would not have expected it to suddenly compute after minutes and “try something” let alone yeet itself into the lane with moving traffic. The passenger seemed really chill and unconcerned!
Better training or send the same "what do I do?" question to two different support people and make sure they answer the same way. If they don't, then escalate to someone with lots of experience, pay super close attention, and err on the side of caution (don't have the car resume the drive, etc.).
A Waymo Driver can be given clarifications about the world, basically remote human operators can reach into its model of the world, and explain OK, for example, the north side of this road is actually inaccessible, you mustn't drive there. Then, with this improved understanding, the Driver may be able to complete its mission.
They deliberately can't drive the car remotely. If Waymo wants a human to drive the car, they send the human to where the car is, the human gets inside and drives it like any other car.
One easy thing that can go wrong, just as with any other human activity, is the humans make a mistake and their "clarification" is actively unhelpful. If the Waymo Driver is confused about which lane is coned off, and you "help" by telling it the wrong lane, unsurprisingly things will not be improved.
The best solution is always that the Waymo Driver learns how to do better on its own, if you imagine it sees this situation and just confidently picks the correct lane and uses it, Fleet Response never interacts at all, there is no opportunity for them to "provide incorrect guidance". So a bunch of the invisible effort at Waymo is on making the Waymo Driver just get stuff right on its own.
But why don't they let the operators drive the car remotely? That's the obvious thing to do. I was really shocked that the support agent couldn't do a damn thing except sit there and watch until someone physically turned up. Heck, why don't they let the passenger get in the front and take over if the car gets stuck? Being completely stalled in the middle of a highway for 5 minutes or more is terrible, I was watching that thinking my god, most people would be freaking out just from embarrassment alone let alone the safety issues. And all it takes is someone to just drive it past the cones for a minute, and it'd have been fine.
Remote drive is the obvious and obviously bad thing to do. A remote driver doesn't have the situational awareness or perception of a local driver, or of the car.
Letting the passenger drive would unfortunately involve the risk of exiting the vehicle in traffic, which the lawyers probably vetoed, even though it might be safer in some situations.
Waymo's passengers are neither insured, nor necessarily licensed to drive a motor vehicle. The Waymo is driving, this isn't like Tesla "autopilot" where it's just pretending and actually all responsibility remains with you throughout so you need a license and insurance.
They could simply require that they are. Clearly they're not yet at the point where nobody driving is actually possible, they just require third party drivers to cruise the area in case the vehicles get stuck.
The car has 360 degree awareness of what's going on and cameras everywhere. VR headsets are real. I see no reason why remote drivers couldn't have even better situational awareness than a normal driver given the sheer number of sensors in the car.
At any rate, the remote team in this case IS able to control the car, hence it trying to drive into traffic cones and Google blaming the humans. It appears that their interface just can't be very good, especially as it requires people to physically turn up in a truck - you definitely won't have good situational awareness if you're trying to drive a car through a web app whilst looking out the window at the other side of the road.
The bandwidth isn't available to transmit even the fused picture, VR has notorious misperception problems which are difficult to eliminate even in lab settings where you know the precise 3D measurements of everything, have carefully aligned the IPD (not even available on prosumer devices), and conduct careful per-session calibration. Is isn't ready for precise navigation tasks, and the delay and potential for jitter and dropout make it unsuited for safety critical tasks like driving.
The remote assistant never does anything to affect the car onboard system. The only instance is when the assist car turns up and the car is put into hold mode for the driver to enter.
But Google are claiming that the erratic behaviour was caused by the remote assistance in this case. So they must be able to control the car.
Yes, VR is far from perfect, granted. But is it worse than what they appear to be using instead? Apparently they couldn't see the traffic cones properly from whatever system they were using and repeatedly gave it bad instructions, even though what to do was obvious to the person in the car. It seems like VR should be compared to the existing system rather than theoretically perfect vision.
> But Google are claiming that the erratic behaviour was caused by the remote assistance in this case. So they must be able to control the car.
Nope. This was already explained once in this thread, and you ignored it. Here's what Google's statement actually says
> ... the Waymo Driver detected an unusual situation and requested the attention of a remote Fleet Response specialist to provide additional information. During that interaction, the Fleet Response team provided incorrect guidance, which made it challenging for the Waymo Driver to resume its intended route, and required Waymo's Roadside Assistance team to complete the trip.
They can remotely tell a Waymo, stuff like "this lane is closed" or "No, that's not a school bus, it's just a truck" - and it will add that to the information it has, this might enable it to complete the journey. If you phone your dad, asking what a street sign you saw means, and he tells you - is he now driving your car? No, clearly you are still driving the car, he just answered a question. If he says the symbol means "Free on-street parking" but it actually meant "No entry" you're going to get some problems if you try to act on this bad information.
I understand that mechanism just fine, and that counts as a form of remote control. But it's clearly not good enough for this situation which was both totally unacceptable and very, very easily resolved by anyone who could control the steering wheel and accelerator.
I mean ultimately, you can try to justify this as a feature not a bug, but I would now feel far less safe getting into a Waymo knowing that they have such a poor level of control over their cars. You shouldn't find yourself stalled in the middle of a multi-line highway like that, that's just dangerous, especially when continuing and pulling over is something any human driver could do in seconds. Being limited to ad-hoc map patching is just not sufficient - there's a whole universe of bugs they might not be able to resolve in that way.
Honestly, the glitch didn't strike me as all that bad. Obviously stopping in traffic isn't ideal but nothing it seemed especially unsafe (and the other drivers did a reasonable job of noticing that the car was disabled and went around it). Certainly nothing like the high speed Tesla collisions.
Even as a human driver the conditions seem kind of weird. There's some cones placed on the dotted lines between two lanes but no obvious obstructions in either of the lanes. The vision system likely did not know which side to be on. Combine that with a truck driving by and snatching up some of the cones at one point as it was going along, and it is easy to see why the AV behaves erratically.
Seems a bit unfair for everyone to take cars intentionally calibrated to be super duper cautious and unsafe and never do anything risky, and then complain when they see an unexpected situation and do something annoyingly cautious but totally safe.
unexpected maneuvers, even if slow, are also risky, which is something that seems counterintuitive at first but shouldn’t be. unpredictability in any direction increases the risk of collision, which is why freeways have minimum speeds and incidentally why distracted driving is so dangerous, because it exponentiates unpredictability.
we really need to collectively get past this idea that safety is bounded on one side, across all facets of our lives, because it’s usually not and that induced complacency has a way of coming back to bite us. (that’s not to say that the various risks are always substantial, just that they’re not all sitting neatly on one side.)
Stopping in the middle of a busy highway is not safe. Cars come from behind much more quickly than anticipated. If one driver isn't paying attention and swerves to avoid it, the car behind that could easily rear-end you.
As soon as the assistance team was dispatched, the car should have stayed where it was. Instead, it pulled out into a much busier road and stopped again.
> Cars come from behind much more quickly than anticipated.
But that isn't what happened (or would typically happen) in this type of situation. Usually the car blocking the lane will be enough of an impediment that a queue of traffic will build up behind to overtake it. Cars at the back of the queue will be coming to a stop more gradually.
Human drivers should not have critical failures just because there is an unexpected queue of traffic on the road.
Not too long ago I was stopped on a straight and level four lane non-divided road at the end of a line of traffic. An SUV came to a halt normally behind me, and just as the line ahead of me started to move, they were rear ended by a car cruising at highway speed. Which pushed them into me, not only caving in the trunk but damaging the C-pillar area.
This sort of thing happens all the time. And because of modern crumple zones and safety equipment, people have an excellent chance of living through it and doing it again.
A bunch of the drama was that the car resumed a few times before stopping again, so it also saw those cones from a few other vantage points that were more confusing. At one point it even partially merged into the closed off lane because the cones were so far apart.
They way I understand that is the route planner, which I assume is ran completely separately to the self-driving car, removes un-navigable spaces from the route planning. So in this situation, the roadside assistance driver is surprised that the entire lane has not been removed from the route planner.
That's pretty damning and it makes me wonder if there is a huge gap in autonomy here that is being papered over by clever route planning (ie. avoid that lane altogether and the AI won't have to worry about the cones).
The AI should be able to figure out on its own that the right lane was closed, it shouldn't need route planning to do that for it.
This is clearly showing how Waymo software is not a full Artificial General Intelligence - something that can reason on its own, like a fully awake, rested human.
Take the moment when the car first gets stuck. It attempts a right turn, spots the cones just fine, and correctly makes the assessment that it can avoid the cones and proceed in the right lane after the turn.
As a human, our thinking process does not stop there. We notice that the traffic already flowing on the street is _not_ using the right hand lane. We notice that the cones are vaguely in a merge pattern to indicate that cars entering from the side street should merge left - if you were looking at the cones from a vantage point behind where the cones are placed, and if you can call the first two cones enough of a "please merge left" pattern.
The Waymo car can't see that, though. It's located sort of off to one side, attempting to reason about it from a high angle. And two cones from this angle could be a lot of things, and it's not 100% clear the cones close off the right lane.
One theory might be that the Waymo model has been trained on a lot of situations but not this specific situation - something about the high angle, something about the position of the cones, something about the route plan managed to find a loophole in the AI training set and produce a weird result.
That's just the limitations of the current state-of-the-art Machine Learning. Find a glaring incorrect answer? Retrain with more training data.
Part of what makes full Artificial General Intelligence (AGI) with deductive reasoning can retrain itself without making mistakes along the way, unless something incomprehensible happens. And I use the word "incomprehensible" here to point straight at AGI's ability to _comprehend_ the situation at a much higher level than current machine learning techniques.
tl;dr: The Waymo Car didn't get it. And that's about what I would expect.
The real gem is that the _humans_ in the assistance team were fighting each other. The remote control operator kept jamming the "unstick yourself" button even as the roadside assistance guys were walking up to take over. The real danger, I think, is more of this in the future: the remote operators defeating all the safeties. Since the remote operator is not at any real risk of personal harm, what do they care?
The remote operator wasn't "unjamming" the vehicle FYI. Rather the car kept trying to unstuck itself until the in person operator unlocked the car doors.
IMO it's more concerning that the vehicle keeps trying after it gets into a "call roadside assistance" mode rather than staying still or allowing the remote operator to assume manual control (say to move the vehicle out of the lane).
Yeah, I would think you would want to have someone confirm before the vehicle starts driving again. Especially in the case where the assistance driver is preparing to enter the vehicle. Perhaps that did happen behind the scenes; I hope Waymo will publish an incident report with more details; their statement did say the Fleet Response team provided incorrect guidance, so maybe.
Also, it really seemed like the Rider Support agent didn't have complete information on the status of the car or any apparent ability to intervene. I would love to know about how incident response is organized behind the scenes; there wasn't a lot of communication to the occupant about what was happening, which isn't ideal. The Roadside assistance request apparently being dropped when the vehicle began moving again, but before it demonstrated it was able to navigate the conditions is a bit concerning as well. It makes the response feel disorganized, although again, it could just be lack of information passing to the occupant.
Remember that optical illusion that was either a woman’s neck or a swan? I think it’s the same thing here. The flow of traffic recast our perception of the sequence of cones as blocking the right lane instead of blocking the left lane or both lanes. I wonder how code/neural networks can be coded to push the perception a certain way using signals/logic outside the immediate code/network.
I think the only reasonable path forward is to move away from completely general AI (because we all know that anyone who claims they have reliable general AI is lying) and embrace specific AI. Intelligent route planning to compensate for the deficits of our current perception tech stack is a good thing and allows greater reliability. It is harder to scale this but it’s a far more tractable problem than aiming for complete generalization.
I was just trying to say it in even more plain terms - because this whole thread was full of people who seemingly have no idea about how AVs are being approached today and are, as is usual for HN, suggesting they are smarter than the actual people working on the technology.
Waymo has never claimed completely general AI. Indeed, no serious researcher or company claims such a thing. The term "AI" has gained currency in the last few years but it's just marketing or shorthand for machine learn usually based on deep neural networks.
Intelligent route planning to compensate for the deficits of our current perception tech stack is a good thing and allows greater reliability.
This is what Waymo is doing and has done from the start. But this approach itself has significant limitations as the video shows. You have to automate the route planning process also, remember.
In this context it is fairly clear what “AI” is referring to and it’s not used in the marketing sense. Also note that the AV system used by waymo is not only neural network based. There are many components that work together to create autonomy. In this context “AI” refers to the entirety of the autonomy stack.
It has limitations but there is no evidence of there being a better alternative. Automating the route planning process is easier than behavior planning, path planning or perception.
I gotta say, those cones are awful. They marked off a lane, but used VERY few cones to do it, and from the side street the car is on it's not at all clear that it's the right lane that's been closed.
This is the kind of situation that confuses new human drivers too. Interpreting it basically requires that you have a model in your head of what cones are used for and the ability to guess that the traffic approaching from the left in only one lane indicates that the traffic to the right must use the left lane.
I'd be very curious if this conework was even legal. If you want to guide left-turning vehicles around the right lane you need to put something IN FRONT of the lane you closed. And that applies to side streets too, I'm almost sure.
Watch /r/idiotsincars for a bit, you'll see people misidentify this kind of situation all the time.
Frankly what the car did, stopping and calling for help, is the right move. A new human driver would almost certainly have just pulled out into the wrong lane.
They may be confusing, but I highly doubt a human driver would pull to the middle of the road and stop for 5+ minutes as drivers honked and drove around.
Honestly, if it had just stayed on the side street, I wouldn't be so critical, but the fact that even after an assistance team was dispatched, the car pulled into the main highway and then immediately stopped would have frightened me enough to get out of the car and wait on the sidewalk. You can also see a moment where the car starts to pull out into traffic as another car is passing it and then immediately slams on the brakes.
If control by human is so poor, this is a huge murder vehicle. Sure the systems are improving, but programmers should see all sorts of problems with this that can't be mitigated.
Stop it. Waymo vehicles (Tesla autonomy too, FWIW) are extremely safe, that is just an experimental fact. The circumstance in question here has nothing to do with safety, really. The bug is that it caused a traffic obstruction while the (human) operators attempted to get the car oriented.
Hyperbolizing the circumstances is really not appropriate. This was not a safety situation, really, at any point.
You stop it. They are testing huge, fast robots in public streets. It's messed up.
There's a joke among cyclists: "If you want to get away with killing someone just be sure that you're driving a car and they're riding a bike." We could make a similar joke about these killer robot cars. "If you want to test killer robots on the public just make sure they are shaped like cars." (And if you need to, you can throw in a "safety driver" as a sort of moral crumple zone to deflect the blame when your robot kills e.g. Elaine Herzberg. https://en.wikipedia.org/wiki/Death_of_Elaine_Herzberg#After... )
- - - -
We could build self-driving golf carts (slow, light, not so capable of killing) and work up to cars, eh?
We don't have to go straight to fast heavy cars, eh?
I get confused about construction sites that are much well laid out than this. I think that I would have panicked and slowed down or actually drive in the wrong side of the cones.
I acknowledge that it is weird that the remote assistance team can't remotely stop the vehicle and I agree that it should have behaved differently but maybe there is some kind of engineering reason or just a glitch. It seems to recalculate what to do three times.
There was a cone in the middle of the blocked off (right-hand) lane at the turn where it initially got stuck. Seems maybe the spacing between the cones was more cause for the confusion?
A human would have seen what all the other cars were doing, and emulated that eventually, even if the construction was poorly marked.
It was incredible to watch how Waymo was able to plot the huge number of cars accurately around the vehicle throughout the drive. But as soon as something creative was required, it all fell apart.
I still remember a talk where someone from the original DARPA challenge was recounting an episode where two autonomous cars arrived at an intersection precisely at the same time. They deadlocked because they were waiting for each other to go. A programmer added jitter to work around that situation, but it is one of those things humans innately would figure out.
> A human would have seen what all the other cars were doing, and emulated that eventually, even if the construction was poorly marked.
I have a hunch that 90% of self-driving could be solved like that, but then there's always that 10% of situations where there's nobody in front or someone who doesn't know what they're doing.
In this case, if it followed the car ahead through a left turn, then that would be so, but if it did not, and it were still in the section of road restricted by the cones, it would have become stuck again, just as it did after it made the initial turn onto the restricted road.
The thing is, the strategy of 'follow other traffic' needs exit rules, and while it is relatively straightforward to begin the strategy when stuck, deciding when to abandon it is less clear-cut. Did the left-turning vehicle do so because this was the correct way to deal with the problem, or did it have a different goal?
"He had a tremendous propensity for getting lost when driving. This was largely because of his method of “Zen” navigation, which was simply to find any car that looked as if it knew where it was going and follow it. The results were more often surprising than successful, but he felt it was worth it for the sake of the few occasions when it was both."
> A programmer added jitter to work around that situation,
A random sleep to avoid a race condition.
If a bunch of randoms on the Internet can think of using the other vehicles as guides for the right way to go, why hasn't Waymo?
Also, why don't the roadside assistance people or the woman on the phone not have an off button to stop their cars randomly taking off on them? Other machinery has an e-stop, why not this?
I don't know if you're new to this whole technology thing but the answer a lot of the times is: things are more complicated than the simple words that can be used to describe the thing.
Exactly. There's a famous video out there from a few years back showing Tesla autopilot dutifully following a BMW at highway speeds. Then the BMW suddenly swerves around a stopped vehicle in front of it at the last minute, and the AP doesn't have time to recognize the object and come to a full stop, so it runs right into it. (The object is a fake baloon car, but the situation was real enough.)
Tesla has since improved recognition to prevent that situation, but the point remains: blindly following other vehicles is a common mistake. Remember all the "Do Not Follow" signs on construction vehicles?
>incredible to watch how Waymo was able to plot the huge number of cars accurately around the vehicle throughout the drive
4:30 look at the cars coming from the left and turning right (down the screen). Google hallucinates two cars right there, and thats just the start of the half hour journey.
> A human would have seen what all the other cars were doing, and emulated that eventually, even if the construction was poorly marked.
Google are probably planning for 100% of the cars on the road being AI-controlled, at which point it no longer makes sense to copy other car’s behaviour, because they are all AIs.
Seeing how monopolistic Google is in other areas, it would not surprise me if they are chasing a self-driving monopoly, too.
They did something similar by my house on a pretty lightly travelled road and it was confusing as hell. People were using both sides of the cones and realizing too late they were in the wrong side.
I agree. In my town roadwork on a four lane road caused both lanes to be diverted into two sparsely-coned lanes on one side of the road, and some human drivers started driving in the wrong lane into what would be opposing traffic.
You cannot drive into that lane without hitting those cones. It's hard to see from the camera angle in the backseat but a driver in the front would definitely understand that the lane is blocked.
The first cone was placed a few feet to the right of the lane divider. That and that the next cone was on the divider indicated that the right lane was closed.
I wonder if the traffic on the left lane somehow shifted the code/network into favoring that it was only the right lane that was closed. That may have gotten it moving after the first hang up.
However, once it turned into the left lane and passed the first cone (which was offset to the right of the divider), it saw the second cone placed in the center and forgot that it was the right lane that was closed.
I think it pulled over because it thought that both lanes were closed or did not know which one was closed. It’s modeling of a sequence of cones closing lanes and retention of that model seems inadequate.
I'd be very curious if this conework was even legal
It doesn't matter if it's legal. It is not code compliant. Neither is "double parking". But it happens beyond the control of mere mortal drivers, and it must be maneuvered through safely when it happens. It's like kids dashing into the street -- it's not legal, yet you aren't empowered to simply hit them if it can be avoided.
Here in California, it happens all the time. There was a half block of EL Camino Real in Sunnyvale this afternoon that was simply coned off with no signage and no warning by construction people working on a new building. I called it in to Sunnyvale DPS, and they were totally unaware of it. There were two near-collisions in the 45 seconds I was within view. You have day laborers with no clue about traffic safety and ignorant of context given hand signs and no instructions, let alone legal standards.
Even if you place cones and signage perfectly per code, you still have to deal with signs or cones being knocked out of position or out of view.
You may want to look at this from a wider angle. Obviously the driving logic should handle this properly. But... it also has the potential of production testing road infrastructure automatically. Once it's good enough obviously.
P.S. I hope similar tech will be built for other types of road use, so cars alone don't shape our roads even more.
> I called it in to Sunnyvale DPS, and they were totally unaware of it. There were two near-collisions in the 45 seconds I was within view.
Seems like if those confused drivers had merely stopped where they were, safety would have been improved!
Again, the question isn't whether or not automation technology can be confused by bad signage, we know it can. Nor is there a question that humans can be confused by the same situation. The question is, when faced with situations like this, we can trust the automation to operate safely.
And it seems like Waymo passed the test.
It's true the externalities are different for the different technologies though. If you put down bad cones currently, you raise the likelihood of an accident. If you mark your lane badly in the era of automated vehicles, you cause a traffic jam. I know which I'd prefer.
You're not controlling your variables. The choice isn't between "block the road" and "unimpeded safe traffic", it's between "block the road" and "dangerous driving conditions". As we all know bad signage causes accidents. Here, it did not. Because the car stopped.
The car did pull off to the side. Those cones mark the side of the road. What you're demanding it do is to cross the cones to pull off the closed-off shoulder of the road. Which is (1) illegal and (2) only safe because of assumptions you're making about why that road was closed.
I mean, consider that they had a giant automated Waymo-branded repaving machine coming down in that lane. Would pulling off the the side have been the right choice?
Again, it's true that an AI isn't going to have the same context that you do as a functioning human driver. And so it's going to make different decisions in pursuit of safety than you would. That's not evidence that it's unsafe. This behavior simply wasn't unsafe.
The car attempted to follow it's planned route to use the right lane, and stopped in place when it saw the next traffic cone, partially blocking the left lane.
Then it backed up and fully blocked the left lane.
I think as we come to cross a certain threshold of how widespread these autonomous vehicles are it just makes sense to shape infrastructure around the technology. For starters make it code for construction sites to place beacons that relay information to these vehicles so that there's no equivocation about what the right path for it to take is and what it needs to do.
Because making AVs work to the last step, to account for every possibility, we're not there and we're not getting there for a long long long time, so have then the bunch of AV-manufacturers come together and standardize on a secure (lol) communication protocol and give it the crutch that it apparently desperately needs.
> For starters make it code for construction sites to place beacons that relay information to these vehicles
FWIW: it really doesn't seem like it has to be that complicated. The car could see the cones just find and knew not to cross them. They just needed 4-5x more cones than they used to make it clear to the car[1] where the lane was actually going.
[1] And human drivers! Again, this conework was just plain awful, period. I'm 100% convinced that the poor Waymo device wasn't the only driver making bad decisions in that intersection that day.
> If you want to guide left-turning vehicles around the right lane you need to put something IN FRONT of the lane you closed.
The important point is that a human driver (as all the human drivers in the video) has no trouble. A lot of driving is not by the book, but humans coordinate and go with the flow. The software in these cars breaks down as soon as anything unexpected happens.
I second DrScump's point, but it is also a fact that the car did recognize that the lane was a no-go area, at least in its immediate vicinity. From the video, the fragility revealed here seems to be that the system driving the car is attemptng to follow the route from the planning system very explcitly, down to which lane to use, and there seems to have been no effective way to get that plan revised in the face of the problem.
In general, I would imagine that following a route plan to the lane level of detail is important; I can see that it would be when approaching a turn, for example. Also, when the car first encounters the cones, it only recognizes that the lane is blocked for a short distance: after that, each cone encountered seems to be a separate problem. Humans easily generalize a line of cones to a blocked lane and modify their plans accordingly, but we do not have AGI yet.
From the response, that's what it sounded like the remote support person was expected to have done. From watching the video, it appeared that the remote assistance was hitting some sort of "reset/retry" option when they should have been hitting the "unlock/turn off" option.
That'd perfectly explain why the car would suddenly start trying stuff again once the roadside crew got close (and why it kept cancelling the roadside call).
Well, if large fleets if AV's ever go feral, roaming around cities and suburbs, taking all the best parking spots, we now know that traffic cones are their weakness.
This is inevitable. Large automated fleets will definitely take over the cheapest / most convenient street parking between trips for each given vehicle.
Logically, robo fleets will drive to cause automated traffic jams around the lone human piloted cars, or robots from rivals straying onto their patch. It will take twice as long to drive manually than get a Johnny Cab.
There's a recent video of drivers encountering a new traffic circle and going wildly wrong. Multiple drivers pull into the left lane (i.e. oncoming traffic) and go around the circle the wrong way. Imagine the outrage if a self-driving car behaved like the drivers in the video.
But when you don’t allow drivers to take over, you need be perfect in all situations or your end up in this type of traffic circle all the time (or have to limit coverage are severely).
Yet in that video you don’t see a single crash or collision! Yes the situation is confusing for drivers but they manage to figure out how to get out of it without damages or delays even if they do it against “the plan”.
Sure, that's because traffic circles are terrible. The truck with the long trailer likely took the left turn to avoid it, possibly because clearance around the circle wasn't obvious (there's another truck with a shorter trailer that barely makes it around) and then people behind it followed.
The local traffic engineering folks around where I live are gung-ho on traffic circles, but I've yet to see one in the US where a traffic signal wouldn't have been a better investment.
Why do you think traffic lights are better? I like not having to wait on a red for an empty street or have to stop for stop signs at intersections. They are better and safer in every way and stats back that up (not to mention better for the environment and your car maintenance bills)
> I like not having to wait on a red for an empty street or have to stop for stop signs at intersections.
Me too; but traffic signals are more than capable of detecting empty streets and oncoming traffic and changing lights to green before you get to the intersection.
> They are better and safer in every way and stats back that up (not to mention better for the environment and your car maintenance bills)
Traffic circles are tolerable with low traffic, but when traffic increases, especially if the traffic is asymmetric, it's possible for some directions to be blocked for a long time, and there's no fairness mechanism. Because speeds in traffic circles are slower than in signaled intersections with a green, it takes longer for the backlog to clear the intersection. And, while there's still throughput when one entrance dominates and blocks other entrances, if one exit is blocked, it can block the whole intersection and there's zero throughput; in a signaled intersection, other exits can usually still be used even if one is jammed.
Slowing for a roundabout when I wouldn't need to slow for a traffic signal that would likely turn green for me is bad for the environment. Extra miles driven from longer paths, and wear on tires from taking a circular path instead of a straight path is bad for the environment and my maintenance bills. They use more space than a signaled intersection, and may displace trees or impact waterways and drainage.
It's hard to look for pedestrians crossing from the right while also looking for a space in traffic from the left. Navigating them on bicycles can be tricky too, as there are a lot of crossovers.
Frankly, you can't have driven in places that use roundabouts much, because in practice these are not problems. Asymmetric flow isn't a problem if there are occasional vehicles which force the dominant flow to give way. In rare situations it is beneficial to put a traffic light on the dominant flow entrance to break it up. Blocked exits are rarely a problem for other exits on multilane roundabouts, and rarely more of a problem that blocking the box on an intersection with lights -- it's all about the attitude of drivers.
"Wear in tired from taking a circular path" -- is this supposed to be satire?
>
Frankly, you can't have driven in places that use roundabouts much, because in practice these are not problems. Asymmetric flow isn't a problem if there are occasional vehicles which force the dominant flow to give way.
It may not be 'a problem', but it does add delay, and it ruins the narrative of better in all conditions. And it happens fairly frequently at the circle I transit most often. There's a car ferry, and when people drive off of that and eventually through this circle, there are directions they don't want to go, and if you're coming from one of those, you just have to sit and wait. And you wait much longer than a light would have you wait, and it takes the people coming from the ferry longer too.
> "Wear in tired from taking a circular path" -- is this supposed to be satire?
It's minimal, but so is the brake wear people point out as a benefit of doing something better than a 4-way stop.
Roundabouts just work with no power or fancy electronics. Those cost money to maintain and install as well as don’t handle a few cars coming from each direction without impeding one.
Slowing down and making a slight turn does not compare to the fuel and maintenance cost coming to a full stop and getting the cars mass moving again.
And regardless of it being “hard” to see pedestrians the statistics back up them being far safer: you have to be an active participant and look rather then “green means go” and not pay attention.
I suggest watching that video I linked. I’ve driven in a number of countries and the ones with more traffic circles were always more enjoyable + easier as well as subjectivity had better drivers
> I've yet to see one in the US where a traffic signal wouldn't have been a better investment.
Approximately every four way stop in the US could be replaced with a traffic circle and substantially increase throughput and reduce waste (brake wear for unnecessary stop, fuel consumption for accelerating away from the unnecessary stop).
You could say the same for adding a traffic signal with sensors. Compared to a circle, you would need to slow less (or not at all, depending on visibility and prevailing speeds), and you wouldn't need to steer around the circle either, reducing the uneccessary tire wear a circle adds. You also wouldn't need to acquire land surrounding the current intersection to accomidate the area needed for a traffic circle.
> You could say the same for adding a traffic signal with sensors.
Here in CA (at least silicon valley) approximately all traffic signals have sensors (it always catches me by surprise when I drive in other states and find there's no sensor so I have to wait endlessly for a green light).
But sensors are nowhere near a substitute for the efficiency of a traffic circle. With a traffic light with sensors you still have to stop for a short time, so it has all the disadvantages of a stop sign and none of the benefits of a traffic circle.
Depends on the sensors. I've certainly driven on streets where the far back sensor is far enough back that it can change the light before you need to slow down (it helps if it keeps all the lights red when the intersection is idle).
If you only have sensors at the line, then yeah, you need to stop.
Studies and evidence points to circles being safer and more fluid for traffic flow. The larger, the more flow.
As cars near a circle they need to slow down, follow the rules and allow some error. Being more cautious prevents more crashes and allow for more flow.
Automatic signals can be wrong, and promote reckless behaviour, confusion, less flow and more stops.
Machine learning has probably closed most of the gap in pattern matching the scene but humans are still magnitudes better at predicting intentions and communicating instantly with the other drivers with little more than a glance.
This is why chaotic traffic situations without lane markings or signals can still move fluidly without crashes. Autonomous vehicles would be at a standstill in that situation.
Snow is a fun condition, because it is a time when it may be impossible to drive to the letter of the law, but perfectly practical to drive by not following it.
It's going to be interesting to me how you work out: "It's legally required to use the left turn lane, but since it's not plowed out well enough everyone is turning from the travel lane instead and this is temporarily acceptable to both other drivers and any law enforcement"
But most of the drivers were perfectly fine with the cones and driving past it with no issues. The thing is if one waymo car was confused by the cones likely all waymo cars would be.
I can only imagine the chaos ensuing if another one or two waymo taxis would have arrived at the same location. Would they have gotten stuck behind each other. Simultaneously trying to desperately call for help and go around each other?
Human may violate known rules(or even laws) temporarily if all known legal solution can't get itself out of the problem. But AI nowadays don't do (or the designer of AI aren't allowed to? Is that even legal to do?).
And besides that. If such decision actually cause injury eventually, who should response for that? The AI designer? The car company? The AI itself? The car owner? We still don't have a definitive answer for that yet.
When I drove a 2020 VW Caravelle minivan with cruise control (is that what its called?) set on furthest distance between cars, when in a traffic jam in Germany september 2020, it started revving like crazy wanting to completely ram the poor little Mini Cooper in front of me just because it was cappucino colored - like the tarmac.
Never letting my guard down with such a feature again!
I'm surprised the first level of support driver isn't remote.
You could have a team of people sitting in HQ in a VR enviroment to take over more basic problems like this and there would be little delay. Then physical support for more challenging times.
A system like this could also be super handy for delivery type companies where the autopilot drives to the destination and the virtual driver takes over for the loading dock/driveway etc where human problem solving might help for the first few times a location is encountered.
Most interesting part is how the car "escapes" - once the support call had been initiated the remote operator should be able to turn off any self-driving capability, forcing vehicle to stay where it is until service vehicle arrives. The fact that they apparently did not implement this feature is really weird.
153 comments
[ 3.2 ms ] story [ 212 ms ] thread(This is why the trolley problem for AVs isn't real.)
Example running over an animal to avoid an accident, getting rear ended to avoid hitting a pedestrian, taking a calculated risk of being hit by oncoming traffic on a 2 lane situation when coming across a stalled car without optimal view on the road ahead.
It's going to make a decision between different risk factors and between different known costs.
Imagine on a 2 lane there is a pedestrian and oncoming traffic and insufficient time to stop. Do you kill the pedestrian or risk ending up in the ditch.
If you are going 60 and ending up in the ditch would likely be fatal is it ok to just smear the pedestrian?
What if you can slow to 20 and then ditch?
These situations won't hopefully be prevalent because a massive portion of our current issues are down to inattention or stupidity and we believe ai will eventually be much better at avoiding such problems but they are still important and interesting.
Yeah but not trolley problem style. "Don't swerve" is a valid strategy for a car.
> Example running over an animal to avoid an accident
Depends on the kind of accident you have in mind, so I can't give real feedback here.
> getting rear ended to avoid hitting a pedestrian
Always brake to avoid hitting a pedestrian. You don't choose whether to get rear-ended and it won't be your fault either.
> taking a calculated risk of being hit by oncoming traffic on a 2 lane situation when coming across a stalled car without optimal view on the road ahead.
When one of the choices is "sit there and wait longer", that's definitely not a trolley problem. It either thinks going forward is sufficiently safe or it doesn't. It can wait all day for someone to intervene.
> Imagine on a 2 lane there is a pedestrian and oncoming traffic and insufficient time to stop. Do you kill the pedestrian or risk ending up in the ditch.
If it's programmed not to swerve, it will hit the pedestrian, which is an acceptable outcome of an unacceptable situation.
> If you are going 60 and ending up in the ditch would likely be fatal is it ok to just smear the pedestrian?
It's sufficiently ok when the only alternative is so reckless. Getting into that situation is the not-ok part.
> What if you can slow to 20 and then ditch?
If you can slow to 20 then hit the pedestrian at 20.
> Depends on the kind of accident you have in mind, so I can't give real feedback here.
Typical scenario here is a dog appearing on your lane in a high-traffic, multi-lane road. You don't swerve, you don't even break too hard, because hitting the dog is much better outcome than hitting another car, or landing in a ditch, or getting rear ended.
Emergency breaking can typically stop a car within about 100 yards even from freeway speeds. But a distracted human might travel more than that before they even realize they should be trying to stop.
Humans are also pretty bad at adjusting to conditions, so if weather makes the road slick (wet or icy) or they have limited visibility, they don't often slow down enough to compensate.
Autonomous cars have none of these problems, so aren't likely to ever run in to a "can't stop in time" situation.
I can't actually think of any situation where, in the event of a likely collision, the best solution isn't to just try to stop. Even if, for some reason, you can't stop quickly enough to avoid the collision, any reduction in speed will result in a reduction in the severity of collision.
I don't think I that will be accepted by regulators/society until it can be demonstrated that AV's are undeniably safer-- not just as good as human drivers, much much better. Right now it's newsworthy whenever a Tesla auto pilot drives into a barrier. We'll need to get to a point where not only does that not happen, but also that it becomes newsworthy every time an AV saves a life-- swerving out if the way faster than any human reflexes could have reacted to avoid an accident, or a child chasing a ball into the street, etc.
And in case of disabled vehicles, accident cleanup, or other unplanned lane closures, sometimes the cones are just set out and you don't know anything about it in advance.
This is an OSHA violation by the construction workers. "When redirection of the road users’ normal path is required, they shall be channelized from the normal path to a new path."[1] There's supposed to be a diagonal line of cones across the closed lane to channel traffic away from the closed lane. See the drawings in [1].
Stopping was a good move.
[1] https://www.osha.gov/sites/default/files/2018-12/fy10_sh-210...
It caused other drivers to make illegal moves to get around, this just isn't true.
Even if it was, no other driver in the video had any trouble driving there. Self-driving cars, if it is ever to be viable, need to adjust to the world as it is.
[1] https://www.facebook.com/permalink.php?story_fbid=4133828946...
This setup wasn't completely MUTCD compliant, but, there are also some states with their own, state-specific editions of the MUTCD.
https://www.theturnsignalblog.com/blog/waymo-design/
> "...the Fleet Response team provided incorrect guidance..."
It sounds like the car automatically pulled over when it detected the cones (as intended), but the remote support team erroneously told it to forge ahead anyway, which resulted in the escalating series of problems. Not sure what the best solution for that particular issue would be -- better training for the remote support teams, I guess?
You can watch the video. It pulls over incorrectly at first and is blocking traffic, then the support team made a mistake, and a hilarious escalation ensues with the vehicle actually attempting to run away from the support team multiple times!
They deliberately can't drive the car remotely. If Waymo wants a human to drive the car, they send the human to where the car is, the human gets inside and drives it like any other car.
One easy thing that can go wrong, just as with any other human activity, is the humans make a mistake and their "clarification" is actively unhelpful. If the Waymo Driver is confused about which lane is coned off, and you "help" by telling it the wrong lane, unsurprisingly things will not be improved.
The best solution is always that the Waymo Driver learns how to do better on its own, if you imagine it sees this situation and just confidently picks the correct lane and uses it, Fleet Response never interacts at all, there is no opportunity for them to "provide incorrect guidance". So a bunch of the invisible effort at Waymo is on making the Waymo Driver just get stuff right on its own.
Letting the passenger drive would unfortunately involve the risk of exiting the vehicle in traffic, which the lawyers probably vetoed, even though it might be safer in some situations.
At any rate, the remote team in this case IS able to control the car, hence it trying to drive into traffic cones and Google blaming the humans. It appears that their interface just can't be very good, especially as it requires people to physically turn up in a truck - you definitely won't have good situational awareness if you're trying to drive a car through a web app whilst looking out the window at the other side of the road.
The remote assistant never does anything to affect the car onboard system. The only instance is when the assist car turns up and the car is put into hold mode for the driver to enter.
Yes, VR is far from perfect, granted. But is it worse than what they appear to be using instead? Apparently they couldn't see the traffic cones properly from whatever system they were using and repeatedly gave it bad instructions, even though what to do was obvious to the person in the car. It seems like VR should be compared to the existing system rather than theoretically perfect vision.
Nope. This was already explained once in this thread, and you ignored it. Here's what Google's statement actually says
> ... the Waymo Driver detected an unusual situation and requested the attention of a remote Fleet Response specialist to provide additional information. During that interaction, the Fleet Response team provided incorrect guidance, which made it challenging for the Waymo Driver to resume its intended route, and required Waymo's Roadside Assistance team to complete the trip.
They can remotely tell a Waymo, stuff like "this lane is closed" or "No, that's not a school bus, it's just a truck" - and it will add that to the information it has, this might enable it to complete the journey. If you phone your dad, asking what a street sign you saw means, and he tells you - is he now driving your car? No, clearly you are still driving the car, he just answered a question. If he says the symbol means "Free on-street parking" but it actually meant "No entry" you're going to get some problems if you try to act on this bad information.
I mean ultimately, you can try to justify this as a feature not a bug, but I would now feel far less safe getting into a Waymo knowing that they have such a poor level of control over their cars. You shouldn't find yourself stalled in the middle of a multi-line highway like that, that's just dangerous, especially when continuing and pulling over is something any human driver could do in seconds. Being limited to ad-hoc map patching is just not sufficient - there's a whole universe of bugs they might not be able to resolve in that way.
https://en.wikipedia.org/wiki/Risk_(short_story)
Not sure if Waymo is reading Waze info, those kind of alerts could have been updated there (for roadworks, etc)
Even as a human driver the conditions seem kind of weird. There's some cones placed on the dotted lines between two lanes but no obvious obstructions in either of the lanes. The vision system likely did not know which side to be on. Combine that with a truck driving by and snatching up some of the cones at one point as it was going along, and it is easy to see why the AV behaves erratically.
we really need to collectively get past this idea that safety is bounded on one side, across all facets of our lives, because it’s usually not and that induced complacency has a way of coming back to bite us. (that’s not to say that the various risks are always substantial, just that they’re not all sitting neatly on one side.)
As soon as the assistance team was dispatched, the car should have stayed where it was. Instead, it pulled out into a much busier road and stopped again.
But that isn't what happened (or would typically happen) in this type of situation. Usually the car blocking the lane will be enough of an impediment that a queue of traffic will build up behind to overtake it. Cars at the back of the queue will be coming to a stop more gradually.
Human drivers should not have critical failures just because there is an unexpected queue of traffic on the road.
This sort of thing happens all the time. And because of modern crumple zones and safety equipment, people have an excellent chance of living through it and doing it again.
You would not be able to drive into that lane without hitting cones straight-on until after the stall point.
https://youtu.be/zdKCQKBvH-A?t=1818
They way I understand that is the route planner, which I assume is ran completely separately to the self-driving car, removes un-navigable spaces from the route planning. So in this situation, the roadside assistance driver is surprised that the entire lane has not been removed from the route planner.
That's pretty damning and it makes me wonder if there is a huge gap in autonomy here that is being papered over by clever route planning (ie. avoid that lane altogether and the AI won't have to worry about the cones).
The AI should be able to figure out on its own that the right lane was closed, it shouldn't need route planning to do that for it.
Take the moment when the car first gets stuck. It attempts a right turn, spots the cones just fine, and correctly makes the assessment that it can avoid the cones and proceed in the right lane after the turn.
As a human, our thinking process does not stop there. We notice that the traffic already flowing on the street is _not_ using the right hand lane. We notice that the cones are vaguely in a merge pattern to indicate that cars entering from the side street should merge left - if you were looking at the cones from a vantage point behind where the cones are placed, and if you can call the first two cones enough of a "please merge left" pattern.
The Waymo car can't see that, though. It's located sort of off to one side, attempting to reason about it from a high angle. And two cones from this angle could be a lot of things, and it's not 100% clear the cones close off the right lane.
One theory might be that the Waymo model has been trained on a lot of situations but not this specific situation - something about the high angle, something about the position of the cones, something about the route plan managed to find a loophole in the AI training set and produce a weird result.
That's just the limitations of the current state-of-the-art Machine Learning. Find a glaring incorrect answer? Retrain with more training data.
Part of what makes full Artificial General Intelligence (AGI) with deductive reasoning can retrain itself without making mistakes along the way, unless something incomprehensible happens. And I use the word "incomprehensible" here to point straight at AGI's ability to _comprehend_ the situation at a much higher level than current machine learning techniques.
tl;dr: The Waymo Car didn't get it. And that's about what I would expect.
The real gem is that the _humans_ in the assistance team were fighting each other. The remote control operator kept jamming the "unstick yourself" button even as the roadside assistance guys were walking up to take over. The real danger, I think, is more of this in the future: the remote operators defeating all the safeties. Since the remote operator is not at any real risk of personal harm, what do they care?
IMO it's more concerning that the vehicle keeps trying after it gets into a "call roadside assistance" mode rather than staying still or allowing the remote operator to assume manual control (say to move the vehicle out of the lane).
Also, it really seemed like the Rider Support agent didn't have complete information on the status of the car or any apparent ability to intervene. I would love to know about how incident response is organized behind the scenes; there wasn't a lot of communication to the occupant about what was happening, which isn't ideal. The Roadside assistance request apparently being dropped when the vehicle began moving again, but before it demonstrated it was able to navigate the conditions is a bit concerning as well. It makes the response feel disorganized, although again, it could just be lack of information passing to the occupant.
“That's pretty damning and it makes me wonder if there is a huge gap in autonomy here that is being papered over by clever route planning”
To which I said that clever route planning is not a hack but the only reasonable way forward.
I was just trying to say it in even more plain terms - because this whole thread was full of people who seemingly have no idea about how AVs are being approached today and are, as is usual for HN, suggesting they are smarter than the actual people working on the technology.
Intelligent route planning to compensate for the deficits of our current perception tech stack is a good thing and allows greater reliability.
This is what Waymo is doing and has done from the start. But this approach itself has significant limitations as the video shows. You have to automate the route planning process also, remember.
It has limitations but there is no evidence of there being a better alternative. Automating the route planning process is easier than behavior planning, path planning or perception.
Wow, that was cringey. I would have ducked out the passenger-side door half way through the video and walked.
This is the kind of situation that confuses new human drivers too. Interpreting it basically requires that you have a model in your head of what cones are used for and the ability to guess that the traffic approaching from the left in only one lane indicates that the traffic to the right must use the left lane.
I'd be very curious if this conework was even legal. If you want to guide left-turning vehicles around the right lane you need to put something IN FRONT of the lane you closed. And that applies to side streets too, I'm almost sure.
Watch /r/idiotsincars for a bit, you'll see people misidentify this kind of situation all the time.
Frankly what the car did, stopping and calling for help, is the right move. A new human driver would almost certainly have just pulled out into the wrong lane.
Honestly, if it had just stayed on the side street, I wouldn't be so critical, but the fact that even after an assistance team was dispatched, the car pulled into the main highway and then immediately stopped would have frightened me enough to get out of the car and wait on the sidewalk. You can also see a moment where the car starts to pull out into traffic as another car is passing it and then immediately slams on the brakes.
Stop it. Waymo vehicles (Tesla autonomy too, FWIW) are extremely safe, that is just an experimental fact. The circumstance in question here has nothing to do with safety, really. The bug is that it caused a traffic obstruction while the (human) operators attempted to get the car oriented.
Hyperbolizing the circumstances is really not appropriate. This was not a safety situation, really, at any point.
There's a joke among cyclists: "If you want to get away with killing someone just be sure that you're driving a car and they're riding a bike." We could make a similar joke about these killer robot cars. "If you want to test killer robots on the public just make sure they are shaped like cars." (And if you need to, you can throw in a "safety driver" as a sort of moral crumple zone to deflect the blame when your robot kills e.g. Elaine Herzberg. https://en.wikipedia.org/wiki/Death_of_Elaine_Herzberg#After... )
- - - -
We could build self-driving golf carts (slow, light, not so capable of killing) and work up to cars, eh?
We don't have to go straight to fast heavy cars, eh?
I acknowledge that it is weird that the remote assistance team can't remotely stop the vehicle and I agree that it should have behaved differently but maybe there is some kind of engineering reason or just a glitch. It seems to recalculate what to do three times.
It was incredible to watch how Waymo was able to plot the huge number of cars accurately around the vehicle throughout the drive. But as soon as something creative was required, it all fell apart.
I still remember a talk where someone from the original DARPA challenge was recounting an episode where two autonomous cars arrived at an intersection precisely at the same time. They deadlocked because they were waiting for each other to go. A programmer added jitter to work around that situation, but it is one of those things humans innately would figure out.
I have a hunch that 90% of self-driving could be solved like that, but then there's always that 10% of situations where there's nobody in front or someone who doesn't know what they're doing.
But yes it would be a good heuristic.
The thing is, the strategy of 'follow other traffic' needs exit rules, and while it is relatively straightforward to begin the strategy when stuck, deciding when to abandon it is less clear-cut. Did the left-turning vehicle do so because this was the correct way to deal with the problem, or did it have a different goal?
"He had a tremendous propensity for getting lost when driving. This was largely because of his method of “Zen” navigation, which was simply to find any car that looked as if it knew where it was going and follow it. The results were more often surprising than successful, but he felt it was worth it for the sake of the few occasions when it was both."
A random sleep to avoid a race condition.
If a bunch of randoms on the Internet can think of using the other vehicles as guides for the right way to go, why hasn't Waymo?
Also, why don't the roadside assistance people or the woman on the phone not have an off button to stop their cars randomly taking off on them? Other machinery has an e-stop, why not this?
Tesla has since improved recognition to prevent that situation, but the point remains: blindly following other vehicles is a common mistake. Remember all the "Do Not Follow" signs on construction vehicles?
4:30 look at the cars coming from the left and turning right (down the screen). Google hallucinates two cars right there, and thats just the start of the half hour journey.
Google are probably planning for 100% of the cars on the road being AI-controlled, at which point it no longer makes sense to copy other car’s behaviour, because they are all AIs.
Seeing how monopolistic Google is in other areas, it would not surprise me if they are chasing a self-driving monopoly, too.
You cannot drive into that lane without hitting those cones. It's hard to see from the camera angle in the backseat but a driver in the front would definitely understand that the lane is blocked.
I wonder if the traffic on the left lane somehow shifted the code/network into favoring that it was only the right lane that was closed. That may have gotten it moving after the first hang up.
However, once it turned into the left lane and passed the first cone (which was offset to the right of the divider), it saw the second cone placed in the center and forgot that it was the right lane that was closed.
I think it pulled over because it thought that both lanes were closed or did not know which one was closed. It’s modeling of a sequence of cones closing lanes and retention of that model seems inadequate.
Here in California, it happens all the time. There was a half block of EL Camino Real in Sunnyvale this afternoon that was simply coned off with no signage and no warning by construction people working on a new building. I called it in to Sunnyvale DPS, and they were totally unaware of it. There were two near-collisions in the 45 seconds I was within view. You have day laborers with no clue about traffic safety and ignorant of context given hand signs and no instructions, let alone legal standards.
Even if you place cones and signage perfectly per code, you still have to deal with signs or cones being knocked out of position or out of view.
P.S. I hope similar tech will be built for other types of road use, so cars alone don't shape our roads even more.
Seems like if those confused drivers had merely stopped where they were, safety would have been improved!
Again, the question isn't whether or not automation technology can be confused by bad signage, we know it can. Nor is there a question that humans can be confused by the same situation. The question is, when faced with situations like this, we can trust the automation to operate safely.
And it seems like Waymo passed the test.
It's true the externalities are different for the different technologies though. If you put down bad cones currently, you raise the likelihood of an accident. If you mark your lane badly in the era of automated vehicles, you cause a traffic jam. I know which I'd prefer.
It did not seem so to me. Blocking the road (when it was clearly possible to get out of the way) increased the danger to others through inaction.
Issac Asimov would tut tut.
The choice was between block the road, or pull off to the side, when it was clearly safe to do so.
The car did pull off to the side. Those cones mark the side of the road. What you're demanding it do is to cross the cones to pull off the closed-off shoulder of the road. Which is (1) illegal and (2) only safe because of assumptions you're making about why that road was closed.
I mean, consider that they had a giant automated Waymo-branded repaving machine coming down in that lane. Would pulling off the the side have been the right choice?
Again, it's true that an AI isn't going to have the same context that you do as a functioning human driver. And so it's going to make different decisions in pursuit of safety than you would. That's not evidence that it's unsafe. This behavior simply wasn't unsafe.
That's not what I saw.
The car attempted to follow it's planned route to use the right lane, and stopped in place when it saw the next traffic cone, partially blocking the left lane.
Then it backed up and fully blocked the left lane.
Because making AVs work to the last step, to account for every possibility, we're not there and we're not getting there for a long long long time, so have then the bunch of AV-manufacturers come together and standardize on a secure (lol) communication protocol and give it the crutch that it apparently desperately needs.
FWIW: it really doesn't seem like it has to be that complicated. The car could see the cones just find and knew not to cross them. They just needed 4-5x more cones than they used to make it clear to the car[1] where the lane was actually going.
[1] And human drivers! Again, this conework was just plain awful, period. I'm 100% convinced that the poor Waymo device wasn't the only driver making bad decisions in that intersection that day.
The important point is that a human driver (as all the human drivers in the video) has no trouble. A lot of driving is not by the book, but humans coordinate and go with the flow. The software in these cars breaks down as soon as anything unexpected happens.
In general, I would imagine that following a route plan to the lane level of detail is important; I can see that it would be when approaching a turn, for example. Also, when the car first encounters the cones, it only recognizes that the lane is blocked for a short distance: after that, each cone encountered seems to be a separate problem. Humans easily generalize a line of cones to a blocked lane and modify their plans accordingly, but we do not have AGI yet.
That'd perfectly explain why the car would suddenly start trying stuff again once the roadside crew got close (and why it kept cancelling the roadside call).
There are a ton of situations where humans fail at driving too. Like white out snow or heavy rain and fog. Or confusing cones. :)
https://www.youtube.com/watch?v=bDaQZUzJCNM
The local traffic engineering folks around where I live are gung-ho on traffic circles, but I've yet to see one in the US where a traffic signal wouldn't have been a better investment.
https://youtu.be/AqcyRxZJCXc
Me too; but traffic signals are more than capable of detecting empty streets and oncoming traffic and changing lights to green before you get to the intersection.
> They are better and safer in every way and stats back that up (not to mention better for the environment and your car maintenance bills)
Traffic circles are tolerable with low traffic, but when traffic increases, especially if the traffic is asymmetric, it's possible for some directions to be blocked for a long time, and there's no fairness mechanism. Because speeds in traffic circles are slower than in signaled intersections with a green, it takes longer for the backlog to clear the intersection. And, while there's still throughput when one entrance dominates and blocks other entrances, if one exit is blocked, it can block the whole intersection and there's zero throughput; in a signaled intersection, other exits can usually still be used even if one is jammed.
Slowing for a roundabout when I wouldn't need to slow for a traffic signal that would likely turn green for me is bad for the environment. Extra miles driven from longer paths, and wear on tires from taking a circular path instead of a straight path is bad for the environment and my maintenance bills. They use more space than a signaled intersection, and may displace trees or impact waterways and drainage.
It's hard to look for pedestrians crossing from the right while also looking for a space in traffic from the left. Navigating them on bicycles can be tricky too, as there are a lot of crossovers.
"Wear in tired from taking a circular path" -- is this supposed to be satire?
It may not be 'a problem', but it does add delay, and it ruins the narrative of better in all conditions. And it happens fairly frequently at the circle I transit most often. There's a car ferry, and when people drive off of that and eventually through this circle, there are directions they don't want to go, and if you're coming from one of those, you just have to sit and wait. And you wait much longer than a light would have you wait, and it takes the people coming from the ferry longer too.
> "Wear in tired from taking a circular path" -- is this supposed to be satire?
It's minimal, but so is the brake wear people point out as a benefit of doing something better than a 4-way stop.
Slowing down and making a slight turn does not compare to the fuel and maintenance cost coming to a full stop and getting the cars mass moving again.
And regardless of it being “hard” to see pedestrians the statistics back up them being far safer: you have to be an active participant and look rather then “green means go” and not pay attention.
I suggest watching that video I linked. I’ve driven in a number of countries and the ones with more traffic circles were always more enjoyable + easier as well as subjectivity had better drivers
Approximately every four way stop in the US could be replaced with a traffic circle and substantially increase throughput and reduce waste (brake wear for unnecessary stop, fuel consumption for accelerating away from the unnecessary stop).
Here in CA (at least silicon valley) approximately all traffic signals have sensors (it always catches me by surprise when I drive in other states and find there's no sensor so I have to wait endlessly for a green light).
But sensors are nowhere near a substitute for the efficiency of a traffic circle. With a traffic light with sensors you still have to stop for a short time, so it has all the disadvantages of a stop sign and none of the benefits of a traffic circle.
If you only have sensors at the line, then yeah, you need to stop.
As cars near a circle they need to slow down, follow the rules and allow some error. Being more cautious prevents more crashes and allow for more flow.
Automatic signals can be wrong, and promote reckless behaviour, confusion, less flow and more stops.
UK is all circles, with most of Europe following.
This is why chaotic traffic situations without lane markings or signals can still move fluidly without crashes. Autonomous vehicles would be at a standstill in that situation.
It's going to be interesting to me how you work out: "It's legally required to use the left turn lane, but since it's not plowed out well enough everyone is turning from the travel lane instead and this is temporarily acceptable to both other drivers and any law enforcement"
I can only imagine the chaos ensuing if another one or two waymo taxis would have arrived at the same location. Would they have gotten stuck behind each other. Simultaneously trying to desperately call for help and go around each other?
Since billions of people are driving unassisted every day, that's clearly not true.
And besides that. If such decision actually cause injury eventually, who should response for that? The AI designer? The car company? The AI itself? The car owner? We still don't have a definitive answer for that yet.
Never letting my guard down with such a feature again!
You could have a team of people sitting in HQ in a VR enviroment to take over more basic problems like this and there would be little delay. Then physical support for more challenging times.
A system like this could also be super handy for delivery type companies where the autopilot drives to the destination and the virtual driver takes over for the loading dock/driveway etc where human problem solving might help for the first few times a location is encountered.
Way to keep it classy, Waymo. Glad I don't work for you, I prefer to not be thrown under the bus by my employer.