Like the issues we have seen with Bing and ChatGPT, it seems that self-driving cars are a long way from being resilient to real world uses. The issue is the complexity of the domain space, which which might not lend itself well to the mathematical model. You can get impressive results, in some cases exceeding human performance, but the problem is when they fall short in spectacular ways. The human brain is much more flexible in recovering from certain types of errors. Bing might suddenly profess its undying love for a user, while self driving might become inexplicably convinced that a cliff is the correct path because of some odd arrangement of pixels. The problem is especially keen when there are adversarial influences dedicated to creating problems.
Directly. It once again shows that so called AI has no world model. For it all the possiblities are more or less equal why the human will spot what's wrong at once.
You could come up with a pattern that a human wouldn't recognize as a STOP sign but a machine would. Would that mean that the human doesn't have a world model but the machine does?
But that pattern wouldn't be allowed on a stop sign, it would have to almost entirely obstruct it to work since we're so used to looking for the shape and even a hint of red, all it took to confuse the AI was a word above and below that covers a very small percent of the area. Stickers on stop signs are ubiquitous in a lot of places, so this is a fairly probable situation, whereas I've never in my life seen a stop sign that was so obstructed that I wouldn't recognize it. I'm sure someone could design a stop sign the machine could recognize and we couldn't but that's not what's on the roads and arguing about that is pointless because the amount of infrastructure you'd have to replace to get there is so massive that it's not relevant.
This was not done using random stickers, but by using stickers that specifically exploit the recognition mechanism of the car. It's like creating an optical illusion for a human to fool him into thinking it's not a stop sign.
Yeah but an optical illusion to confuse a human would have to cover most of the sign I think, and I know this isn't a random sticker but if this is possible there's no way that people don't make some and put them on signs just to mess with the cars because they can. In fact, I'm sure they'd do the same with stickers that would confuse people, it's just that there aren't any (that we know of, but considering self driving is a lot newer than people driving I'd say finding one for cars already but not people is telling on it's own) small enough that they wouldn't get immediately removed.
If you remove a sign it gets replaced right away, if you put a sticker that barely covers it like this then it probably doesn't get taken off for weeks, if at all.
Sure but these look like the sort of thing that would be a low probability to fix where as something that’d fool humans would be at least a high priority (not gonna say every town would fix it quickly)
Also something that fools humans about street signs would probably also fool AI
I think the important distinction is that humans aren’t confused by the stickers that confuse “self-driving” technology. The stickers in the article, for example, are unlikely to confuse a human.
Although the article didn't say which stickers were used - they __implied__ that the signs shown fooled the vehicles but they didn't state it explicitly. Who knows, maybe they stuck 65 mile/hour speed limit sign over a stop sign.
You’re right; I think these are the actual ones[1]. My intuition is that a human is equally unlikely to be fooled by these stop sign modifications, although the curve warning sign might fool someone at night.
Stop sign has a unique shape for the reason. Humans would recognise it by shape only – and that's the reason for it being unique. Even if it is covered in snow, you know that's it is the STOP sign. Similarly, the bottom-up triangle can only mean "yield".
Who knows, maybe they stuck 65 mile/hour speed limit sign over a stop sign.
That wouldn't fool a person. People use a massive amount of contextual data, historical knowledge, and common sense to interpret the world around them. A random out-of-place sign like a 65mph marker at a junction where a stop sign might be isn't enough to trick someone.
Self-driving cars probably won't ever be as good as a person at this. That's fine because we're able to accept a small risk if the reward is enough of a benefit. Law-makers should be considering it and making "tampering with a road sign" a much more serious explicit crime that's worse than just basic vandelism though.
This isn't kindergarten or a 1on1 meeting where you wrap the negative comments in positive comments. This is a harsh review of a piece of tech being touted as the next coming yet has such horrible faults that it can't be anything but a hobby project at best. This has the ability to kill people because of these faults, so telling bobbyDriverlessCar that he did pretty good, but has one area that needs improvement because you don't want to destroy bobbyDriverlessCar's over inflated ego is just a dumb way to think of this.
This was my thought as well. And I think it is an important fact to keep in mind. You can fool an AI by putting a sticker on a stop sign, but then again you can fool a human by stealing the entire stop sign.
Either way you're defacing infrastructure in a way that's potentially dangerous. It's just that the particular failure cases for an AI is different (and somewhat incomprehensible) to humans.
Of course, a self driving system could also be linked to a network that has redundant information about the road and could also broadcast its position and immediate intent to all other nearby self driving systems.
Now, on the other hand. If a police officer sees a missing stop sign, then it's pretty obvious that there is something that needs to be reported and addressed. However, if there's a small sticker on a stop sign (possibly something even confusable as just dirt or bird poop) then it might not be obvious that this represents a road hazard.
When I read about self-driving cars being so reliant on street signs, I can only imagine the total shitstorm they would cause in a country like my own (Brazil).
When seeing this, are we meant to assume that not one company in the self-driving space, in the 5-6 years of research and development since this study, has moved to training their AI on anything other than stock, pristine, textbook roadsigns?
But that's the point, and main argument people use against deep-learning based AI for real-world applications like driving. The moment you go outside the training set, failure generally follows
Cool. And the moment you get an incompetent, emotional or impaired human driving a car, failure generally follows. All you need to do is make sure your training set is complete enough to do slightly better than the average human.
I believe that if car crash deaths were reduced to 4,500/year in the U.S. with only self-driving cars on the road — a 90% drop from last year's actual 45,000 — an overwhelming majority of Americans would still reject self-driving cars because of the loss of agency involved.
That's one factor contributing to fear of flying, though with a car you can stop and get out at least.
Edit: Also, I think that many of the casualties resulting from autonomous vehicles will be situations that humans could have easily avoided (vs. lots of saved situations where humans couldn't have done anything).
I'm sure there's numbers, but a quick listicle says that most accidents are caused by distracted driving (e.g. using phones), reckless / aggressive driving, and driving drunk. Those aren't solved by automatic driving per se, but mitigated.
But then, aggressive drivers will be the last ones to put the autopilot on I'm sure, they're also the ones that will insist on the agency aspect of driving.
Since by nature autonomous driving requires the use of cameras and other data collection devices, how long will it take for the most aggressive drivers to be bleed poor by having streets full of autonomous vehicle reporting an aggressive driver? Plus, insurance companies are sure to want to use that data to determine fault, likely reversing the trend of no-fault laws. Seems like those who drive manually for the purpose for driving aggressively will be outnumbered and snitched to poverty by those who accept the more sedate driving habits of their vehicle.
I'm not saying that there couldn't be a huge reduction in accidents by autonomous vehicles but that the few accidents that do happen may be so inexplicably tragic and mind-numbingly preventable for a human driver, that selling the shift may be difficult.
This would not just affect the adoption of autonomous vehicles but all kinds of automated systems without direct human oversight. If the system makes too many mistakes a human would catch but prevents many mistakes a human wouldn't, the best of both worlds is to have a human in the loop and catch all the mistakes that can be made.
I also think people who don’t speed, and don’t drink (ever, or especially before driving) want to put themselves at risk to a dumb software bug. Like it’s great if we could reduce deaths but man I’d feel horrible about my kid dying because some computer vision engineer misidentifies a truck for a road.
Teslas have consistently ranked at the top of most crash test benchmarks. They are extremely safe. Additionally their driver assistance systems enforce more checks that the driver is paying attention (more than Volkswagen at least).
Can you please link to the lawsuit? It could be one related to unnecessary emergency breaking which would likely be annoying but not cause death.
How many of those life savings don't require FSD? Speed limiters would make significant impact. If we want to be really extreme, ignition interlocks could eliminate many of the 10k+ that die from alcohol-related accidents. I'm not saying we should go that far, but I'm not certain that's any further than FSD.
I was just thinking yesterday that it would be nice to have the feeling of a tactile detent or similar when the car reaches the posted speed limit that requires pushing through. Most of the time my car knows the speed limit through its own maps or CarPlay. The cruise control speed already changes its color if the limit is exceeded.
Just that subtle hint that the speed limit has been reached would likely keep me near the limit most of the time. I don’t even want it to prevent me from exceeding the speed limit.
This doesn’t help with the 95% of other cars that don’t know the speed limit, but it has to start somewhere. It seems odd that my car has redundant features and alerts for avoiding speed traps and traffic cameras, but little to actively help me stay within posted traffic rules.
I had suggested that concept on Twitter a few years ago to a certain CEO. I don't (and didn't) have the visibility that certain people have to get a reaction. I'm sure NHSTA would have something bad to say about it like "well, we can't have the pedal behave differently because well... we suck". I would love this tactile response as that would definitely help stay at the posted speed limit when driving with no cruise or autopilot. EVs hav a crapload of torque, and it's so easy to speed up quickly (or slow down with regen) where a constant speed it quite difficult.
I pretty much always use cruise control these days. I don't "obey" the speed limit - I set it for Houston norms (5 over on city streets, 10 on freeway). Even then I'm getting passed. I do find that it keeps me mostly sane when I'm driving (especially since getting an AWD EV that can out-accelerate virtually every gas car), something I've had issues with in the past.
I'll be honest, I prefer to die in an accident "of today" where for example someone crashes into me which maybe an AI could stop, than to die in an accident where if I had the AI disabled I would never crash as a human driver.
When it engages it does necessarily replace or override the human as if the human broke it wouldn't engage. Of course it's not intended to be relied on exclusively but self driving cars as of today similarly require you to pay attention and override when necessary
Yes, auto-brake technology is catastrophically bad in its current half-implemented state.
I assure you that false collision alarms are causing accidents that aren't being accounted for, some of them serious. Just as with first-generation airbags that did more harm than good to children and properly-belted passengers, it'll take a while for the actuaries and statisticians to catch up with how modern cars are actually behaving on the road. At that point a cynic might predict (or even hope for) the trial lawyers to take it from there.
Last to catch on will be the legislators who have mandated auto-braking tech without imposing strict performance and reliabilty standards.
Personally, I’d simply prefer not to die. If something can reduce the risk of that happening by 90%, that sounds great.
People were worried about dying from electricity—a power source out of their control—to turn on their lights, and it certainly did and still does kill people, but more people died by starting fires in their homes to read at night or stay warm. But the number of people wanting to replace their lightbulbs with candles for a little extra control of their energy source is quite low (although they do exist).
> I believe that if car crash deaths were reduced to 4,500/year in the U.S. with only self-driving cars on the road — a 90% drop from last year's actual 45,000 — an overwhelming majority of Americans would still reject self-driving cars because of the loss of agency involved.
And very rightly so. Someone who never drinks under influence and nearly always drives during the day (say to/from school) and is driving since 30 years has a probability so much lower to be involved in a deadly crash than a 18 years old on alcohol or drug driving on a Saturday night that it'd require much more than a 90% drop for that person to be safer by moving to a self-driving car if they were to only create 90% less accidents.
Here's something totally bonkers: in several statistics the car brand the less involved in car crashes (per miles or per 1000 cars of that brand or per anything you want) is...
Porsche.
You read that correctly. And it's not because Porsche repeatedly gets amazing short braking distances compared to other car brands: it's because young inexperienced people don't have the means to buy a Porsche (and Porsche are still pretty much regular cars, not crazy nervous stuff like Ferrari or Lamborghini).
If self-driving cars could be implicated in 90% less accident, the winning strategy would very likely be something like this: "put anyone under 25 years (and over a certain age too) in a self-driving car".
And maybe, maybe, once self-driving cars are implicated in 99% less accidents, experienced drivers never driving under influence would accept to move to self-driving cars.
There's absolutely no reason that those the less likely to have crashes should be moving to cars that are less likely to create crashes for those the most likely to be implicated in crashes.
"Moving everybody to self-driving cars" is the wrong way to think about the issue.
It's the segment at risks that should be moved to self-driving cars.
People just look at the total number of accidents and don't realize that "Porsche on the highway driven by an experienced drive bringing his kids back from school during daytime" is not anywhere near as likely as having a serious crash compared to "teenager on alcohol or drugs driving on a secondary road on a Friday night".
They're not even comparable. A rounding error.
Put it this way: it you were to get a 90% by moving everybody to self-driving car, you'd get an even bigger than 90% drop by moving only those in the segment at risks to self-driving cars.
And then, later on, once self-driving cars drive near perfection, you can move everybody to self-driving cars.
P.S: and to put things into perspective, we're talking about self-driving cars being fooled by stickers or bird poo on street signs. There's still some way to go.
It's all a matter of the context that humans understand. I've seen stop signs for the left hand turn lane that the main lanes didn't have. I've also seen signs a bit askew (probably from kids messing with the sign) that were angled in such a way that an automated system may think belongs to the wrong street, but a human has no issues with.
When I see efforts like this, I feel like I'm taking crazy pills; i.e. the only reasonable way any of this works is that you'll need not just the "car" to be "smart," but the environment as well. Sensor/GPS/location thingies on other cars and roads, etc.
53 comments
[ 4.0 ms ] story [ 41.9 ms ] thread[1] Man beats machine at Go, https://arstechnica.com/information-technology/2023/02/man-b...
In fact, road signs are just stickers on a bit of metal themselves.
Also something that fools humans about street signs would probably also fool AI
[1]: https://www.caranddriver.com/news/a15340148/researchers-find...
That wouldn't fool a person. People use a massive amount of contextual data, historical knowledge, and common sense to interpret the world around them. A random out-of-place sign like a 65mph marker at a junction where a stop sign might be isn't enough to trick someone.
Self-driving cars probably won't ever be as good as a person at this. That's fine because we're able to accept a small risk if the reward is enough of a benefit. Law-makers should be considering it and making "tampering with a road sign" a much more serious explicit crime that's worse than just basic vandelism though.
When discussing this I don't hear the positives
This isn't kindergarten or a 1on1 meeting where you wrap the negative comments in positive comments. This is a harsh review of a piece of tech being touted as the next coming yet has such horrible faults that it can't be anything but a hobby project at best. This has the ability to kill people because of these faults, so telling bobbyDriverlessCar that he did pretty good, but has one area that needs improvement because you don't want to destroy bobbyDriverlessCar's over inflated ego is just a dumb way to think of this.
Either way you're defacing infrastructure in a way that's potentially dangerous. It's just that the particular failure cases for an AI is different (and somewhat incomprehensible) to humans.
Of course, a self driving system could also be linked to a network that has redundant information about the road and could also broadcast its position and immediate intent to all other nearby self driving systems.
Now, on the other hand. If a police officer sees a missing stop sign, then it's pretty obvious that there is something that needs to be reported and addressed. However, if there's a small sticker on a stop sign (possibly something even confusable as just dirt or bird poop) then it might not be obvious that this represents a road hazard.
I believe that if car crash deaths were reduced to 4,500/year in the U.S. with only self-driving cars on the road — a 90% drop from last year's actual 45,000 — an overwhelming majority of Americans would still reject self-driving cars because of the loss of agency involved.
Edit: Also, I think that many of the casualties resulting from autonomous vehicles will be situations that humans could have easily avoided (vs. lots of saved situations where humans couldn't have done anything).
But then, aggressive drivers will be the last ones to put the autopilot on I'm sure, they're also the ones that will insist on the agency aspect of driving.
This would not just affect the adoption of autonomous vehicles but all kinds of automated systems without direct human oversight. If the system makes too many mistakes a human would catch but prevents many mistakes a human wouldn't, the best of both worlds is to have a human in the loop and catch all the mistakes that can be made.
People often pitch this view but don’t provide real numbers.
Can you please link to the lawsuit? It could be one related to unnecessary emergency breaking which would likely be annoying but not cause death.
Just that subtle hint that the speed limit has been reached would likely keep me near the limit most of the time. I don’t even want it to prevent me from exceeding the speed limit.
This doesn’t help with the 95% of other cars that don’t know the speed limit, but it has to start somewhere. It seems odd that my car has redundant features and alerts for avoiding speed traps and traffic cameras, but little to actively help me stay within posted traffic rules.
Basically I prefer to be exposed to risks that a human "can't" avoid than from something like this which a human wouldn't do: https://www.sfgate.com/bayarea/article/tesla-firetruck-car-c...
I assure you that false collision alarms are causing accidents that aren't being accounted for, some of them serious. Just as with first-generation airbags that did more harm than good to children and properly-belted passengers, it'll take a while for the actuaries and statisticians to catch up with how modern cars are actually behaving on the road. At that point a cynic might predict (or even hope for) the trial lawyers to take it from there.
Last to catch on will be the legislators who have mandated auto-braking tech without imposing strict performance and reliabilty standards.
People were worried about dying from electricity—a power source out of their control—to turn on their lights, and it certainly did and still does kill people, but more people died by starting fires in their homes to read at night or stay warm. But the number of people wanting to replace their lightbulbs with candles for a little extra control of their energy source is quite low (although they do exist).
And very rightly so. Someone who never drinks under influence and nearly always drives during the day (say to/from school) and is driving since 30 years has a probability so much lower to be involved in a deadly crash than a 18 years old on alcohol or drug driving on a Saturday night that it'd require much more than a 90% drop for that person to be safer by moving to a self-driving car if they were to only create 90% less accidents.
Here's something totally bonkers: in several statistics the car brand the less involved in car crashes (per miles or per 1000 cars of that brand or per anything you want) is...
Porsche.
You read that correctly. And it's not because Porsche repeatedly gets amazing short braking distances compared to other car brands: it's because young inexperienced people don't have the means to buy a Porsche (and Porsche are still pretty much regular cars, not crazy nervous stuff like Ferrari or Lamborghini).
If self-driving cars could be implicated in 90% less accident, the winning strategy would very likely be something like this: "put anyone under 25 years (and over a certain age too) in a self-driving car".
And maybe, maybe, once self-driving cars are implicated in 99% less accidents, experienced drivers never driving under influence would accept to move to self-driving cars.
There's absolutely no reason that those the less likely to have crashes should be moving to cars that are less likely to create crashes for those the most likely to be implicated in crashes.
"Moving everybody to self-driving cars" is the wrong way to think about the issue.
It's the segment at risks that should be moved to self-driving cars.
People just look at the total number of accidents and don't realize that "Porsche on the highway driven by an experienced drive bringing his kids back from school during daytime" is not anywhere near as likely as having a serious crash compared to "teenager on alcohol or drugs driving on a secondary road on a Friday night".
They're not even comparable. A rounding error.
Put it this way: it you were to get a 90% by moving everybody to self-driving car, you'd get an even bigger than 90% drop by moving only those in the segment at risks to self-driving cars.
And then, later on, once self-driving cars drive near perfection, you can move everybody to self-driving cars.
P.S: and to put things into perspective, we're talking about self-driving cars being fooled by stickers or bird poo on street signs. There's still some way to go.
Fooling a 2017 Tesla or fooling a 2017 Waymo is a big difference.