This is as good of a demonstration of the big "common sense" gap that end-to-end computer vision approaches have as it is of adversarial examples.
Even with an occluded speed limit sign, humans have strong priors on what a sensible speed limit would be based on a more general understanding of the environment (width of lanes, curvature of roads, visibility, road quality, frequency of law enforcement vehicles ;), etc.).
You likely could train it to learn about trends and certain contexts. But then you still have to think and enumerate all of these separate possibilities.
Many cars now come with a database of speed limits and display it on the dashboard. Although this shouldn't be relied on for self driving, it could be used to double check. OSM data for types of roads and areas is pretty accurate too, so you could use that ('an 85mph speed limit for a single lane within a residential area doesn't sound correct'). This doesn't need AI, it just needs someone with common sense to come up with a few basic rules about speed limits. If there is some doubt you could prompt the driver to confirm the speed limit change, and use that to build up a database of real world data.
It sounds like the Tesla is just using what the camera sees - which is bad if true. Admittedly this was tested on a race track, so maybe there is no data saying otherwise (or even the opposite).
I don't think that this in particular is really a good example of this gap. It feels more like a sloppy implementation. Just like humans, an autopilot has multiple sources available for guessing what an appropriate speed would be. Simply including information from a map application should be enough to prevent these kind of mistakes/hacks. Now things like children playing on the side of the road certainly seem more difficult to handle..
Further proving that "self driving" is approximately as hard as AGI, and is nowhere near as "close" as everyone thinks to think it is. A human would have context and common sense and thus know that a residential street will simply not ever have an 85 mph speed limit.
And yes, you could special case this case in the code, but there are hundreds of cases where "common sense" is used in driving, and you will never teach your NNs all of it.
Prediction: no actual "self driving" (to a point where driver can legally be asleep) on public streets till > 10 years from now
I don't see why you need special cases in the code. Sat nav units have had speed limits built in for years. Why not just look it up in a database? (the car knows where it is after all, and requiring a GPS fix could be mandatory - most drones won't arm without it, for example)
However if the sat nav gives you a prior that says 35 mph, and the road sign says 85 mph, do you trust it? Especially if the adjacent roads are also low speed, and it's clearly not a highway etc. This is perhaps a bad example as there are plenty of roads in the UK that transition from national to 30 in rural areas (and plenty of hamlets that you can bomb through).
Roadworks are an obvious situation where you drive at a _lower_ speed. This would be a case where you could use computer vision to identity an obstruction, identify roadworks and then adjust speed.
Those databases shouldn't be out of date. Most highway agencies (UK for sure) publish planned works and improvements weeks in advance. Google and Apple both show road closures for example. I belive speed changes are similar.
65 in a 55 is much less dangerous than 85 in a 35. It wouldn’t be common that road conditions would permit very high speeds while at the same time having an artificially lowered speed limit (e.g construction).
This shouldn’t be a problem, in general, with a couple extra sanity checks.
I would expect the correct solution to more commonly have the opposite problem: too slow in a faster zone.
Even so, what's the worst-case scenario in biasing towards lower speed and
assisting your vision system with a knowledge base?
You perform the task of vision 100% of the time driving. From a purely probabilistic standpoint, even assuming highly accurate ML, the relatively infrequent database updates still make for a sensible prior.
This is somewhat adjacent to what Gary Marcus has been arguing, and I think it makes a lot of sense; there seems to be no compelling reason to rely exclusively on primitives (i.e. vision) when good priors are easily accessible.
Out of date is probably not that big a deal. If a speed limit is lowered from 40 to 35, or even 65 to 55, and the database doesn't get updated, it's unlikely to create a dangerous situation if a car doesn't know about it.
But a car thinking a 35mph zone is an 85mph zone... that could be a very big deal indeed.
Regardless, if self-driving cars (or at least "computer assisted driving" cars) were to be widely deployed enough and shown to depend on these sorts of databases, I'd expect it to become a legal requirement to make sure these databases are kept up to date by city/county/state agencies, and that car systems get those updates. Otherwise, expensive liability ensues.
It gets complicated fast. Currently if there is some sort of emergency or requirement to notify road users of a change to driving conditions, a person can just put a sign up. It mostly just works. If road workers are also required to update a database that needs to be read by a car, that’s a lot of infrastructure that needs to be working. You also need to have the humans and machines reading the signs as the same at every time point, as “road closed” or “one way” needs to be correctly understood by everyone at the same time.
In an emergency that may involve bad weather or some other sort of disaster, a remote database is not going to work very well. The car needs to read the actual sign properly.
How much can you trust that data? My car tell me what the speed limit is by using a database and reading signs. On my commute to work there is a part where it tell me the speed limit is 5km/h when it actually is 80km/h and another place where it for a short bit shows the limit as 80km/h where the correct is 30km/h
Both Netherlands and Germany have these time based speeds on a lot of highways where the special case is in text underneath the sign.
BMW handles it if the navigation database is up to date. At least on the A2 in the Netherlands, which has the 06:00 to 19:00 changes between 100 and 130. It also understands temporary changes for roadworks, but gets it wrong when a roadwork/temporary sign has an exception like "when road is wet"
That tells me the camera only understands the main sign and then the navigation database tells it about exceptions / special cases. Nice as a small reminder on your screen, but I wouldn't trust it to self drive based on that data.
Neither is ideal. But you should use what information you have. There is still no excuse for making a decision like 85 in a 35. Worst case this should be a disengage or the car could defer to you - eg visual warning on the dash "speed limit unclear, override?"
In the UK smart motorways can vary from 30mph to 70mph according to traffic. This can change within minutes and obviously cannot be in databases. So this is a good case for sign recognition. In theory though, there is some network that's updating the speed limits and there's no technical reason why that couldn't be a public information service (maybe it already is).
Then you have the sibling comments - speed limits which are time based. I would imagine this is country specific and is, mostly, a matter of public record. But would you trust CV to recognise text on a sign with specific hours? I don't know how complex the signage is, but trying to use ML for this seems absurdly wasteful of resources.
Finally you have other information - how fast are the cars around you travelling? Have you detected an obstruction on the road (eg cones)? Is the weather inclement and therefore you should drive more slowly, etc.
I would imaging that if self driving cars become ubiquitous there will eventually be a system for real time speed limit determination, whether that's some kind of wireless beacon at regular intervals or an online database that's kept up to date I don't know.
How is the accuracy of sign recognition on the LED signs stating the current speed limit? My experience is that it is not so good, maybe other cars are better at reading LED signs.
For the car to react according to how other cars drive, you need to have other cars on the road at the same time. I have a narrow gravel road where the limit is 30km/h but safe driving is between 10 and 20, but my car tell the limit is 50km/h. It can be weeks without me seeing other cars on that road at the same time as me. Some mornings during the winter I can drive because I know where the road is, not because all of it is visible. Good luck having an autopilot doing that
Well, quite. But again this is all supplementary information. The car should have a basic idea of the speed limit from a database and it should adjust as appropriate.
Similarly the car should be able to identify road surface conditions and also how much power it's needing to put in to achieve a set speed. If it's identified that traction is being lost over gravel, then it should reduce power.
Equally the autopilot should consider visibility. There are plenty of roads that are national speed limit and you'd be insane to drive at 60 on them due to blind or extremely tight corners.
But with LEDs this seems like massively over-engineering the problem. Why use computer vision when you can potentially broker a deal with the highways agency to get live speed limits on a particular stretch of highway?
Current Teslas CANNOT have this error simply because MobilEye patented reading speed limit signs and Tesla no longer uses the MobilEye system and haven't since mid-2016.
I think "> 10" is even still pretty optimistic. I'd be surprised and impressed to see driver-asleep self-driving in the next 25 years, unless we see some serious unexpected breakthroughs in computing.
Do the NNs actually need to learn this? I’ve always assumed that NNs primarily being used to do computer vision, but that all the driving decisions would essentially be a fancy rule system.
The ability to accurately map surroundings is a fundamental prerequisite to AGI but it's pretty much the whole meal for self driving.
From there the car needs to segment its interpreted reality and run path finding algorithms.
AGI needs to take that reality and seemingly consciously understand the many different ways that it can interact with it.
I'm oversimplifying the post mapping implementation of self driving - but not so as to remove an insurmountable problem.
Tesla's biggest fuck ups (such as driving under a truck) seem to me to be the consequences of initial mistakes in building a map of reality, rather than in what's being done with that map.
For self driving, as far as I can tell, the map of reality is the hard part.
Is no one pointing out that this is the older Tesla Model S that doesn't even use Tesla's autopilot? It's MobilEye. MobilEye patented reading speed limit signs, so Tesla's current autopilot doesn't even do this.
People saying "see, THIS PROVES self-driving is as hard as AGI" without even applying any critical thinking is just confirmation bias.
You can tell the Model S is old because it has the pre-April-2016 bubble nose. I believe all vehicles with the bubble nose have the old MobilEye system at best (which was never intended for self-driving).
Him: self driving is more than a decade away.
You: No, because this attack is for an old vehicle (implying that newer vehicles are able to evade it). You see, the new cars don't have this feature at all!
Cool, so the new cars have even less features than before. And that justifies that self driving is near?
> I believe all vehicles with the bubble nose have the old MobilEye system at best (which was never intended for self-driving).
If the MobilEye system is not trustworthy then why is the vehicle accelerating based on the data it receives from it? Why doesn't Tesla update the software to stop doing that?
The reason we don't already have self driving cars is that there hasn't been enough computing power in a small package to make the sort of decision logic needed practical until recently.
As long as the prerequisite for self driving has been to emulate human's driving skills, there just hasn't been the computing power available. Now that there is, progress is being made, but really it's just a transitional period.
What's eventually going to happen is that roads everywhere will get machine assists - basically, small beacons running on low power that provide easy machine readable data on everything from precise road location to speed limits to local road conditions. They'll be small, low powered radio transmitters or even NFC devices near the road that the car can just download data from.
There's always been a chicken and egg problem with self driving vehicles, where no one wanted to invest in machine assists for all roads because no one wanted to put that much money and effort into something that no one needed (no one had self driving cars) but no one wanted to make self driving cars a product that wouldn't be successful without machine assists as well as a way to drive without them as a backup.
Now that self driving cars are out there and everyone is getting comfortable with the concept, machine assists will be coming, and they're going to have benefits to ordinary drivers, too.
If you want an example of why this will work, look at aircraft. Most jet liners today are perfectly capable of landing themselves in almost any weather because of the electronic navigation system they have built in to most airports. The planes aren't using machine vision and AI code, they're landing on a piece of ground defined by numbers in the navigation system which is precisely located by GPS coordinates.
The same will work for cars, it just hasn't been built out yet.
Did you ever try to play GeoGuessr game?
I'm not living in USA, I'm living in country which is infamous for its notoriously bad roads (you could say that it is 3rd world country and I will not blame you), so I will not use my homeland as example. But when I'm playing GeoGuessr in USA I often found "myself" on small road with not-so-good tarmac, with wore-out road markings, without any light posts… Google car was here (or there will be no street view), but still.
Are you sure "roads everywhere will get machine assists" soon? Who will pay for that, if now nobody pays to fix tarmac and road markings, not to say about lights installation, which will help EVERYBODY RIGHT NOW?
Airports are completely irrelevant example. There are ~2500 IATA recognized airports out of ~50K "airfields" total, and only ~1000 of these 2500 are equipped with ILS. It is not comparable with millions of kilometers of roads around the planet, even if we will count only "1st world countries".
Your predictions might well be true, but this hack is not any evidence for it.
So you can attack an AI driver with targeted sign changes which wouldn't work with human drivers. I don't think you'll need experimental evidence to agree that you could also attack human drivers with targeted sign changes which wouldn't work with a particular AI driver. Relevant xkcd: https://m.xkcd.com/1958/.
An AI driver will remain dumber than a human until we're very close to AGI, if not forever. But it never gets distracted, and always tries its best to follow the rules. This may well be enough to compensate for its low intelligence - and beyond. The worst human drivers ain't always the lowest IQ ones.
I'm not what you'd call a self-driving car "skeptic" since I do think that we will eventually be able to iron out these types of rules, but I think it will be another 30-50 years before I feel comfortable truly trusting one.
It's nice as a research part. But more people die because of human error and drunk drivers every day.
I have yet to see someone die because of the illegal and "labor intensive" situation they created in this "research".
> Would you feel better if you died due to someone breaking the law?
I can respond that back, but then with problems that are factual ( drinking and driving). Not fictional or harder to execute do it in practice ( this research)
I think the issue isn't always entirely rational. Even if self-driving cars could have a better accident record than humans by several orders of magnitude, incidents would still cause humans to react poorly -- "if I were in control that wouldn't have happened" -- even if it would have.
Altering a speed limit sign may not be legal, but if it's done in a way that wouldn't fool a human, but would fool a self-driving system, that's just a way that a self-driving system could be unsafe that a human driver wouldn't, and people will react negatively toward the self-driving system. That's just human nature, even if it's not logical.
Regardless, we need to be aware of these possible "exploits" on self-driving systems. Sure, no one would bother altering a speed limit sign to cause crashes, because human drivers have context and common sense that would make that mostly ineffective. But if it were known that some popular self-driving systems could be fooled by that, I'd expect to see this happen more often, perpetrated by people who would probably otherwise try to hurt people in other ways. It's not a "new idea" in the context of driving in general, but it's a new -- and potentially very effective -- idea to dangerously mess with self-driving systems.
Why would the legality make this test invalid or unfair? Illegal things still happen. I have seen countless street signs with graffiti on them. I could see rebellious teenagers doing this on purpose without fully understanding or caring about the consequences. Search google images for 'street sign graffiti' and you will see thousands of examples.
There has been an outbreak of modified stop signs near me. Some are quite professionally done with stickers championing pet projects. Stop animal farming, stop eating meat etc. If they weren’t at the end of a street, they would definitely look like a marketing campaign (which I suppose is exactly what they are).
Self driving will need to account for any possible edge case on the road to be perfect. To be better than humans? I don't think so. When a human driver kills someone due to lawbreaking from a third party throwing an edge case that they couldn't handle, we say the driver was "not at fault". We even abuse it for "edge cases" that the driver could obviously handle if not for normal human distraction. Why hold the robot to such a higher standard?
>It's also illegal to do this, so it's not a valid edge-case.
Road signs can get dirty or damaged. There definitely are edge-cases where a self-driving car might not comprehend an imperfect sign which a human would read just fine.
Maybe this test in particular isn't fair, but it does raise questions.
Some sick people get their kicks tossing rocks off of overpasses. Imagine the games people are going to play with cardboard cutouts of whatever or false signage. I predict an arms race.
I live in Atlanta and there's a digital speed limit sign on 75. Sometimes the light is out and when it means to say the speed limit is "55" it just says "5". Come to think of it I've always noticed traffic slowing down nearing that sign. I wonder if any Tesla's have any contribution to that.
I would slow down to a speed limit of 5, even if I was sure it was a defective display. It's effectively an unknown speed limit. Maybe not if I'm very familiar with the road, but not everyone is.
Yes, I'm sure the courts should throw out the ticket, but sometimes they mess up, and often you lose time and money to pursue your case.
While scary, it's not exactly relevant to any Tesla made after they broke off from MobilEye (Which I think was 2016?).
Current Teslas don't read speed limit signs, they access an online database. Vandalize the speed limit signs all you want, as modern Teslas aren't even reading them.
Shouldn’t it know that 85 is beyond the speed limit? You’d think it would know the laws based on where the GPS is, like in some states above 80 is reckless driving. Seems trivial to make this data available.
Very misleading. The car will only accellerate to the speed already chosen by the driver. It doesn't seem odd for itself to accellerate up to speed limits
Nice for them use Tesla but I'm pretty sure you can do it with every sign reading auto throttle system. There are plenty of cruise control systems which employ similar technologies.
Or a person who misreads the sign. Though the whole point is that a Tesla doesn't have the context to know that a residential zone with an 85mph speed limit sign is obviously some sort of error.
You don't need residential context. You just need US context. I have never seen an 85 MPH speed limit sign in the US, ever. Up to 65 is common, with 70 or 75 sometimes on well maintained roads between urban areas. I can't recall seeing an 80, but I feel like maybe once or twice.
There's a pretty long stretch of road between Austin and San Antonio that's 85 the whole way. It's not as big as the I-35, but the higher speed limit means it draws quite a bit of traffic for people traveling between the two cities.
Point stands: 85 is ridiculous in anything like a residential area.
Accelerating to 85mph on a street where that speed is unsafe is absolutely not what a person who misreads the sign would do. 35mph tend to be that way for a reason (residential, stop signs, curved streets)
In particular, Tesla no longer uses sign-reading (it is patented). They use OSM. Using Teslas for this experiment was undoubtedly calculated for the increased exposure it would get.
In the future, road work should go into a database for self driving cars to access in real time. The police and government should have the ability to put in place speed limiting blocks on public roads for various reasons. A good example would be school zone at specific times.
This solution is so unrealistic I almost want to laugh. The local "police and government" barely have any funds to operate, potholes go unfixed for months, streetlights are switched off at night to save money, but yes, someone will keep an eye out to update a database for self driving cars. Yep, I'd predict the chances of that happening here are exactly zero.
It doesn't seem that far fetched to me. When the freeway closes for a wreck, flood, etc, Google maps knows very quickly. Construction projects are planned, so it shouldn't be a stretch for that information to be available online pretty accurately.
Road work or situations where the road is not in good condition will be hard to be put in a dabase realtime. Also, since this is united states, which database? Every state will have a different one.
Either way, my point is digitized information is helpful, but not sufficient
If the police are to be believed (well, one cop who studies this kind of thing), about 90% of drivers choose their speed based on road conditions. Only about 10% are religiously following the signs.
Ah, that's especially good. They just changed the 3 in "35 MPH" speed limit sign to look more like an 8, and the Teslas dutifully sped up. A mistake no human driver would make.
Rodney Brooks points out that if we ever get AGI [1], we'll have solved the autonomous vehicle problem. But it's far from clear to me that we'll truly solve the problem much before then, as cars and roadways are built with GI expectations in mind.
And probably prosecutable as felony manslaughter, the same way removing a stop sign would be, because you're acting in a way you know will get people killed.
According to the article, only when they engage TACC for the first time while the sign is visible, not if they were driving for awhile and it was already turned on.
Also note that TACC isn’t autopilot, one just controls your speed automatically, the other actually drives your car. It wasn’t clear if the flaw affected auto pilot as well, since it can’t be used for non freeway driving.
So basically they modified a 35MHP street sign to make it look like a 85MPH.
I looked at it from a distance, and it was hard to tell between 35 and 85.
So if it is hard for a human, then it seems obvious that a tesla would also fail to recognize a 35.
Maybe the difference is that a human might not be certain and slow down just in case, whereas a deep neural network might be certain in its errors.
A human has the basic intuition to assume the sign itself might be wrong or misleading, the programmers of the car automation failed to realize that in a way that might've gotten someone killed.
There's several factors they could use to bring a level of uncertainty to the algorithm. Compare against the previously clocked speed limit, compare against map data, compare against the speed of other vehicles in a proximity, judge based on density of crosswalks, parked cars, traffic, etc.
This isn't a failure of autonomy imo. Speed limits are arbitrary placeholders that are more for supplementing the slow reaction speed of the human brain. If we started focusing more on making the algorithm implement actual analysis of the environment instead of "the rules of the road", we might see better results.
> So if it is hard for a human, then it seems obvious that a tesla would also fail to recognize a 35.
You need to be a special type of stupid to think that a road with a speed limit of 35 MPH is 85 because of a road sign. The 'drive into the lake because the GPS nav tells me to' level of stupid.
Yes ... the older Teslas have the MobilEye chipsets which use this technology. Many other cars use the same MobilEye chipsets today .. so potentially could be an issue in other cars as well.
Its likely now that this error was pointed out that Tesla may add some speed limit validation code to ensure a car doesn't speed up much when a speed limit jumps by more than 25% or something like that..
It only works when this is the first sign TACC sees. If TACC was previously engaged and saw other signs previously, the exploit doesn’t work. I’m sure Tesla fixed this, but given the nature of the edge case, it wasn’t simply a matter of sanity checking.
Maybe because they were probably already doing sanity checking, but the kind that relied on some previous context? That seems reasonable, I’m sure they’ve thought about this kind of attack before, missing an obscure edge case.
"Front-facing cameras detect speed limit signs on AP1 vehicles and display the current limit on the dashboard or center display. Limits are compared against GPS data if no signs are present or if vehicle is HW2 or HW2.5"
My understanding is the limits come from OSM. They seem to be relatively accurate, at least in my area. The display doesn't always change exactly where the sign is, and sometimes it's off by a quarter mile, but usually it's closer than that.
I do know of one instance where modifying street signs tricked humans: there's a short speed-trap section of road outside a rural town north of me that's labeled 25 mph even though the approach roads are 45 mph. Apparently it's zoned residential even though there are only like 5 houses set way back from the road (looks like it should be 45, maybe 35). Seems like someone in the 70s thought that zoning something for higher density caused high-density housing developments to arrive.
Of course, this section was where the cops liked to sit. When I was in high school, some kids pulled legitimate signs off a different section of road and re-labeled it 45 mph. While the average speed of traffic to school in the morning sped up dramatically for a couple weeks, the police did not agree that the presence of these signs changed the speed limit on the road. Fortunately, the perpetrators got off with a slap on the wrist, unfortunately, the county didn't review and revise the speed limits.
>the police did not agree that the presence of these signs changed the speed limit on the road
Now this would be an interesting legal challenge. The official wording will vary by state but it'll be something like
"a person must not, when the sign is in place on the highway, drive or operate a vehicle on a highway at a greater rate of speed than that indicated on the sign for that category of motor vehicle, unless another sign on a specific highway in the defined area so indicates"
So... can you really be faulted if there's a not-obviously-fake speed limit sign? If I got a ticket for that I would definitely be disputing it.
They literally unbolted the 25 mph signs and dumped them in the ditch... I think there was still a "speed limit reduced ahead" sign.
You'd have a pretty easy case, I think, the only defenses would be "there's a sidewalk here so that's typically 25" and "you drove here before so you should remember it's 25".
Isn't sign reading necessary for say, construction areas, that would have a lower speed limit? If so, researchers could do the same thing by placing false construction signs.
But at least the software could assert that the construction speed was lower than what the map stated the speed limit was so you could only slow down a car which would hopefully be a safer failure.
Cars slowing down when they shouldn't be can create dangerous situations, though I'd agree it's relatively safer when compared to speeding (particularly in residential areas, or roads not made for faster travel).
This is actually one of my biggest gripes with autopilot. It's aggressive at following reduced speed limits, even when traffic is not. I don't know the solution, since the speed limit really does drop to 25-35 mph in a couple spots in our local freeway interchange, but traffic flows through at 55-65 mph all day long, and having the car suddenly brake to 30 mph is unexpected. If there's one thing you want to be on the road, it's predictable. It's when you deviate that accidents happen.
It might be necessary for higher level self-driving, but AFAIK Tesla isn't doing that due to patent issues. Autopilot is just very fancy cruise control, and you're supposed to be in charge. You'd be expected to change the cruise control set point manually when you enter a construction area.
Putting aside the fact that this was found on an old Tesla with autopilot 1.0, this is not even a technically hard problem to solve. The current version of autopilot uses a database and gps to determine the speed limit. In the future if Tesla wants to return to a vision based speed limit system they can sanity check the vision readings against their database and throw out results like 85mph.
If the point is that autonomous cars are succeptable to deliberate malicious actors, I've got bad news about the rest of the world. Human drivers are incredibly succeptable to me throwing a $2 brick throught the front windshield as well. It sounds like the issue is already resolved in this case.
I guess the difference here is the scope of how many you can impact.
In order to get all Teslas who go past a sign on the highway, you just have to modify the sign slightly and everyone is affected (in theory).
In order to affect the same amount of people with your "brick through the windshield" strategy, you'll need a lot more manpower than just a sticker on a sign.
That depends on the size of wreck you manage to cause, don't you think? In any case, it takes no more people to drop a brick than it does to sticker a sign.
To be clear, I find this entire concept of unsupervised robot vehicles both dangerous and absurd - even more so than the already dangerous and absurd baseline of a society so intimately bound up with automotive travel as ours. Exceeding such a high baseline as that is in its way impressive, and certainly demonstrates the astonishing overconfidence rampant in some segments of our very young and rather careless industry, but let's not get distracted from that essential point and waste our efforts on inconsequential arguments over whether a sticker is more dangerous than a brick.
> That depends on the size of wreck you manage to cause, don't you think?
No, I think it's the scope of affecting many after another without doing something more. Throwing bricks requires continues action while a sticker is a thing you do once and then it "does it for you".
See it as working every day and getting paid for that, versus a savings account where you get returns without really doing anything.
And sorry, I didn't really join the conversations to argue against "concept of unsupervised robot vehicles [is] both dangerous and absurd " so I agree with the rest of your message.
What if this wasn't malicious. What if someone had taped something over the sign earlier, like for construction, and removed all but a tiny bit of tape later?
Different state's Department of Transportation seem to do this themselves! When adjusting lanes on highways, they will paint over the old stripes with "black" paint. During good lighting conditions it is clear what you are supposed to do. But as soon as conditions are worse, like heavy rain in low light, all I can see are crisscrossing lane markers!
So a malicious actor could put black "tape" down on a highway, and just wait for the right conditions for people to get confused.
I wanted to link that not because of the "there's an XKCD for everything" meme but because it makes an interesting point: sabotaging roads isn't difficult now, with human drivers. There's no reason to assume it would suddenly become a substantial threat when autonomous cars are commonly adopted. An issue worth considering and accounting for, but not worth public worry.
What I like about Tesla is they have over the air updates using a fleet of network data to quickly address issues or exploits that may arise... but this article is quite funny exposing the gap between AI and basic human intuition. I do believe human intuition may just be another model to train from in some sense in the end.
>The safety of Tesla's autopilot features has come under close scrutiny, but CEO Elon Musk has predicted the company will have "feature-complete full self-driving" this year.
I think that is a real possibility if we are referring to years on Jupiter (which has an orbital period of 12 Earth years).
Sounds like an easy fix in most cases. Residential road or road work? Can't be 85, period. There are relatively few roads in the US where 85 is the actual speed limit.
That's pretty common IME though the ones I've driven didn't make decisions based on that.
Last few cars I've rented had a speed limit sign generally matching the speed limit on the dash, and on the one I rented this christmas passing a speed limit sign (or an other sign triggering a speed limit change) while on limiter or cruise control would flash a message suggesting pressing a button twice to change the configured speed to that (rather than manually adjust using the +/- buttons).
They were mediums (C-segment) and compact MPV not large or luxury, which I expect is why the sign reading was mostly to purely indicative.
Yes, many cars do now. Some of the "less smart" cars don't necessarily know which signs are which. My 2018 Accord thought the speed limit was 100mph when I was on Highway 99.
The human operating the vehicle who reads a maliciously edited sign. It seems unquestionable to me anybody modifying street signs would be held liable for the modifications.
140 comments
[ 2.9 ms ] story [ 214 ms ] threadIt sounds like the Tesla is just using what the camera sees - which is bad if true. Admittedly this was tested on a race track, so maybe there is no data saying otherwise (or even the opposite).
And yes, you could special case this case in the code, but there are hundreds of cases where "common sense" is used in driving, and you will never teach your NNs all of it.
Prediction: no actual "self driving" (to a point where driver can legally be asleep) on public streets till > 10 years from now
Road construction, for example is a reason why
Roadworks are an obvious situation where you drive at a _lower_ speed. This would be a case where you could use computer vision to identity an obstruction, identify roadworks and then adjust speed.
Those databases shouldn't be out of date. Most highway agencies (UK for sure) publish planned works and improvements weeks in advance. Google and Apple both show road closures for example. I belive speed changes are similar.
This is literally Google's back yard
This shouldn’t be a problem, in general, with a couple extra sanity checks.
I would expect the correct solution to more commonly have the opposite problem: too slow in a faster zone.
You perform the task of vision 100% of the time driving. From a purely probabilistic standpoint, even assuming highly accurate ML, the relatively infrequent database updates still make for a sensible prior.
This is somewhat adjacent to what Gary Marcus has been arguing, and I think it makes a lot of sense; there seems to be no compelling reason to rely exclusively on primitives (i.e. vision) when good priors are easily accessible.
But a car thinking a 35mph zone is an 85mph zone... that could be a very big deal indeed.
Regardless, if self-driving cars (or at least "computer assisted driving" cars) were to be widely deployed enough and shown to depend on these sorts of databases, I'd expect it to become a legal requirement to make sure these databases are kept up to date by city/county/state agencies, and that car systems get those updates. Otherwise, expensive liability ensues.
BMW handles it if the navigation database is up to date. At least on the A2 in the Netherlands, which has the 06:00 to 19:00 changes between 100 and 130. It also understands temporary changes for roadworks, but gets it wrong when a roadwork/temporary sign has an exception like "when road is wet"
That tells me the camera only understands the main sign and then the navigation database tells it about exceptions / special cases. Nice as a small reminder on your screen, but I wouldn't trust it to self drive based on that data.
In the UK smart motorways can vary from 30mph to 70mph according to traffic. This can change within minutes and obviously cannot be in databases. So this is a good case for sign recognition. In theory though, there is some network that's updating the speed limits and there's no technical reason why that couldn't be a public information service (maybe it already is).
Then you have the sibling comments - speed limits which are time based. I would imagine this is country specific and is, mostly, a matter of public record. But would you trust CV to recognise text on a sign with specific hours? I don't know how complex the signage is, but trying to use ML for this seems absurdly wasteful of resources.
Finally you have other information - how fast are the cars around you travelling? Have you detected an obstruction on the road (eg cones)? Is the weather inclement and therefore you should drive more slowly, etc.
I would imaging that if self driving cars become ubiquitous there will eventually be a system for real time speed limit determination, whether that's some kind of wireless beacon at regular intervals or an online database that's kept up to date I don't know.
For the car to react according to how other cars drive, you need to have other cars on the road at the same time. I have a narrow gravel road where the limit is 30km/h but safe driving is between 10 and 20, but my car tell the limit is 50km/h. It can be weeks without me seeing other cars on that road at the same time as me. Some mornings during the winter I can drive because I know where the road is, not because all of it is visible. Good luck having an autopilot doing that
Similarly the car should be able to identify road surface conditions and also how much power it's needing to put in to achieve a set speed. If it's identified that traction is being lost over gravel, then it should reduce power.
Equally the autopilot should consider visibility. There are plenty of roads that are national speed limit and you'd be insane to drive at 60 on them due to blind or extremely tight corners.
But with LEDs this seems like massively over-engineering the problem. Why use computer vision when you can potentially broker a deal with the highways agency to get live speed limits on a particular stretch of highway?
Current Teslas CANNOT have this error simply because MobilEye patented reading speed limit signs and Tesla no longer uses the MobilEye system and haven't since mid-2016.
Is this not how most self-driving works?
From there the car needs to segment its interpreted reality and run path finding algorithms.
AGI needs to take that reality and seemingly consciously understand the many different ways that it can interact with it.
I'm oversimplifying the post mapping implementation of self driving - but not so as to remove an insurmountable problem.
Tesla's biggest fuck ups (such as driving under a truck) seem to me to be the consequences of initial mistakes in building a map of reality, rather than in what's being done with that map.
For self driving, as far as I can tell, the map of reality is the hard part.
For AGI, it gets you to the hard part.
People saying "see, THIS PROVES self-driving is as hard as AGI" without even applying any critical thinking is just confirmation bias.
You can tell the Model S is old because it has the pre-April-2016 bubble nose. I believe all vehicles with the bubble nose have the old MobilEye system at best (which was never intended for self-driving).
Cool, so the new cars have even less features than before. And that justifies that self driving is near?
If the MobilEye system is not trustworthy then why is the vehicle accelerating based on the data it receives from it? Why doesn't Tesla update the software to stop doing that?
The reason we don't already have self driving cars is that there hasn't been enough computing power in a small package to make the sort of decision logic needed practical until recently.
As long as the prerequisite for self driving has been to emulate human's driving skills, there just hasn't been the computing power available. Now that there is, progress is being made, but really it's just a transitional period.
What's eventually going to happen is that roads everywhere will get machine assists - basically, small beacons running on low power that provide easy machine readable data on everything from precise road location to speed limits to local road conditions. They'll be small, low powered radio transmitters or even NFC devices near the road that the car can just download data from.
There's always been a chicken and egg problem with self driving vehicles, where no one wanted to invest in machine assists for all roads because no one wanted to put that much money and effort into something that no one needed (no one had self driving cars) but no one wanted to make self driving cars a product that wouldn't be successful without machine assists as well as a way to drive without them as a backup.
Now that self driving cars are out there and everyone is getting comfortable with the concept, machine assists will be coming, and they're going to have benefits to ordinary drivers, too.
If you want an example of why this will work, look at aircraft. Most jet liners today are perfectly capable of landing themselves in almost any weather because of the electronic navigation system they have built in to most airports. The planes aren't using machine vision and AI code, they're landing on a piece of ground defined by numbers in the navigation system which is precisely located by GPS coordinates.
The same will work for cars, it just hasn't been built out yet.
Airports are completely irrelevant example. There are ~2500 IATA recognized airports out of ~50K "airfields" total, and only ~1000 of these 2500 are equipped with ILS. It is not comparable with millions of kilometers of roads around the planet, even if we will count only "1st world countries".
We already have "driver-asleep self driving" on public streets: https://www.theverge.com/2019/12/9/21000085/waymo-fully-driv...
So you can attack an AI driver with targeted sign changes which wouldn't work with human drivers. I don't think you'll need experimental evidence to agree that you could also attack human drivers with targeted sign changes which wouldn't work with a particular AI driver. Relevant xkcd: https://m.xkcd.com/1958/.
An AI driver will remain dumber than a human until we're very close to AGI, if not forever. But it never gets distracted, and always tries its best to follow the rules. This may well be enough to compensate for its low intelligence - and beyond. The worst human drivers ain't always the lowest IQ ones.
It's also illegal to do this, so it's not a valid edge-case.
I have yet to see someone die because of the illegal and "labor intensive" situation they created in this "research".
> Would you feel better if you died due to someone breaking the law?
I can respond that back, but then with problems that are factual ( drinking and driving). Not fictional or harder to execute do it in practice ( this research)
Altering a speed limit sign may not be legal, but if it's done in a way that wouldn't fool a human, but would fool a self-driving system, that's just a way that a self-driving system could be unsafe that a human driver wouldn't, and people will react negatively toward the self-driving system. That's just human nature, even if it's not logical.
Regardless, we need to be aware of these possible "exploits" on self-driving systems. Sure, no one would bother altering a speed limit sign to cause crashes, because human drivers have context and common sense that would make that mostly ineffective. But if it were known that some popular self-driving systems could be fooled by that, I'd expect to see this happen more often, perpetrated by people who would probably otherwise try to hurt people in other ways. It's not a "new idea" in the context of driving in general, but it's a new -- and potentially very effective -- idea to dangerously mess with self-driving systems.
If selve driving vehicles become popular. The exploiter would have been recorded somewhere, identified and put to trial.
Story would go viral and the cat and mouse game continues.
No one is going to risk jail for letting cars drive 80 miles per hour, until the next road sign.
This is a stupid exploit.
Road signs can get dirty or damaged. There definitely are edge-cases where a self-driving car might not comprehend an imperfect sign which a human would read just fine.
Maybe this test in particular isn't fair, but it does raise questions.
Yes, I'm sure the courts should throw out the ticket, but sometimes they mess up, and often you lose time and money to pursue your case.
Ironically, current Teslas are largely immune to this as they use a database of speed limits.
Current Teslas don't read speed limit signs, they access an online database. Vandalize the speed limit signs all you want, as modern Teslas aren't even reading them.
There's a pretty long stretch of road between Austin and San Antonio that's 85 the whole way. It's not as big as the I-35, but the higher speed limit means it draws quite a bit of traffic for people traveling between the two cities.
Point stands: 85 is ridiculous in anything like a residential area.
Either way, my point is digitized information is helpful, but not sufficient
I obviously don't have any data, but I think this behavior is strongly correlated with IQ.
Rodney Brooks points out that if we ever get AGI [1], we'll have solved the autonomous vehicle problem. But it's far from clear to me that we'll truly solve the problem much before then, as cars and roadways are built with GI expectations in mind.
[1] https://en.wikipedia.org/wiki/Artificial_general_intelligenc...
And probably prosecutable as felony manslaughter, the same way removing a stop sign would be, because you're acting in a way you know will get people killed.
Also note that TACC isn’t autopilot, one just controls your speed automatically, the other actually drives your car. It wasn’t clear if the flaw affected auto pilot as well, since it can’t be used for non freeway driving.
So if it is hard for a human, then it seems obvious that a tesla would also fail to recognize a 35.
Maybe the difference is that a human might not be certain and slow down just in case, whereas a deep neural network might be certain in its errors.
This isn't a failure of autonomy imo. Speed limits are arbitrary placeholders that are more for supplementing the slow reaction speed of the human brain. If we started focusing more on making the algorithm implement actual analysis of the environment instead of "the rules of the road", we might see better results.
You need to be a special type of stupid to think that a road with a speed limit of 35 MPH is 85 because of a road sign. The 'drive into the lake because the GPS nav tells me to' level of stupid.
Abstractly, it's interesting, if only because it probably wouldn't trick a human -- everyone would see the 85mph as BS regardless of the defaced sign.
Has anyone hacked into openstreetmaps yet and fiddled with speed limits?
Its likely now that this error was pointed out that Tesla may add some speed limit validation code to ensure a car doesn't speed up much when a speed limit jumps by more than 25% or something like that..
https://en.wikipedia.org/wiki/Tesla_Autopilot#Speed_assist
From what I've gathered elsewhere, the GPS data is OSM.
https://wiki.openstreetmap.org/wiki/Speed_limits
I do know of one instance where modifying street signs tricked humans: there's a short speed-trap section of road outside a rural town north of me that's labeled 25 mph even though the approach roads are 45 mph. Apparently it's zoned residential even though there are only like 5 houses set way back from the road (looks like it should be 45, maybe 35). Seems like someone in the 70s thought that zoning something for higher density caused high-density housing developments to arrive.
Of course, this section was where the cops liked to sit. When I was in high school, some kids pulled legitimate signs off a different section of road and re-labeled it 45 mph. While the average speed of traffic to school in the morning sped up dramatically for a couple weeks, the police did not agree that the presence of these signs changed the speed limit on the road. Fortunately, the perpetrators got off with a slap on the wrist, unfortunately, the county didn't review and revise the speed limits.
Now this would be an interesting legal challenge. The official wording will vary by state but it'll be something like
"a person must not, when the sign is in place on the highway, drive or operate a vehicle on a highway at a greater rate of speed than that indicated on the sign for that category of motor vehicle, unless another sign on a specific highway in the defined area so indicates"
So... can you really be faulted if there's a not-obviously-fake speed limit sign? If I got a ticket for that I would definitely be disputing it.
You'd have a pretty easy case, I think, the only defenses would be "there's a sidewalk here so that's typically 25" and "you drove here before so you should remember it's 25".
In order to get all Teslas who go past a sign on the highway, you just have to modify the sign slightly and everyone is affected (in theory).
In order to affect the same amount of people with your "brick through the windshield" strategy, you'll need a lot more manpower than just a sticker on a sign.
To be clear, I find this entire concept of unsupervised robot vehicles both dangerous and absurd - even more so than the already dangerous and absurd baseline of a society so intimately bound up with automotive travel as ours. Exceeding such a high baseline as that is in its way impressive, and certainly demonstrates the astonishing overconfidence rampant in some segments of our very young and rather careless industry, but let's not get distracted from that essential point and waste our efforts on inconsequential arguments over whether a sticker is more dangerous than a brick.
No, I think it's the scope of affecting many after another without doing something more. Throwing bricks requires continues action while a sticker is a thing you do once and then it "does it for you".
See it as working every day and getting paid for that, versus a savings account where you get returns without really doing anything.
And sorry, I didn't really join the conversations to argue against "concept of unsupervised robot vehicles [is] both dangerous and absurd " so I agree with the rest of your message.
What if it was mud? Or salt? Or bird shit?
So a malicious actor could put black "tape" down on a highway, and just wait for the right conditions for people to get confused.
I wanted to link that not because of the "there's an XKCD for everything" meme but because it makes an interesting point: sabotaging roads isn't difficult now, with human drivers. There's no reason to assume it would suddenly become a substantial threat when autonomous cars are commonly adopted. An issue worth considering and accounting for, but not worth public worry.
I think that is a real possibility if we are referring to years on Jupiter (which has an orbital period of 12 Earth years).
Next step: sql injection.
https://hackaday.com/2014/04/04/sql-injection-fools-speed-tr...
Last few cars I've rented had a speed limit sign generally matching the speed limit on the dash, and on the one I rented this christmas passing a speed limit sign (or an other sign triggering a speed limit change) while on limiter or cruise control would flash a message suggesting pressing a button twice to change the configured speed to that (rather than manually adjust using the +/- buttons).
They were mediums (C-segment) and compact MPV not large or luxury, which I expect is why the sign reading was mostly to purely indicative.
No matter what, the answer is probably both. You can't modify street signs now anyway. You also can't drive 85 in residential/city areas.
You must react to the environment as well...