My amateur interpretation of this is that the vision algorithm briefly confused the yellow of the headlights of the oncoming car as the double yellow line in the road, which caused it to believe it was seriously out of lane. After verring, the algorithm reoriented, and realized an impact was imminent and diverted off the road.
I think what's interesting about this example is that in this case, an alert human driver would not be confused in this way. It shows issues with the particular machine learning approach to automated driving.
I don't think that a root cause analysis like this is good for anything else than improving the algorithm.
It's the same as with face detection. The algorithms are better far superior to humans, but they fail at occasions where a human would not even think twice
>vision algorithm briefly confused the yellow of the headlights of the oncoming car as the double yellow line
not the lights, but the vertical lens flares captured by single front facing center mounted camera cluster - afaik Tesla still has no stereo vision.
>the algorithm reoriented
more like driver jerked to the right while having a heart attack and seeing itself going into a full frontal collision thanks to move fast and break things mentality
Oh dear. FSD (Fools Self Driving) Beta once again causing havoc and putting the life of this driver at risk and deciding to hit another car after the driver regaining control of the vehicle very quickly and going off road, especially in the most dangerous time to drive, which is at night.
Had the driver not kept their eyes on the road or was sleeping in the car, this system would have crashed into that incoming vehicle. This video just proves that night-vision is non-existent in this 'safety-critical' system.
Not from the exterior. When full self driving beta mode is active, there is a circular, blue steering wheel icon visible at the top of the car's control screen. So unless you're inside the Tesla, you won't be able to tell.
I would not bet on this continuing to be the case, mind you. If regulators allow this sort of thing to continue at all, I'd expect warning lights visible to pedestrians and other drivers will be required.
Sure. So far I’ve driven maybe 500k kilometers and I’m doing pretty okay so far. Twice a deer at low speed and that’s about it.
Every time when my car issues AEB, I’m ahead of it mentally and AEB only interferes with what I want to do. But I’ve driven race cars on Nordschleife so I guess I’m an outlier here.
What situation is the computer going to do objectively better? I don't really care to swap any number of survivable accidents for one situation like that.
It does seem like I'd have to maintain the same level of alertness and attention as when driving, in order to have the continuity of awareness needed to react. Including having my hands on the wheel.
I should know this, but don’t… when the vehicle is in computer driving mode with your human hand on the wheel, does the wheel still turn?
I assume for mechanical and practical reasons it does, but I don’t know to what degree it will “fight you”. How much pressure is needed to override the system? If I’m holding it and refuse to let the wheel turn, I assume it gives up and computer mode turns off? But there has to be some degree of resistance it’ll overcome right?
Yea, that’s kind of what I thought it would be like. Hard to tell how much of that is fighting, and how much is “tactile wiggle alert” where it’s not steering that input but is wiggling.
They can wiggle alert safely if it’s full electronic steering and not connected to a tie rod. IDK if that is the case, typically sucks for feeling the road.
Video description contains details about what happened, which might not be clear from the video snippet. FSD swerved into oncoming traffic, and the driver manually pulled back to the correct lane, but lost control and went off the road.
Is there corroboration that this actually is self-driving? I suppose we'd need Tesla's cooperation. Is it possible to replay the "AI view" of the same thing? Can any FSD beta users chime in?
I will say that two lane highways (aka offset impact at 110mph+ IF/BIG IF they aren't speeding) are pretty damn dangerous and I'd only do self-driving on them with a LOT of repeats. I think I'm getting close to the point of trusting self-driving on divided highways under good conditions, the big caveat being stopped cars/obstacles.
It sure doesn't help that every Tesla post on HN has comments that seem like obvious astroturfs. It's like jalopnik in the comments.
Transparency is a must. The new NHTSA head doesn't help, we do need someone to hold Tesla's claim to the light, but appointing someone with publicly posted anti-Tesla screeds? That doesn't help anything.
Self driving could save tens of thousands of lives (perhaps hundreds of thousands worldwide) each year and be a legitimate boon to the economy. NHTSA needs a dedicated group of government officials with legit industry and software experience to test, regulate, and analyze incidents. Plus it would be apocalypse to one of my less favorite sectors in the world: the auto insurance industry.
NHTSA should have immediate delivery of Self-Driving accidents with all the footage and any recorder information immediately uploaded to government servers with an active software harness to replay the logs. And it's not like this would be occaisionally tested, even with high-quality theoretical self driving, there will be daily accidents to replay from every significant vendor.
"Self driving could save tens of thousands of lives (perhaps hundreds of thousands worldwide) each year..."
Launching a self driving car which crashes head on into traffic is not a good start. If they used unmaned Teslas for the live traffic tests they would be risking less lives. /s
It is interesting that consumers kind of volunteer to test a potentially lethal product. I say "kind of volunteer" because they are actually paying for the car.
can we get the "frontal impact == impact at sum of speed" myth die already? it's been thoroughly debunked over and over. the only way it's relevant is if there's a large weight difference between vehicles.
There's just something I don't like about the word "bug". It suggests a condition of the software, not a condition of the developer. Maybe it's the full phrase "caused a ... bug". As if the bug wasn't present until the condition occurred. Unless that condition is actually altering the commands in the software, the bug was already there. I could even talk about what "weird" means in this context. It's not a poke at you - I heard phrases like "caused a weird bug" all the time in our industry. In generally, and especially in situations involving safety, it's important we not engage in minimization, but call it was it is: a flaw, where conditions in the operating environment weren't properly considered.
I'm suspicious that this may not have occurred as described for 2 reasons: the recording doesn't show the part of the screen where the autosteer indicator is, and the original uploader took down the video as soon as it got significant attention.
that wasn't a crash! the car simply decided that the turn was dangerous and took it upon itself to disrupt the road construction industrial complex by forging a new and safer path.
This is $TSLAQ (you can see from the higher tweets)
It's disappointing HN is part of something so lame or to stupid to know they are part of it.
If you have money invested and are part of the hack, perhaps not lame. But most dumb lamers are doing it just to follow.
You can see anti-TSLAQ will get flagged/downvoted by the "bots" on HN (This account is shadowbanned, it's hard to use search for examples since they get flagged)
It doesn't surprise me that it's this particular scenario, I've seen a lot of Youtube videos of them going way over the center line on turns in this kind of road (2 lane rural with 55 mph) even in broad daylight
Don’t forget with this sensational “headline” we don’t actually know the truth. We can’t see the driver in the video and we have only his word to go by. Statistically it’s much more likely he’s lying versus FSD causing the accident in this situation. FSD has a long way to go but it’s already demonstrably safer than humans.
Regardless, Hope this guy is okay and wasn’t injured in the accident.
For what it is worth, I want to try FSD beta, and have a safety score of 99, and can’t.
You have to jump through so many hoops to even have FSD beta right now, that this is the devil’s deal (i.e. potential manslaughter) you accept by even trying it.
Tesla should investigate, and continue to keep FSD in beta with these strong obstacles while it improves. And eventually (arguably), it needs to put up or pay up for level five FSD which it has been selling as vaporware for something like 5 years now. Maybe this is where public and private law needs to step in.
But in my opinion, casual outrage over FSD is unwarranted until Tesla rolls out software that is less safe than humans to a lot of its users that will operate it as if it is level 5 regardless of actual performance.
There are loads of videos of Tesla self driving fails, like this one posted yesterday, where the Tesla tries to hit a pedestrian and prevents the driver from taking control, tries to drive on the train track, and stops for the Wells Fargo sign.
Flagged this. Some random video with no official investigation and countless speculations.
Keep in mind that Tesla will sue your ass for defamation if you post BS video and lie about it. Just ask the Chinese dude who posted about Tesla accelerating by itself into a crash. Turns out he stepped on the pedal himself but of course when someone on twitter posts about a Tesla fuck-up, it’s taken as gospel.
Yes, I’ll happily jump to Tesla’s defense (not that they need it) when baseless crap is thrown around, because for every 1 of me, there are 100 who can’t wait to see Tesla crash and burn.
61 comments
[ 7507 ms ] story [ 591 ms ] threadI think what's interesting about this example is that in this case, an alert human driver would not be confused in this way. It shows issues with the particular machine learning approach to automated driving.
It's the same as with face detection. The algorithms are better far superior to humans, but they fail at occasions where a human would not even think twice
not the lights, but the vertical lens flares captured by single front facing center mounted camera cluster - afaik Tesla still has no stereo vision.
>the algorithm reoriented
more like driver jerked to the right while having a heart attack and seeing itself going into a full frontal collision thanks to move fast and break things mentality
Had the driver not kept their eyes on the road or was sleeping in the car, this system would have crashed into that incoming vehicle. This video just proves that night-vision is non-existent in this 'safety-critical' system.
Outrageous.
Not per situation where you think you would have done better ignoring the ones where the computer would have done objectively better
Every time when my car issues AEB, I’m ahead of it mentally and AEB only interferes with what I want to do. But I’ve driven race cars on Nordschleife so I guess I’m an outlier here.
https://amp.theatlantic.com/amp/article/620808/
I assume for mechanical and practical reasons it does, but I don’t know to what degree it will “fight you”. How much pressure is needed to override the system? If I’m holding it and refuse to let the wheel turn, I assume it gives up and computer mode turns off? But there has to be some degree of resistance it’ll overcome right?
They can wiggle alert safely if it’s full electronic steering and not connected to a tie rod. IDK if that is the case, typically sucks for feeling the road.
Video description contains details about what happened, which might not be clear from the video snippet. FSD swerved into oncoming traffic, and the driver manually pulled back to the correct lane, but lost control and went off the road.
I will say that two lane highways (aka offset impact at 110mph+ IF/BIG IF they aren't speeding) are pretty damn dangerous and I'd only do self-driving on them with a LOT of repeats. I think I'm getting close to the point of trusting self-driving on divided highways under good conditions, the big caveat being stopped cars/obstacles.
It sure doesn't help that every Tesla post on HN has comments that seem like obvious astroturfs. It's like jalopnik in the comments.
Transparency is a must. The new NHTSA head doesn't help, we do need someone to hold Tesla's claim to the light, but appointing someone with publicly posted anti-Tesla screeds? That doesn't help anything.
Self driving could save tens of thousands of lives (perhaps hundreds of thousands worldwide) each year and be a legitimate boon to the economy. NHTSA needs a dedicated group of government officials with legit industry and software experience to test, regulate, and analyze incidents. Plus it would be apocalypse to one of my less favorite sectors in the world: the auto insurance industry.
NHTSA should have immediate delivery of Self-Driving accidents with all the footage and any recorder information immediately uploaded to government servers with an active software harness to replay the logs. And it's not like this would be occaisionally tested, even with high-quality theoretical self driving, there will be daily accidents to replay from every significant vendor.
Launching a self driving car which crashes head on into traffic is not a good start. If they used unmaned Teslas for the live traffic tests they would be risking less lives. /s
It is interesting that consumers kind of volunteer to test a potentially lethal product. I say "kind of volunteer" because they are actually paying for the car.
can we get the "frontal impact == impact at sum of speed" myth die already? it's been thoroughly debunked over and over. the only way it's relevant is if there's a large weight difference between vehicles.
It's disappointing HN is part of something so lame or to stupid to know they are part of it.
If you have money invested and are part of the hack, perhaps not lame. But most dumb lamers are doing it just to follow.
You can see anti-TSLAQ will get flagged/downvoted by the "bots" on HN (This account is shadowbanned, it's hard to use search for examples since they get flagged)
Regardless, Hope this guy is okay and wasn’t injured in the accident.
https://www.motorbiscuit.com/tesla-beta-testers-say-full-sel... https://www.inputmag.com/tech/tesla-recalls-12000-cars-over-... https://www.roadandtrack.com/news/a35878363/teslas-full-self... https://electrek.co/2021/11/15/tesla-serious-phantom-braking...
https://www.businessinsider.com/tesla-full-self-driving-beta... (with video!)
Or just look on Youtube for videos of people using the FSD, and see how frequently they have to stop it from doing something dangerous.
You have to jump through so many hoops to even have FSD beta right now, that this is the devil’s deal (i.e. potential manslaughter) you accept by even trying it.
Tesla should investigate, and continue to keep FSD in beta with these strong obstacles while it improves. And eventually (arguably), it needs to put up or pay up for level five FSD which it has been selling as vaporware for something like 5 years now. Maybe this is where public and private law needs to step in.
But in my opinion, casual outrage over FSD is unwarranted until Tesla rolls out software that is less safe than humans to a lot of its users that will operate it as if it is level 5 regardless of actual performance.
Fail to buy real tyres
Or is Tesla’s stability control so bad that it underperforms 2 decade old economy cars?
https://mobile.twitter.com/TaylorOgan/status/146940457943982...
Keep in mind that Tesla will sue your ass for defamation if you post BS video and lie about it. Just ask the Chinese dude who posted about Tesla accelerating by itself into a crash. Turns out he stepped on the pedal himself but of course when someone on twitter posts about a Tesla fuck-up, it’s taken as gospel.
The experience of “your Tesla tries to do something INSANE” is shared by literally every Tesla driver I know.