Who could've ever predicted this scenario? A great example of something that's trivial for human beings to understand but where ML training sets will come up short.
That’s what’s frightening about this. I feel not enough people internalize just how primitive the car’s understanding/perception of the world is. A two year old human brain would have no confusion in this scenario.
We discount how much the concept of “understanding” is required in visual perception. We don’t just see shapes we have a complete understanding of what we’re seeing.
I might see a big rectangle flopping in the lane in front of me. I can immediately ascertain whether it’s a piece of tumbling plywood, or foam based on movement characteristics, color, apparently size, etc. I can then use that understanding to decide what evasive actions are required.
A Tesla it seems has absolutely zero of this capability.
I'm very curious how a LIDAR based system would have faired here. It very much looks like problem that would only appear with a limited vision-only system like Tesla has.
One time, I fell asleep as a passenger in a car. I hit a bump, looked up, and screamed as I saw what looked like a car coming right for me.
It was a tow truck towing a car backwards. It was just enough in my half-asleep state to scare the shit out of me.
Humans have millions of years of evolution behind our visual processing systems. We have developed hacks that prevent our brain from getting tricked by unusual situations like traffic lights on trucks or backwards cars being towed. We only developed the first computers a hundred years ago, and only in the last 40 years have a small subset of people started learning about visual processing systems.
It's easy to look at this video and scoff because of how trivial it seems. But it's instead a marvel of our minds that we can pick up on context clues so quickly and accurately that such oddities basically never puzzle us. Given the pace of our innovation, it wouldn't surprise me if our computer systems match ours within a few human generations at latest.
>One time, I fell asleep in a car. I hit a bump, looked up, and screamed as I saw what looked like a car coming right for me.
>It was a tow truck towing a car backwards. It was just enough in my half-asleep state to scare the shit out of me.
There are some funny(?) youtube videos of people in the passenger seat waking up to this with tractor trailers being towed in reverse.
Except we're trusting ML to perform surgery, choose conviction sentencing, evaluate job CVs, determine acceptable marriage partners (why not?), determine who can have kids (why not?), determine who gets into college (why not?), determine who gets a loan (why not?), determine who gets to work on ML (why not?). And drive cars.
> The data subject shall have the right not to be subject to a decision based solely on automated processing, including profiling, which produces legal effects concerning him or her or similarly significantly affects him or her.
Ah. Sounds great. But unfortunately I live in the US, the consumer's paradise.
Question on your snippet though. It says "subject to a decision based solely on automated processing ..."
Since I do live in the consumer's paradise I'm naturally suspicious. I wonder if people will sometimes be subject to decisions based solely on automated processing, and if they don't bother to ask for a human in the loop after the automated decision, then that's that.
And how would you even know, one way or the other?
We still use electricity even though it's deadly and it accidentally kills people every year. Why not use ML even if it has it's own issues? As humans who know better we're responsible to take all the precautions.
It's a fair question. One difference is that electricity is not making decisions or suggestions on the fly.
Another example is roads, where the design of the road allows one to turn into an opposing vehicle. The road itself is not deciding to turn your car into a semi trailer because it looks like sky, but an ML driven car has, multiple times.
I don't even own a Tesla, and I resent having to participate in its beta program. ML in a car is quite different from a brake recall, or even a stuck accelerator software recall.
In case of electricity, the precautions are relatively simple and well known. What are the precautions with ML-powered self driving? If the precaution is "requires constant human supervision" then it's a joke.
Tesla clearly needs to hire a "Red Team" to find weaknesses of their autopilot (and other systems).
The odd thing is that even a randomly stupid AI for self driving is statistically safer than most drivers. Clearly there is a ton of room for improvement in both AI and Humans.
> The odd thing is that even a randomly stupid AI for self driving is statistically safer than most drivers.
This hasn’t been shown yet at all. Statistics showing autopilot have less crashes per mile always ignore that Autopilot is doing the type of driving that has the least accidents per mile (motorway driving).
> The odd thing is that even a randomly stupid AI for self driving is statistically safer than most drivers.
A strong claim, lacking actual evidence for it. All we have to go on are some Elon tweets (rather the definition of a biased source) and the actual crash rate. Without a lot more data (which Tesla steadfastly refuses to release) about environments, corrections, etc, it's quite impossible to make that sort of statement with any confidence.
The Tesla hardware is a weird combination of capable and insanely dumb, and it's far from obvious which it will be in any given situation until it's gone through it.
If an honest statistical analysis of the data indicated that Tesla's automation was better than human drivers (or better than other driver assist systems), I would fully expect them to have released the values. Since they haven't, and only hint at it and make statements that sound statistical but really aren't, I assume they've done the numbers internally and know it's not nearly as good as they like to imply.
If I drove in a city like their "self driving" beta was a few months back, I would be hauled from the car on suspicions of driving while hammered.
> The odd thing is that even a randomly stupid AI for self driving is statistically safer than most drivers. Under sunny highway conditions? Perhaps. In a night snowstorm? Probably yes - because the AI would be at the side of the road waiting for the human to drive.
I think self driving is a typical 80/20 problem. We won't have "full" self driving because the costs are exponential for each step closer to it. But driving on 80% of roads on 80% of days, with supervision? That could happen.
But that said: we won't accept AI that just makes traffic safer "on average". I'm fine with human shortcomings causing accidents. People will not accept car manufacturers cutting corners and causing accidents, even if statistically it's safer. So the very high bar for self driving isn't just "as safe as humans".
Using Tesla's "autopilot" isnt very different than working as an unpaid driving instructor to keep an eye on the "autopilot" trainee. To make things worse, the trainee is tripping on mushrooms and sees nonexistent cars appearing from nowhere, trees morphing into pedestrians and pedestrians morphing into traffic lights. The trainee also has a problem with epilepsy and youre expected to take control on a second notice. Edit: also, the trainee has cognitive ability of a mentally challenged frog.
I actually don’t want autopilot for the fast, freeway speed driving. That’s easy. I want it for the mundane, exhausting stop-and-go, bumper to bumper traffic as we all slowly creep through the traffic.
I think only the new Honda L3 system(only on one very limited edition car in Japan) and a Subaru tech on some mainstream cars(in the US too) have this capability.
Subaru Eyesight goes down to zero and handles stop and go with lane centering. Though I guess provided you've started the adaptive cruise control system above the minimum speed.
It's not smooth though. Once traffic starts again, Eyesight accelerates hard and then brakes hard when it get up to the car, because it's using cruise control and trying get up to speed and there is a minimum speed (25mph? maybe 10mph? i forget) to set the cruise control.
Not sure if it’s different now, but older models activated Lane Keeping Assist at 45mph and keeps lanes down to 40, and always seems to be just out of reach of usefulness in the city, perhaps by design.
LKA is just that, an assist, because if you let it drive it’ll bounce back and forth between the lane markers, not keep you seemingly centered.
My mom's Subaru Forester does hands-off stop-and-go traffic from 0 to 90 mph. It has 2 cameras and does lane departure too. It will even activate antilock breaks if it needs to panic stop.
My Toyota Corolla with Toyota Safety Sense 2.0 (fitted in nearly every Toyota post 2018/2019) has stop and start, either press resume if you've stopped for more than 5 seconds or tap the accelerator, and that's with lane tracing assist as well. I don't think there's a second I don't have both Adaptive Cruise Control and Lane Tracing Assist activated.
Edit: You can also accelerate without disabling it.
I feel like I missed a huge generational gap driving a 2017 toyota. None of this lane assist, no backup camera, the only new bell and whistle I get that's different from my car 20 years ago is that this one beeps at me like I have a missile locked on my tail when someone is too close in front of me.
Best part is it's standard on nearly every model. I don't think there's a minute when I don't have it enabled and it's full range so works at any speed and road. The millimetre wave radar is great when road markings aren't visible or when cars are manoeuvring around parked cars or other obstacles.
Naturally. In my manual North American market Miata I have basic cruise control but if you touch the clutch or gear selector it disengages cruise. I can’t imagine a safe system that would work in stop and go traffic with a manual transmission.
I only use it in the rare times I'm driving on a quiet highway for long multi-hour stretches. That's definitely not my preferred way of driving, as I usually trade around a half hour of extra travel time to take more winding, fun, and scenic routes.
The current VW Golf R has adaptive cruise control available with the manual transmission. I test-drove one. Not sure whether it does stop-start or whether this applies to other trim levels. Unfortunately it was not available yet when I got my 2015.
That's all Autopilot is, adaptive cruise control and lane keeping. Autopilot doesn't react to stoplights, stop signs, or basically anything else. That is a separate feature set that they misleadingly call Full Self Driving.
It was lost due to Tesla's decision to remove MobilEye as a supplier.
Mobileye generally focuses on highly optimized HW/SW that does individual things very well, in a manner similar to how factory automation works (e.g., they basically built a "lane keeping + auto-braking + sign reading" appliance).
Tesla decided that 1) It was bad to outsource automation 2) Starting from scratch and 'learning' how to drive using ML was better than iteratively teaching a car how to do discrete tasks very very well (this is why Autopilot regressed a bunch in 2016).
In general, it's another symptom of Elon's 'I have an extremely specific idea, let's figure out' mentality that sometimes works and sometimes results in useless tunnels under Las Vegas.
May I ask, is traveling through a bumper to bumper traffic a frequent occurrence for you? If so, are there options for you to avoid it (e.g. change commuting times, work from home, walk, bike, or use public transportation)? If also yes, why do you tolerate it?
Personally I see bumper to bumper traffic maybe 3 or 4 times a year (when driving home from a vacation or being forced to a doctor’s appointment during rush hour). And I honestly don’t get why anyone would subjugate them self to this kind of traffic as part of their daily commute.
Try living in New York City, the whole tristate, not just Manhattan. 3 to 4 times a year is laughable. Try 30 to 40 times a year if you take the Belt Parkway.
I used to live in LA where Santa Monica Blvd was always backed up. I doubt LA traffic has gotten better either.
I'm surprised by your insolence with regard to how shitty traffic circumstances are in big cities. Simply changing one's commuting times doesn't failsafe the issue.
Large parts of NYC (much of the outer boroughs) are not near the subway, and not within walking distance of a supermarket (especially if you're carrying groceries for an entire family). Millions of people live there because they can't afford Manhattan rents, especially for a family-sized apartment.
Also, transit lines tend to connect well to mid-Manhattan but poorly between other locations. So if you live in Queens and work in Brooklyn, good luck getting to work reliably by public transportation. (Before you object to that arrangement, consider: If you own a house and have kids in school, you're not necessarily going to uproot your family and move just because your new job is further from home.)
Thus, many ordinary New Yorkers rely on cars to commute to work.
I did a quick google map survey and found that it can take about an hour to commute on public transit between a residential area in Queens and to a commercial area in Brooklyn[1]. Not ideal until you see that driving the same route takes about 40 min. So not a huge different and definitely passes as an alternative to avoid the risk of stop-and-go traffic jams.
If you need to drive to the supermarket you should have the option of choosing a time and route with minimal risk of traffic jams. I find it hard to belief that many people are frequently hitting bumper to bumper traffics on their way to or from the supermarket. Occupationally yes, but frequently no.
Some parts of Brooklyn are much nicer than the ratnest known as Manhattan. Manhattan is so overcrowded. It's a bad place to raise a family. Someone can literally shoot or stab you in broad daylight and easily get away with it. All they have to do is merge into the crowds of pedestrians. Then perhaps they descend into one of the many subway lines, conveniently escaping, scot-free.
when first living in NYC I had a station wagon, previously gifted to me. Not being from the area I couldn't believe there wasn't a decent supermarket. I had another friend out in Long Island near one. I got several Manhattanite friends to come along, none having ever driven. The experience was hilarious, all around. From the traffic, the experience of a wallowing beast of a Pontiac wagon on the LIE, and then the best part, one guy who'd literally NEVER left the City absolutely MARVELED at the selections as the store. Notable quotes being "I've only seen this brand on TV!" and "How could anyone EVER need a TWO-GALLON SIZE tub of laundry detergent?"
This question was specifically addressed to those that have the option of avoiding it and still don’t. I was under the impression that there were several options of escaping bumper to bumper traffic on your daily commute in the New York City area.
In fact I’ve often heard people from that area complain more often about lack of parking near their commuter rail station. Which indicates that people do rather tolerate circling the parking lot in their park-and-ride rather then risking stop-and-go traffic jams.
What do you mean by "have the option?" There is not an option to avoid the random traffic bottlenecks causes by construction, accidents, road closures, and just generally poor infrastructures and even worse drivers. The traffic lights have also been increased in both count and magnitude to induce even more delay. The speed limits have been lowered. The road conditions are abysmal, potholes, cracks, and unevenness abound. You know nothing Jon Snow.
Consider yourself very lucky. Not everyone has the luxury of working from home, choosing their working hours or living close to public transit or within walking/biking distance from their job (in many areas, the farther you get away from city centers, the cheaper the real estate generally is).
In the NYC area, bumper to bumper traffic is common. It can be caused by an accident or construction that blocks one or more lanes, cars merging on to an already congested highway, etc. These conditions frequently happen even outside of rush hour.
> I honestly don’t get why anyone would subjugate them self to this kind of traffic as part of their daily commute.
You may have trouble understanding it, but empirically a huge number of people see this extremely regularly, if not daily.
It's not even necessarily a feature of horrifically long commutes. For one example, lots of places that have basically ok traffic have bottlenecks at bridges, you may be stop and go for a little while every day getting across that.
The first thing that crossed my mind when I saw the clip of autopilot hallucinating flying traffic lights was that it was clearly DUI and there’s no way I’m getting in the car with that guy.
That's basically true, but there's no reason to limit it to Autopilot. Although the high points of AP are better than any other system that I've tried so far. I've seen it do amazingly well in the rain at night, and in the rain on a road with faded markings that were hard to pick out. I've also seen it thoroughly confused by double-dashed lines at hov lanes.
I'll still take that over the ones that beep proudly to tell you that they are about to fail to maintain their lane (hi ProPilot).
> I've seen it do amazingly well in the rain at night
Not anymore, since they removed the radar. Rain seems to really interfere with the vision based system, and apparently auto high beams are required at night, and they flash constantly.
love react because HN only does upvotes. I feel like they're straying from their mission. They did really well with moving the market but now that the market is moving, I have a hard time believing Tesla will stay at the top.
Sure, in 5 years we will see what shakes out. I am not taking my entire savings and investing in tesla stock or anything.
That said, i want a cool fast BEV that makes highway driving easier. Right now, Tesla is the only option. A few others are close. If someone else beats Tesla to FSD with another system, I will be happy to go that way on my next car. Working FSD is worth a lot to me.
It's worth noting that Subaru vehicles have been doing vision-only adaptive cruise / lane keeping / emergency braking for many years, which they brand Eyesight. In my personal experience its adaptive cruise is as good or better than most other cars I've driven which use radar.
(Admittedly the reasons why the Subaru impressed me was more for its programming rather than absolute precision. I liked how smoothly it transitioned between following and open road ahead. Much less jerky.)
Rain really interferes with human vision based systems as well. It's insanity that we don't better design road markings to better compensate. It's often the case where old painted-over lines are more visible on wet roads than the current markings. Making road information systems standardized and better for humans works out great for autonomous cars.
Yeah, I know. The vision based one seems to be more limited in the rain at the moment. I'm questionable as to whether this will be a long term problem, since it does so well at picking up lane line markers. It seems like a solvable issue.
The high beams are a different story. I only like to use auto-high-beam on rural roads as they tend to be problematic on busy highways in Tesla and my non-Tesla. But requiring them even on highways would be a big problem that I don't foresee them solving soon. I'm worried that this is going to take a long time to fix.
IMO, they actually work pretty well on rural roads.
It's fascinating to watch this technology develop, but as a driver I usually feel more anxious using "autopilot" that not using it.
With one huge exception: self parallel-parking.
I don't understand why this innovation doesn't get more love. Tesla's not the only manufacturer to offer this, of course, but this particular innovation has increased m enjoyment of city driving more than ... well, more than anything I can think of.
Even if they never get anything else to work, it would have been worth it just for self parallel-parking. lol
From an engineering perspective, of course there are serious problems that need attention, but sometimes it also good to celebrate the wins.
That’s better than dead grandma who is leaning her foot on the gas just so to get you to keep the same speed aka cruise control. Dead grams is in many many cars.
As an ML engineer I'm a little bit baffled that Tesla has not solved this by now. It's not like they lack data or ML knowledge.
It seems like they should have a million hard test cases that must pass in simulation before releasing a new model. The simulations should be harder and more extreme than anything encountered in real life.
I think the real problem is obvious. They're trying to rush the work because Elon said so.
Years ago, a friend worked in the Autopilot group. It took them a year to procure servers to store the telemetry of the existing cars, and then weeks to have them setup.
They don't work there anymore.
From their experience, I know one thing: I will never work for Elon Musk. He may be a great visionary and salesman, but he's a horrible manager.
Server procurement was a CFO thing, and their specific requirements didn't fit under the existing buckets, so it took them a long time to get it approved.
It was stunning to see the complete disconnect between Musk's grand declarations and what the organization was actually setup to deliver.
Frankly, it just gives me more respect for Tim Cook, who as COO at Apple made his company able to turnaround and deliver HW in record time.
Edit: in retrospect I wonder if Musk's grand public declarations were actually a way to control and pressure his own organization. Remember, Musk didn't actually found Tesla, he rescued it from bankruptcy after the Roadster didn't return as much as needed, so he inherited an existing structure.
Just like your anecdote, I have one to share as well: I know a close friend that works on space lasers at SpaceX and has a blast, best job ever according to him and he thinks Elon is an excellent manager mostly because there is zero bureaucracy and people are not afraid of "Do nothing" option as well as removing complexity. In fact, he wouldn't work anywhere else after seeing the company culture.
Elon spends nearly as much time at SpaceX as he does Tesla. There's no question that Elon regularly mucks in at the lowest levels of engineering at SpaceX.
If you want evidence from a reasonably neutral observer, take Sandy Munro (himself an engineer who has worked on everything from cars to aeroplanes). He recently interviewed Elon, ostensibly about Tesla but the interview was in a meeting room at SpaceX. After the interview he was invited to a two hour design review meeting and was "blown away" at Elon's depth of involvement.
Yes. The human brain and visual systems aren't nearly so trivial to replicate as a lot of people in the tech industry seem to think.
Tesla is just one of many case studies in the paired tech industry arrogance seen so frequently:
- "A human is just a couple really crappy cameras and a neural network, we know how to do better cameras and neural networks, how hard can it be?"
- "We can do anything we dream with 99.995% reliability in the synthetic, computer-based world of the internet because we know code. Therefore, we can do anything we want in the physical reality with code!"
Both are far from evident in practice, but the belief in them continues, despite it being increasingly obvious to everyone else that neither one is true.
Human vision and world processing is quite impressive - and, as pointed out elsewhere in this thread, a two or three year old would have no trouble working out that the obstacles were some things on a truck. I've got a nearly three year old, and I guarantee he wouldn't confuse those for stoplights in the slightest. I also wouldn't let him out on the road, though he does well enough with a little Power Wheels type toy. But there is far more going on in the visual processing system than we even understand yet, much less have the slightest clue how to replicate.
And while code may be fine on the internet (where you can retry failed API calls and things mostly make sense), the quote about how fiction is constrained by what's believable and reality sees no such restrictions is very true. Out on the roads, all sorts of absolutely insane things can and do happen on a regular basis - and you can't predict or plan for all of them. But the car has to handle them or it crashes.
As a random example, a year or two ago, I was behind a car that had poorly strapped a chunk of plywood to their roofrack with a good chunk hanging forward, and the front end of it was starting to oscillate awfully hard. I had a good clue that it was going to come apart sometime in the very near future, so backed off from a normal following distance to quite a way back. Sure enough, half a mile later, it failed, went flying through the air, slammed into the road a good distance behind the car, and tumbled a bit. Had I been using a normal in town following distance, it would have either hit me or tumbled into me, but using a human visual system, it was obvious that my existing following distance stood a good chance of being a bad idea.
Stuff like this happens on roads constantly. Meanwhile, state of the art self driving can't tell the difference between stoplights and some poles on a truck. You'll excuse me if I don't think the problem is anywhere remotely close to solved for a general case Level 4 purpose.
Personally I view the problem here not as a failure of object recognition or what would be considered a visual system, but of abstract reasoning (or lack thereof, of course)
Aritical neural networks are pretty good at object recognition, among hundreds of other things, and even better than humans at some of them. They are, however, generally pretty bad at abstract reasoning, critical thinking, 'understanding' concepts in-depth, and so on, and I think that's a more constructive way to phrase the problem we see in this video.
When a problem is fully redicible to a simple vision problem, modern neural networks are a great choice, but being a good driver involves much more than just the visual cortex.
The problem, to the extent there is one, is certainly with the visualisation; it’s not so clear if there’s any problem with the underlying vision system.
This, analogous to God of the Gaps, is a kind of "Humanity of the Gaps". As AI advances we redefine humanity to be those small gaps that are still left unconquered.
I've never understood this argument. Isn't the main bottleneck that you need well-labeled data for training the neural networks? How is having tons and tons of random camera footage going to help?
For a lot of problems, clean data labeled data is no longer the bottle neck, or in some cases, there are ways around it. The bigger issue is dealing with the "long-tail" of unknown new scenarios. This is a currently unsolved challenge.
> How is having tons and tons of random camera footage going to help?
One way to think of this is that that the footage is implicitly labelled: we have the benefit of hindsight: we know what the state/location of the vehicle was going into the future. That benefit of hindsight also can serve as implicit labels by knowledge that the vehicle did not crash or collide with something immediately after the footage.
Fairly simple, they have footage from every time a person had to correct AP's driving. Take that footage label it. Train some of it, save some for test cases. Finally don't release an update until it drives better than the current system.
Right? This should be trivially solved with some sort of temporal filtering on detected objects. If you detect a traffic light at (x,y) in one frame, and it disappears in the next, but there's a new one at (x+dx,y+dy), then you shouldn't place a new one down in the world frame. You should only place a traffic light down if you're confident it exists and is operational. At the very least, the lights should be detected on the back of the truck, but they should move with the truck. At least that matches what's happening.
I don't understand why this is hard for Tesla engineers -- I was doing this kind of thing in grad school a decade+ ago and it worked fine. I've seen it in other demos where object classifications rapidly cycle between person, bike, car, etc. Are they not filtering anything? Is this a symptom of "AI-ing all the things"? Because we did it with bog standard computer vision techniques back then and never got behaviors like this.
Did any of your grad school work make it into the real world at all? Typically I'd suggest that such ideas are simple in theory and difficult in practice.
I'm curious what the issue is here that seems unsolved? It's unconventionally displaying what it is recognizing, but the car isn't doing anything janky.
The car is properly recognizing traffic lights pretty darned well, considering the circumstance. It looks like it has a built in understanding that traffic lights are "always" stationary - hence, assigning them static locations on the 3D map - but it keeps having to update the model because the lights are actually moving.
This seems like a very non-obvious edge case that I wouldn't expect an ML team to even consider as a possibility. Now they need to program into the ML model an understanding that traffic lights are typically stationary. Which seems even more difficult to me, from a technical perspective - you don't want false negatives...
The car isn't braking or making any strange maneuvers from what I can tell. I'm actually impressed that it's handling it this well.
I don't know exactly how Tesla's safety systems are designed, but this is my guess.
Collision detection systems (radar) are accurately not detecting an impending collision because the lights are not actually on a collision course with the vehicle.
Object recognition systems (computer vision) are working very well, because they recognize the lights and are updating the 3D map accordingly, but the traffic light 3D model is not designed to be a moving object - unlike vehicles, which frequently move. Which is why we see the car "passing through" them.
What we are likely seeing is simply a weird edge case in the output for the user-interface. I'd imagine if an object was actually flying at the car and the car could see it, it would brake accordingly.
Also, the map is two-dimensional. The car frequently drives underneath traffic lights that I'm sure also appear "on top of" the car in normal cases.
Object recognition and collision detection, from what I understand, are two very different systems.
Tesla is not using any radar/lidar systems for FSD/autopilot (anymore), it's all visual. It might be that there are completely different systems for recognizing obstacles and what is shown on the map, but this still raises the question why and why one of those systems seem to act like this.
I bet the car always sees the traffic lights at 5-10m distance ahead, but the graphics display is not capable of displaying a moving traffic light correctly. It's not a lidar problem, it's a UI problem.
They’re unlit traffic lights. Would you rather the car slam on the brakes?
But seriously, I’m inclined to be charitable here and assume that this is merely a quirk of the UI display. There’s no evidence that the autopilot did anything unsafe (apparently it wasn’t even engaged?), and until I see evidence of that I’m willing to withhold judgment. (I have seen evidence of other situations where Tesla autopilot did unsafe things and I’m in no way apologetic about those situations.)
I believe other commenters have said that autopilot doesn't respond to stop lights, but aside from that: if a car sees an unlit stoplight it probably should stop. An unlit stoplight could mean that a bulb is burned out, or that there's a power failure in the area. In either case, continuing through an intersection is unsafe because you can't guarantee the cross-traffic is yielding right-of-way.
In this case, obviously it's not a safety concern because the object is being hauled on the back of a truck.
I routinely drive on roads where traffic lights are active only at certain times, and when they are unlit the appropriate behavior is to ignore them. Also, these unlit traffic lights were in the back of a truck, and should definitely be ignored.
Are the hours marked? Everywhere I've lived expected me to treat a broken stop light as a stop sign, not just YOLO through. (They tape a trash bag over it until they're finished installing it.)
As another ML engineer I have all the respect for Andrej Karpathy and wouldn't dare to walk in his shoes. My ML model could kill people? No thanks, I couldn't sleep anymore.
Is the car trying to stop? The visualization here is for autopilot (which ignores traffic lights), so even if the truck driver was trying to do something malicious, the car would ignore it.
I'd wager the average case of a non-lit traffic light is more likely to be a traffic light that's... out (at least around here newly installed ones are covered until they're activated) so no, I wouldn't say ignoring them would make sense.
It would generally be a clue that there's an intersection busy enough to require signals that now lacks signals or signage which would warrant extra caution. I'd expect the car to at least slow down significantly, if not come to a complete stop before proceeding.
Autopilot is not engaged during this video. You can tell because the steering wheel icon is grey, meaning AP is available. If it was engaged, it would be blue. The driver is driving manually.
That the Autopilot world model is extrapolating these lights to follow the passing geography in ~500ms cycles, similar to how game netcode sends players running in their last known direction when encountering packet loss, is an interesting insight into how the system works.
I recently test drove a Tesla just to see how the autopilot system works. The way they handle traffic lights is pretty entertaining. It seems them, particularly yellow lights, in a lot of situations that are truly perplexing. It also had a tendency to turn trees into traffic cones and to be truly impressive at detecting garbage cans. On some level it's hard to understand exactly what they're trying to do there.
I also remember being at an intersection where I was turning left and was waiting behind another car. The display repeatedly showed cars the cars crossing in front of us crashing into the car in front of me. Not sure why.
When I got my Tesla back in November 2019, when stopped at traffic lights, cars next to me would be shown to rotate randomly and jiggle and dance.
It works much better now, but it was always kind of amusing. It only did it when we were all stopped. Moving cars (whether it was them moving or me) were rendered just fine. My guess is that it was extrapolating movement based on more than one "frame" of sensor data, but at a stop there's no changes beyond noise, and the noise was being extrapolated into extreme movements.
I have seen it show cars "crashing" into me, but it's only semis when in the neighboring lane.
I am fairly confident that the autonomous car business model is pretty dead. Electric cars - that's great ! But self driving is waaaay harder. And self driving has a inbuilt expectation of nearly perfect. It's not that if we swapped to self driving cars today we would halve car deaths (which would be something like 500k people pa - which would be a huge very important thing) but it would not be perceived as robots save 500k people a year, it would be instead "robots kill the other 500k per year"
Liability, insurance, legal minefields and plain old marketing would never allow cars ,that perform as well as the best performing cars do today, to be on the roads in "every day" conditions.
My conjecture is we end up building AV only roads. initially one lane of a highway, then ring roads round cities and major warehousing hubs, then across urban areas. Walled off in some way they simply become railways with benefits.
At that point every business model ever written with Self Driving in the title goes in the bin.
I am not saying the tech is useless - frankly it's fucking awesome that this is happening in my lifetime. But fucking awesome tech and workable business model aren't always the same thing.
Not sure where I am going with the rant but I sure hope we get more out of the billions spent here than Teflon.
Nope. You're just not patient enough. ACs will come, but very gradually. They're already (mostly) here but it will take another decade or two to be completely A.
Lol, I guess. ACs aren't the same as the Moller Skycar. It's a very, very complicated "DARPA Grand Challenge" where the goal is to not get sued by Ralph Nader into oblivion because the algorithm went MCAS and killed grandma. Waymo or someone will need to do racing and the Gumball 3000 first before people will accept anything more than semi-A (glorified cruise control and lane following).
I think the issue is bigger than just self-driving. AI/ML is wildly oversold as an engineering solution to real world problems. In the best case, it's solving easy problems under narrowly defined conditions. It's not really solving hard problems robustly under real conditions.
I see this rather differently. In fact AI is already as good as human drivers, statistically. Only a few years back, we could point at people getting killed by software bugs like the Tesla path-under-semi-trailer issue (not a lot, but it did happen and they were real bugs).
That's not happening any more. All we have left is laughing at stuff like this, where the visualization (not even the autopilot!) gets confused by seeing real (!) traffic lights on a truck, so it paints them in space, but then has to re-recognize them because they are moving.
At some point, the luddites will just run out of ammunition. It's sort of happening already.
But the AI drives slowly and gets confused easily. Regular drivers routinely have to go around self-driving cars. Not to say they won't improve, but it seems like current AI is assistive to the point where it might be harmful when drivers rely on it in speeds and situations where they shouldn't. I'm sure it will keep improving, but I feel like this is one of those situations where the amount of data and training required, and the amount of iteration on the software required to handle edge cases is not impossible but is exceptionally difficult.
> But the AI drives slowly and gets confused easily.
Probably better to say that the AI drives slower than you would and gets confused differently. Real drivers die every day to terrible mistakes that look obvious in hindsight. Most of them probably thought AI was terrible too.
The bar here is much lower than you think it is, and frankly existing automation has already crossed it.
> the amount of iteration on the software required to handle edge cases is not impossible but is exceptionally difficult.
If you want to fix this one exception (false positive) you will introduce unwanted false negatives. That's how precision and recall work, there's a trade-off. So I am not sure it's useful to fix it, how many times does this happen? Will fixing it introduce more frequent bugs?
Do you have source for the statistics? I remember reading somewhere that it was just a number one of the AI projects said, with no verification, also with no knowledge of the setup.
> where the visualization (not even the autopilot!) gets confused by seeing real (!) traffic lights on a truck, so it paints them in space, but then has to re-recognize them because they are moving.
Just my hypothesis: but I think the autopilot really did see them as traffic lights, and just got lucky that they weren't powered and ignored them as out of order. Were there a cross street, I suspect the car would have stopped and treated it as an uncontrolled intersection...
Just as with real brains, "the autopilot" isn't a unitary thing. Different parts do different things. The ML that recognizes objects is different from the software that makes decisions as to which recognitions to honor. In this case, all we have is evidence for the first half: the vision system detected (correctly!) images of traffic lights, which then got presented by the UI visualization (correctly!) in the right place.
But the software model is that traffic signs are static, and these were moving, so the visualization had no way to present that. It just left them there until the ML told them they weren't there anymore.
What would the AP have done? We don't know. But I don't know why we should simply assume it would have done the wrong thing. Driving is filled with false-positive indicators that it already knows to ignore successfully.
> Only a few years back, we could point at people getting killed by software bugs like the Tesla path-under-semi-trailer issue (not a lot, but it did happen and they were real bugs).
That's not happening any more.
It doesn't need to be almost perfect. Waymo, for instance has remote human assistance when it gets stuck, and even a human in a normal car drive to the scene and to get it unstuck if it's really bad. There could well be a point where those humans plus the tech are cheaper than a traditional full-time-per-customer human taxi driver.
You might say people don't want those delays but we tolerate delays in normal commuting. An unattended bag on a tube station stopping trains, a car accident blocking traffic, traffic jams blocking traffic, mechanical breakdowns, etc. As long as they're infrequent enough, it should be OK for riders.
Au contraire, seems we're mostly there. It's incredible having watched self-driving, and neural networks, go from new concepts and barely functioning to "order here" and public use (tell me the numerous YouTube FSD videos aren't what they are). Insofar as there are still serious edge cases to address, they're being solved.
Time and again I've watched "ain't happening" technology become the preferred norm practically overnight. Eagerly awaiting my FSD CT, and making long trips without having to micro-manage every foot across thousands of miles.
I have been saying for years that full self driving is extremely unlikely to be developed any time in the near future because it is simply too hard of a problem for any reasonable near-future extension of currently available technology. I am gratified to see that the conventional wisdom has recently come to also see things that way. I am not surprised that journalists bought the hype - I assume that half of them have no idea what they are writing about and/or are just advertisers in disguise. But it does surprise me that so many experienced corporate professionals and investors got taken in. I guess maybe part of it is that they just are not as smart as I once thought they were and maybe another part of it is that some of them used the hype to grab a bunch of cash with both hands, cash that the people who put it up will now never see again.
> My conjecture is we end up building AV only roads. initially one lane of a highway, then ring roads round cities and major warehousing hubs, then across urban areas. Walled off in some way they simply become railways with benefits.
That would be awesome still and make people want to have that technology...how does that murder Tesla's business model? Waymo would easily pivot to licensing their tech to car manufacturers.
Disclaimer: I work on the business side of the autonomous team at a major OEM.
The autonomous car business model is very much alive and well, it has simply shifted in two key areas that cause general consumers to misinterpret what is happening.
1) Use cases (ODD). Many people think "if I can't buy an autonomous car, it's not a real thing". But most manufacturers and AV developers other than Tesla aren't even trying to tackle that use case. Instead, the focus is on autonomous vehicles operating very profitably in specific locations and operating models that work. Heavy trucking on "easy" routes, ridesharing in well-mapped areas of the Southwest US, shuttles on popular metro and airport routes, etc. These use cases represent very large revenue opportunities and the tech development is progressing well.
2) Timing. Too many pundits and CEOs looking to generate PR buzz made ridiculous claims over the last few years about when autonomous vehicles would be ready. This set false expectations among the general public which has now soured perceptions since autonomous tech is not easy. Fully driverless is already ready in some very specific scenarios (ex. Waymo in Chandler) and is on track to roll out to more ODDs over the next 5-7 years. I can pretty much guarantee that you will see a significant number of fully driverless vehicles in suitable areas by 2030. No, they won't drive in upstate New York in a blizzard, but no one actually cares about that from a business perspective.
The other issues you raise - liability, insurance, legal roadblocks, etc. are mostly non-issues that already have solutions. The only one that's a continuing problem is the fragmented nature of legislation across different states. But there is very heavy lobbying going on right now to rectify this by the time it's actually needed.
Thank you for the comments. I am actually writing a book (The Software Mind) covering, well, quite a lot of ground, and would be interested in getting a real perspective on this area - can I reach out to you?
Edit: I would be interested in real experts correcting my poor understanding of important subjects - can we talk ?
> My conjecture is we end up building AV only roads. initially one lane of a highway, then ring roads round cities and major warehousing hubs, then across urban areas. Walled off in some way they simply become railways with benefits.
Close, but I think it's far more probable that we just whitelist roads / intersections / routes known to work well and be reliable--whether with a general-purpose or location-specific algorithm. There's still a ton of value shuttling people to and from the airport and local hotels.
Then there'll be incentive for local restaurants to also be included in the available destinations, so there'll be pressure to both improve local roads / intersections to make them more easily navigable by self-driving cars, and on self-driving cars to get better at navigating them. With time, the locations / routes that self-driving cars can reliably reach will expand until it covers the vast majority of desired destinations.
We actually are following the same pattern with the horses -> cars transition. There are still plenty of places that cars can't reach and horses can. Over decades, though, we just kind of paved roads everywhere anyone really wanted to go. Nowadays it's more or less a given that if you want people to be able to reach somewhere, it'll need to have a road.
We'll see the same thing with self-driving cars. There'll still be roads that self-driving cars avoid and humans drive on, but they'll become less and less relevant as the value of self-driving cars becomes more apparent.
it is very unfortunate that the first & really good electric vehicle is a highly complicated tech gadget. and this sets a bad benchmark for the other companies which want to step into this space. why not a simple electric vehicle : just different fuel and engine , and all the rest same as ICE vehicles . wouldn't that be easier / simpler to deal with ?
sadly, in India too, the major electric 2wheeler Ather, is kinda going the same high-tech way as Tesla. and very expensive !
I agree with your sentiment that I'd like to see EVs made for normal people like me, but I'm grateful that people with lots of money fund R&D on stuff like Tesla's to help bring the price down to where I can get it one day. Kinda reminds me of smart phone evolution.
A big part of Tesla's success was making electric vehicles that were covetable, rather than being obvious "we made this for compliance reasons or to get a government contract" products. This means targeting a luxury market segment. The way they chose to wow luxury consumers was with high performance and a lot of technological gimmickry.
I don't need autopilot until it is 100% (or 99.999%) but I'll be damned is cruise control "auto pilot" in a stop and go commute is nice and works well enough. that takes a bit of the grind off the drive to work.
I still have hope, because we have two different mentalities at play.
The individual still thinks in terms of "I'm an above-average driver, I'm better than a silly bleep-bloop box." They'll resist self-driving en masse if it's not 1000% perfect.
Insurers, however, will see self-drivers as predictable, never drunk, always following the speed limit, and in aggregate preferrable to humans. They will be able to see "3% better" as meaningful even if it's not perfect, and price accordingly. Eventually, you'll pay a rapidly increasing penalty rate for having a steering wheel. This is where the avalanche comes from.
I also figure self-driving could unlock new features-- in particular I'm imagining the only way we get road speeds much over 120kph is self-driving, as the human response time becomes a limiting factor. And of course the in-dashboard wet bar. This will feed back into the consumer market even if it's only closed markets at first: why can't I go 250kph on the highways considering the self-driving airport shuttle does?
Well, of course. This is probably the first time the car has come across a truck carrying multiple street lights, without cover, stacked on the back of the truck, well above eye level. It's a rather unusual edge case.
It's only a matter of time before the software in these cars can handle the vast majority of edge cases as well as or better than human beings. Human vision isn't exactly reliable.[a]
In the meantime, someone should make a playable game in which trucks throw street lights at cars. Maybe someone at Tesla is willing to make this game in good jest?
As the second video in the thread demonstrates, the truck is literally hauling traffic lights. The AI recognition is correct, the only thing worth complaining about is that they're displayed as static objects for the user after recognition, just to be re-recognized a few seconds later in a different place. Note that the car is correctly not detecting they are lit, so not inferring direction (though AP isn't engaged, so I guess we'll never know what it would have done).
No doubt you could play the same game by putting a traffic cone on your bike. The car wants to see important traffic objects, it's literally what it's trained for.
The car handled this just fine, though. (Though obviously AP wasn't engaged so we don't know exactly what it would have done). The bug report here is that the correctly-detected signs, in the correctly-detected state (i.e. off) were presented to the user in a confusing way in the UI visualization.
UI bugs aren't blockers for mythical SAE Level 5 Autonomy. You just... fix them.
I am not sure why you think it was UI blocker and not actual problem affecting the driving?
In many places, the road rules say that if you see a broken traffic light, you should treat it as a stop sign. So a theoretical L5 car should have stopped there, in the middle of the road -- and again, and again. Seems pretty bad to me.
And yes, I agree that you "just.. fix them" -- but of the difficulties of the real road driving is that the number of unique situations like that is very large, and many of them would not happen during test drives / development.
> I am not sure why you think it was UI blocker and not actual problem affecting the driving?
I think it's not an actual problem affecting the driving because the driving was unaffected. Autopilot wasn't engaged, we don't have evidence.
I'm just pointing out that the autopilot doesn't drive by reference to the dash UI (which is showing a pretty obvious visualization -- the UI understands these to be static objects so once it gets one it "animates" it as if the car was passing it), so bugs there aren't very informative as to its behavior.
Very interesting. There must be some kind of assumption in models that traffic lights are stationary objects which results in them falling off the back of the truck.
I mean it makes sense. If you've never seen traffic lights be delivered to a new intersection you'd probably blithely assume you'll never encounter a stack of them that are unlit and moving.
Somewhere at Tesla there's a junior engineer who's telling a senior engineer "I told you so!"
This is a great example of how Tesla's strategy to build out using real-life training data can pay off.
Tesla now has data about what it looks like to drive behind a truck transporting traffic lights! No team in the world would solve this problem, in advance, in simulation.
Not saying their strategy is going to work, not saying it's an unbeatable advantage, but: just look at it! This is a compelling demonstration.
Imagine all the hilarity that can ensue when they release this beta stuff in production... oh wait.
But seriously, these are just slightly funny obscure weird cases, imagine when the hackers start coming up with malicious cases to mess with these cars.
203 comments
[ 3.0 ms ] story [ 213 ms ] threadWe discount how much the concept of “understanding” is required in visual perception. We don’t just see shapes we have a complete understanding of what we’re seeing.
I might see a big rectangle flopping in the lane in front of me. I can immediately ascertain whether it’s a piece of tumbling plywood, or foam based on movement characteristics, color, apparently size, etc. I can then use that understanding to decide what evasive actions are required.
A Tesla it seems has absolutely zero of this capability.
It was a tow truck towing a car backwards. It was just enough in my half-asleep state to scare the shit out of me.
Humans have millions of years of evolution behind our visual processing systems. We have developed hacks that prevent our brain from getting tricked by unusual situations like traffic lights on trucks or backwards cars being towed. We only developed the first computers a hundred years ago, and only in the last 40 years have a small subset of people started learning about visual processing systems.
It's easy to look at this video and scoff because of how trivial it seems. But it's instead a marvel of our minds that we can pick up on context clues so quickly and accurately that such oddities basically never puzzle us. Given the pace of our innovation, it wouldn't surprise me if our computer systems match ours within a few human generations at latest.
And you're sure that it was a bump, and not the driver pulling up rather close behind it and then giving a sharp tap of the brakes to wake you up? ;)
There are some funny(?) youtube videos of people in the passenger seat waking up to this with tractor trailers being towed in reverse.
This is not a FLOPS problem. Moore's law can't save you here.
Except we're trusting ML to perform surgery, choose conviction sentencing, evaluate job CVs, determine acceptable marriage partners (why not?), determine who can have kids (why not?), determine who gets into college (why not?), determine who gets a loan (why not?), determine who gets to work on ML (why not?). And drive cars.
> The data subject shall have the right not to be subject to a decision based solely on automated processing, including profiling, which produces legal effects concerning him or her or similarly significantly affects him or her.
https://eur-lex.europa.eu/legal-content/EN/TXT/HTML/?uri=CEL...
Question on your snippet though. It says "subject to a decision based solely on automated processing ..."
Since I do live in the consumer's paradise I'm naturally suspicious. I wonder if people will sometimes be subject to decisions based solely on automated processing, and if they don't bother to ask for a human in the loop after the automated decision, then that's that.
And how would you even know, one way or the other?
Another example is roads, where the design of the road allows one to turn into an opposing vehicle. The road itself is not deciding to turn your car into a semi trailer because it looks like sky, but an ML driven car has, multiple times.
I don't even own a Tesla, and I resent having to participate in its beta program. ML in a car is quite different from a brake recall, or even a stuck accelerator software recall.
The odd thing is that even a randomly stupid AI for self driving is statistically safer than most drivers. Clearly there is a ton of room for improvement in both AI and Humans.
I'm sorry, did you mean customers?
This hasn’t been shown yet at all. Statistics showing autopilot have less crashes per mile always ignore that Autopilot is doing the type of driving that has the least accidents per mile (motorway driving).
A strong claim, lacking actual evidence for it. All we have to go on are some Elon tweets (rather the definition of a biased source) and the actual crash rate. Without a lot more data (which Tesla steadfastly refuses to release) about environments, corrections, etc, it's quite impossible to make that sort of statement with any confidence.
The Tesla hardware is a weird combination of capable and insanely dumb, and it's far from obvious which it will be in any given situation until it's gone through it.
If an honest statistical analysis of the data indicated that Tesla's automation was better than human drivers (or better than other driver assist systems), I would fully expect them to have released the values. Since they haven't, and only hint at it and make statements that sound statistical but really aren't, I assume they've done the numbers internally and know it's not nearly as good as they like to imply.
If I drove in a city like their "self driving" beta was a few months back, I would be hauled from the car on suspicions of driving while hammered.
the fact that people still trot this out every Tesla thread is super annoying. Sorry to break it to you but this is a 100% false claim
I think self driving is a typical 80/20 problem. We won't have "full" self driving because the costs are exponential for each step closer to it. But driving on 80% of roads on 80% of days, with supervision? That could happen.
But that said: we won't accept AI that just makes traffic safer "on average". I'm fine with human shortcomings causing accidents. People will not accept car manufacturers cutting corners and causing accidents, even if statistically it's safer. So the very high bar for self driving isn't just "as safe as humans".
https://www.forbes.com/sites/bradtempleton/2020/10/28/new-te...
LKA is just that, an assist, because if you let it drive it’ll bounce back and forth between the lane markers, not keep you seemingly centered.
Hyundais have exceptionally-good lane-following.
Edit: You can also accelerate without disabling it.
Or maybe I'm just salty that my cruise control has never worked.
Eh? Yes they do. Bizarre lie.
It was lost due to Tesla's decision to remove MobilEye as a supplier.
Mobileye generally focuses on highly optimized HW/SW that does individual things very well, in a manner similar to how factory automation works (e.g., they basically built a "lane keeping + auto-braking + sign reading" appliance).
Tesla decided that 1) It was bad to outsource automation 2) Starting from scratch and 'learning' how to drive using ML was better than iteratively teaching a car how to do discrete tasks very very well (this is why Autopilot regressed a bunch in 2016).
In general, it's another symptom of Elon's 'I have an extremely specific idea, let's figure out' mentality that sometimes works and sometimes results in useless tunnels under Las Vegas.
Personally I see bumper to bumper traffic maybe 3 or 4 times a year (when driving home from a vacation or being forced to a doctor’s appointment during rush hour). And I honestly don’t get why anyone would subjugate them self to this kind of traffic as part of their daily commute.
I used to live in LA where Santa Monica Blvd was always backed up. I doubt LA traffic has gotten better either.
I'm surprised by your insolence with regard to how shitty traffic circumstances are in big cities. Simply changing one's commuting times doesn't failsafe the issue.
Also, transit lines tend to connect well to mid-Manhattan but poorly between other locations. So if you live in Queens and work in Brooklyn, good luck getting to work reliably by public transportation. (Before you object to that arrangement, consider: If you own a house and have kids in school, you're not necessarily going to uproot your family and move just because your new job is further from home.)
Thus, many ordinary New Yorkers rely on cars to commute to work.
If you need to drive to the supermarket you should have the option of choosing a time and route with minimal risk of traffic jams. I find it hard to belief that many people are frequently hitting bumper to bumper traffics on their way to or from the supermarket. Occupationally yes, but frequently no.
1: https://www.google.com/maps/dir/84-25+168th+Pl,+Jamaica,+NY+...
The population of the tristate is nearly 20 million. Even a tenth of that is a lot of cars on the road.
In fact I’ve often heard people from that area complain more often about lack of parking near their commuter rail station. Which indicates that people do rather tolerate circling the parking lot in their park-and-ride rather then risking stop-and-go traffic jams.
In the NYC area, bumper to bumper traffic is common. It can be caused by an accident or construction that blocks one or more lanes, cars merging on to an already congested highway, etc. These conditions frequently happen even outside of rush hour.
You may have trouble understanding it, but empirically a huge number of people see this extremely regularly, if not daily.
It's not even necessarily a feature of horrifically long commutes. For one example, lots of places that have basically ok traffic have bottlenecks at bridges, you may be stop and go for a little while every day getting across that.
I'll still take that over the ones that beep proudly to tell you that they are about to fail to maintain their lane (hi ProPilot).
Not anymore, since they removed the radar. Rain seems to really interfere with the vision based system, and apparently auto high beams are required at night, and they flash constantly.
I think it was on the way out by maybe 2022
Short term: Global chip shortage made supply tight, and it was either remove it and rush out with the no radar code branch, or stop shipping cars.
Long term I think it's a fine decision, but short term it's kind of a big thing to rush.
That said, i want a cool fast BEV that makes highway driving easier. Right now, Tesla is the only option. A few others are close. If someone else beats Tesla to FSD with another system, I will be happy to go that way on my next car. Working FSD is worth a lot to me.
(Admittedly the reasons why the Subaru impressed me was more for its programming rather than absolute precision. I liked how smoothly it transitioned between following and open road ahead. Much less jerky.)
The high beams are a different story. I only like to use auto-high-beam on rural roads as they tend to be problematic on busy highways in Tesla and my non-Tesla. But requiring them even on highways would be a big problem that I don't foresee them solving soon. I'm worried that this is going to take a long time to fix.
IMO, they actually work pretty well on rural roads.
With one huge exception: self parallel-parking.
I don't understand why this innovation doesn't get more love. Tesla's not the only manufacturer to offer this, of course, but this particular innovation has increased m enjoyment of city driving more than ... well, more than anything I can think of.
Even if they never get anything else to work, it would have been worth it just for self parallel-parking. lol
From an engineering perspective, of course there are serious problems that need attention, but sometimes it also good to celebrate the wins.
I agree. It's really nice to have the radar say "Yes, that space is big enough." and then put the car into it.
It's been in the Prius since 2003.
It seems like they should have a million hard test cases that must pass in simulation before releasing a new model. The simulations should be harder and more extreme than anything encountered in real life.
I think the real problem is obvious. They're trying to rush the work because Elon said so.
They don't work there anymore.
From their experience, I know one thing: I will never work for Elon Musk. He may be a great visionary and salesman, but he's a horrible manager.
It was stunning to see the complete disconnect between Musk's grand declarations and what the organization was actually setup to deliver.
Frankly, it just gives me more respect for Tim Cook, who as COO at Apple made his company able to turnaround and deliver HW in record time.
Edit: in retrospect I wonder if Musk's grand public declarations were actually a way to control and pressure his own organization. Remember, Musk didn't actually found Tesla, he rescued it from bankruptcy after the Roadster didn't return as much as needed, so he inherited an existing structure.
I know it's cool to hate Elon.
If you want evidence from a reasonably neutral observer, take Sandy Munro (himself an engineer who has worked on everything from cars to aeroplanes). He recently interviewed Elon, ostensibly about Tesla but the interview was in a meeting room at SpaceX. After the interview he was invited to a two hour design review meeting and was "blown away" at Elon's depth of involvement.
https://youtu.be/S1nc_chrNQk?t=370 (6:10 to 8:45)
Sources: https://www.reddit.com/r/SpaceXLounge/comments/k1e0ta/eviden...
https://www.youtube.com/watch?v=S1nc_chrNQk&t=377s
[1] https://youtu.be/bvim4rsNHkQ
Yes. The human brain and visual systems aren't nearly so trivial to replicate as a lot of people in the tech industry seem to think.
Tesla is just one of many case studies in the paired tech industry arrogance seen so frequently:
- "A human is just a couple really crappy cameras and a neural network, we know how to do better cameras and neural networks, how hard can it be?"
- "We can do anything we dream with 99.995% reliability in the synthetic, computer-based world of the internet because we know code. Therefore, we can do anything we want in the physical reality with code!"
Both are far from evident in practice, but the belief in them continues, despite it being increasingly obvious to everyone else that neither one is true.
Human vision and world processing is quite impressive - and, as pointed out elsewhere in this thread, a two or three year old would have no trouble working out that the obstacles were some things on a truck. I've got a nearly three year old, and I guarantee he wouldn't confuse those for stoplights in the slightest. I also wouldn't let him out on the road, though he does well enough with a little Power Wheels type toy. But there is far more going on in the visual processing system than we even understand yet, much less have the slightest clue how to replicate.
And while code may be fine on the internet (where you can retry failed API calls and things mostly make sense), the quote about how fiction is constrained by what's believable and reality sees no such restrictions is very true. Out on the roads, all sorts of absolutely insane things can and do happen on a regular basis - and you can't predict or plan for all of them. But the car has to handle them or it crashes.
As a random example, a year or two ago, I was behind a car that had poorly strapped a chunk of plywood to their roofrack with a good chunk hanging forward, and the front end of it was starting to oscillate awfully hard. I had a good clue that it was going to come apart sometime in the very near future, so backed off from a normal following distance to quite a way back. Sure enough, half a mile later, it failed, went flying through the air, slammed into the road a good distance behind the car, and tumbled a bit. Had I been using a normal in town following distance, it would have either hit me or tumbled into me, but using a human visual system, it was obvious that my existing following distance stood a good chance of being a bad idea.
Stuff like this happens on roads constantly. Meanwhile, state of the art self driving can't tell the difference between stoplights and some poles on a truck. You'll excuse me if I don't think the problem is anywhere remotely close to solved for a general case Level 4 purpose.
Aritical neural networks are pretty good at object recognition, among hundreds of other things, and even better than humans at some of them. They are, however, generally pretty bad at abstract reasoning, critical thinking, 'understanding' concepts in-depth, and so on, and I think that's a more constructive way to phrase the problem we see in this video.
When a problem is fully redicible to a simple vision problem, modern neural networks are a great choice, but being a good driver involves much more than just the visual cortex.
https://en.wikipedia.org/wiki/God_of_the_gaps
One way to think of this is that that the footage is implicitly labelled: we have the benefit of hindsight: we know what the state/location of the vehicle was going into the future. That benefit of hindsight also can serve as implicit labels by knowledge that the vehicle did not crash or collide with something immediately after the footage.
I don't understand why this is hard for Tesla engineers -- I was doing this kind of thing in grad school a decade+ ago and it worked fine. I've seen it in other demos where object classifications rapidly cycle between person, bike, car, etc. Are they not filtering anything? Is this a symptom of "AI-ing all the things"? Because we did it with bog standard computer vision techniques back then and never got behaviors like this.
The car is properly recognizing traffic lights pretty darned well, considering the circumstance. It looks like it has a built in understanding that traffic lights are "always" stationary - hence, assigning them static locations on the 3D map - but it keeps having to update the model because the lights are actually moving.
This seems like a very non-obvious edge case that I wouldn't expect an ML team to even consider as a possibility. Now they need to program into the ML model an understanding that traffic lights are typically stationary. Which seems even more difficult to me, from a technical perspective - you don't want false negatives...
The car isn't braking or making any strange maneuvers from what I can tell. I'm actually impressed that it's handling it this well.
When you don't know what to do - do nothing. What if it was a traffic light on roller skates? Or a kid, dressed as a traffic light, on roller skates?
Collision detection systems (radar) are accurately not detecting an impending collision because the lights are not actually on a collision course with the vehicle.
Object recognition systems (computer vision) are working very well, because they recognize the lights and are updating the 3D map accordingly, but the traffic light 3D model is not designed to be a moving object - unlike vehicles, which frequently move. Which is why we see the car "passing through" them.
What we are likely seeing is simply a weird edge case in the output for the user-interface. I'd imagine if an object was actually flying at the car and the car could see it, it would brake accordingly.
Also, the map is two-dimensional. The car frequently drives underneath traffic lights that I'm sure also appear "on top of" the car in normal cases.
Object recognition and collision detection, from what I understand, are two very different systems.
But seriously, I’m inclined to be charitable here and assume that this is merely a quirk of the UI display. There’s no evidence that the autopilot did anything unsafe (apparently it wasn’t even engaged?), and until I see evidence of that I’m willing to withhold judgment. (I have seen evidence of other situations where Tesla autopilot did unsafe things and I’m in no way apologetic about those situations.)
In this case, obviously it's not a safety concern because the object is being hauled on the back of a truck.
Move fast and break things [like tests]
It would generally be a clue that there's an intersection busy enough to require signals that now lacks signals or signage which would warrant extra caution. I'd expect the car to at least slow down significantly, if not come to a complete stop before proceeding.
(the grey steering wheel icon means autopilot is not engaged, it would be blue if it was on)
But it clearly means another edge case like detecting deer (wonder if they can handle our local Kudu) that they need to deal with.
They might as well develop AGI at this rate.
I also remember being at an intersection where I was turning left and was waiting behind another car. The display repeatedly showed cars the cars crossing in front of us crashing into the car in front of me. Not sure why.
It works much better now, but it was always kind of amusing. It only did it when we were all stopped. Moving cars (whether it was them moving or me) were rendered just fine. My guess is that it was extrapolating movement based on more than one "frame" of sensor data, but at a stop there's no changes beyond noise, and the noise was being extrapolated into extreme movements.
I have seen it show cars "crashing" into me, but it's only semis when in the neighboring lane.
(the steering wheel icon at the top of the screen is grey, not blue)
Liability, insurance, legal minefields and plain old marketing would never allow cars ,that perform as well as the best performing cars do today, to be on the roads in "every day" conditions.
My conjecture is we end up building AV only roads. initially one lane of a highway, then ring roads round cities and major warehousing hubs, then across urban areas. Walled off in some way they simply become railways with benefits.
At that point every business model ever written with Self Driving in the title goes in the bin.
I am not saying the tech is useless - frankly it's fucking awesome that this is happening in my lifetime. But fucking awesome tech and workable business model aren't always the same thing.
Not sure where I am going with the rant but I sure hope we get more out of the billions spent here than Teflon.
https://xkcd.com/678/
That's not happening any more. All we have left is laughing at stuff like this, where the visualization (not even the autopilot!) gets confused by seeing real (!) traffic lights on a truck, so it paints them in space, but then has to re-recognize them because they are moving.
At some point, the luddites will just run out of ammunition. It's sort of happening already.
But the AI drives slowly and gets confused easily. Regular drivers routinely have to go around self-driving cars. Not to say they won't improve, but it seems like current AI is assistive to the point where it might be harmful when drivers rely on it in speeds and situations where they shouldn't. I'm sure it will keep improving, but I feel like this is one of those situations where the amount of data and training required, and the amount of iteration on the software required to handle edge cases is not impossible but is exceptionally difficult.
Probably better to say that the AI drives slower than you would and gets confused differently. Real drivers die every day to terrible mistakes that look obvious in hindsight. Most of them probably thought AI was terrible too.
The bar here is much lower than you think it is, and frankly existing automation has already crossed it.
If you want to fix this one exception (false positive) you will introduce unwanted false negatives. That's how precision and recall work, there's a trade-off. So I am not sure it's useful to fix it, how many times does this happen? Will fixing it introduce more frequent bugs?
Just my hypothesis: but I think the autopilot really did see them as traffic lights, and just got lucky that they weren't powered and ignored them as out of order. Were there a cross street, I suspect the car would have stopped and treated it as an uncontrolled intersection...
But the software model is that traffic signs are static, and these were moving, so the visualization had no way to present that. It just left them there until the ML told them they weren't there anymore.
What would the AP have done? We don't know. But I don't know why we should simply assume it would have done the wrong thing. Driving is filled with false-positive indicators that it already knows to ignore successfully.
Will you bet with your life on it?
You might say people don't want those delays but we tolerate delays in normal commuting. An unattended bag on a tube station stopping trains, a car accident blocking traffic, traffic jams blocking traffic, mechanical breakdowns, etc. As long as they're infrequent enough, it should be OK for riders.
Time and again I've watched "ain't happening" technology become the preferred norm practically overnight. Eagerly awaiting my FSD CT, and making long trips without having to micro-manage every foot across thousands of miles.
That would be awesome still and make people want to have that technology...how does that murder Tesla's business model? Waymo would easily pivot to licensing their tech to car manufacturers.
The autonomous car business model is very much alive and well, it has simply shifted in two key areas that cause general consumers to misinterpret what is happening.
1) Use cases (ODD). Many people think "if I can't buy an autonomous car, it's not a real thing". But most manufacturers and AV developers other than Tesla aren't even trying to tackle that use case. Instead, the focus is on autonomous vehicles operating very profitably in specific locations and operating models that work. Heavy trucking on "easy" routes, ridesharing in well-mapped areas of the Southwest US, shuttles on popular metro and airport routes, etc. These use cases represent very large revenue opportunities and the tech development is progressing well.
2) Timing. Too many pundits and CEOs looking to generate PR buzz made ridiculous claims over the last few years about when autonomous vehicles would be ready. This set false expectations among the general public which has now soured perceptions since autonomous tech is not easy. Fully driverless is already ready in some very specific scenarios (ex. Waymo in Chandler) and is on track to roll out to more ODDs over the next 5-7 years. I can pretty much guarantee that you will see a significant number of fully driverless vehicles in suitable areas by 2030. No, they won't drive in upstate New York in a blizzard, but no one actually cares about that from a business perspective.
The other issues you raise - liability, insurance, legal roadblocks, etc. are mostly non-issues that already have solutions. The only one that's a continuing problem is the fragmented nature of legislation across different states. But there is very heavy lobbying going on right now to rectify this by the time it's actually needed.
Edit: I would be interested in real experts correcting my poor understanding of important subjects - can we talk ?
Close, but I think it's far more probable that we just whitelist roads / intersections / routes known to work well and be reliable--whether with a general-purpose or location-specific algorithm. There's still a ton of value shuttling people to and from the airport and local hotels.
Then there'll be incentive for local restaurants to also be included in the available destinations, so there'll be pressure to both improve local roads / intersections to make them more easily navigable by self-driving cars, and on self-driving cars to get better at navigating them. With time, the locations / routes that self-driving cars can reliably reach will expand until it covers the vast majority of desired destinations.
We actually are following the same pattern with the horses -> cars transition. There are still plenty of places that cars can't reach and horses can. Over decades, though, we just kind of paved roads everywhere anyone really wanted to go. Nowadays it's more or less a given that if you want people to be able to reach somewhere, it'll need to have a road.
We'll see the same thing with self-driving cars. There'll still be roads that self-driving cars avoid and humans drive on, but they'll become less and less relevant as the value of self-driving cars becomes more apparent.
sadly, in India too, the major electric 2wheeler Ather, is kinda going the same high-tech way as Tesla. and very expensive !
A big part of Tesla's success was making electric vehicles that were covetable, rather than being obvious "we made this for compliance reasons or to get a government contract" products. This means targeting a luxury market segment. The way they chose to wow luxury consumers was with high performance and a lot of technological gimmickry.
The individual still thinks in terms of "I'm an above-average driver, I'm better than a silly bleep-bloop box." They'll resist self-driving en masse if it's not 1000% perfect.
Insurers, however, will see self-drivers as predictable, never drunk, always following the speed limit, and in aggregate preferrable to humans. They will be able to see "3% better" as meaningful even if it's not perfect, and price accordingly. Eventually, you'll pay a rapidly increasing penalty rate for having a steering wheel. This is where the avalanche comes from.
I also figure self-driving could unlock new features-- in particular I'm imagining the only way we get road speeds much over 120kph is self-driving, as the human response time becomes a limiting factor. And of course the in-dashboard wet bar. This will feed back into the consumer market even if it's only closed markets at first: why can't I go 250kph on the highways considering the self-driving airport shuttle does?
It's only a matter of time before the software in these cars can handle the vast majority of edge cases as well as or better than human beings. Human vision isn't exactly reliable.[a]
In the meantime, someone should make a playable game in which trucks throw street lights at cars. Maybe someone at Tesla is willing to make this game in good jest?
[a] See, for example http://www.ritsumei.ac.jp/~akitaoka/index-e.html
As the second video in the thread demonstrates, the truck is literally hauling traffic lights. The AI recognition is correct, the only thing worth complaining about is that they're displayed as static objects for the user after recognition, just to be re-recognized a few seconds later in a different place. Note that the car is correctly not detecting they are lit, so not inferring direction (though AP isn't engaged, so I guess we'll never know what it would have done).
No doubt you could play the same game by putting a traffic cone on your bike. The car wants to see important traffic objects, it's literally what it's trained for.
UI bugs aren't blockers for mythical SAE Level 5 Autonomy. You just... fix them.
In many places, the road rules say that if you see a broken traffic light, you should treat it as a stop sign. So a theoretical L5 car should have stopped there, in the middle of the road -- and again, and again. Seems pretty bad to me.
And yes, I agree that you "just.. fix them" -- but of the difficulties of the real road driving is that the number of unique situations like that is very large, and many of them would not happen during test drives / development.
I think it's not an actual problem affecting the driving because the driving was unaffected. Autopilot wasn't engaged, we don't have evidence.
I'm just pointing out that the autopilot doesn't drive by reference to the dash UI (which is showing a pretty obvious visualization -- the UI understands these to be static objects so once it gets one it "animates" it as if the car was passing it), so bugs there aren't very informative as to its behavior.
Somewhere at Tesla there's a junior engineer who's telling a senior engineer "I told you so!"
Tesla now has data about what it looks like to drive behind a truck transporting traffic lights! No team in the world would solve this problem, in advance, in simulation.
Not saying their strategy is going to work, not saying it's an unbeatable advantage, but: just look at it! This is a compelling demonstration.
But seriously, these are just slightly funny obscure weird cases, imagine when the hackers start coming up with malicious cases to mess with these cars.