It looks like there are two updates - the infotainment AND the other one .. firmware of some thingies. And the infotainment is a PREREQUISITE to the other one.
That is what I surmised from listening to the "don't do this until we fix it please" notice from Stellantis from this weekend.
I've had a Jeep for a few months, and it bothers me so much that the entire community is about modifying the vehicle as much as possible, but they still come with this locked down OS.
If any car could be the champion of OpenSource, it is a Jeep Wrangler, but they're using an OS made by SiriusXM for some reason.
Isn't Jeep the company that introduced infotainment pop-up ads that play at the stop light? I doubt they'd want you loading an open source OS replacement to avoid that.
I can't for the life of me understand why infotainment systems knock so many engineers for a loop. Is there a particular reason (industry/domain-specific) beyond just low-quality software development?
My Mazda 3 (2018) just had a class action lawsuit for its infotainment system which, completely at random after years of normal operation, starts clicking on menu items and scrolling about the settings (only to stop and not do it again for a couple of months). It can get so bad you just have to disconnect any devices and drive in silence/with the AM/FM radio.
I recently did some testing of Bluetooth connecting an Android phone to a reasonably new Mercedes. Being a "luxury" brand, you'd think the Mercedes would have good infotainment software, but I found at least 5 bugs in the GUI in about 1 hour of testing, and I wasn't particularly looking for bugs, just testing what I thought would be the happy paths for Bluetooth connection. The Android phone, on the other hand, did its job flawlessly.
I know software and embedded systems well enough to know all of the issues I found were preventable, if anyone cared.
The car seems well built in many other respects. It doesn't look like the problem is engineering ability.
(See also: Set-top box GUIs that are painfully slow to render menus, scroll, search etc. on hardware that I know can render 10-100x faster when programmed to.)
it's the consequence of not vertically integrating, having 20000 different ECUs each for a specific purpose and trying to nickel-and-dime suppliers on all of them, and the lack of urgency that traditional manufacturers and tier-1 suppliers have for adopting modern software development practices
Hardware companies outsource these things to the lowest bidder in "best cost countries". At home you have a couple "engineers" who track requirements, but are software illiterate. In the "best cost country" you have dozens/hundreds of people who have absolutely no investment in the product and are not particularly skilled, and who get paid so that that engineer can make a checkmark after some requirement in the shortest amount of time. Nobody in that entire pipeline cares that the system feels awful to use, that it glitches sometimes or that the CPU is totally over utilized.
Tesla was revolutionary because they actually had inhouse software developers, who could build software.
I recently worked at a big home lighting company, working on the OS of the router device that communicates with the light bulbs themselves and the internet/user.
Our OTAU architecture uses A/B system updates [1]. Core idea is that both the kernel and the rootfs (read-only) partitions had 2 different bootslots in storage, and the OTAU would only write to the bootslot that is unused. Hence, if something went wrong, the system would automatically fallback to the previous version by just switching the bootslot used. Over the numerous years that that architecture was used, I couldn't find a single post-mortem that resulted in devices being bricked. Something to note is that the rootfs partition was overlaid with a writable partition for persisting state data etc.
Now that was a $two-figure USD device, not a $5/6-figure USD electric SUV. Is this a cost-cutting measure? At those price levels, doubling your NAND size is not even half of a percent of the total cost of the vehicle.
Unless there was a serious issue that the used bootslot corrupted the unused bootslot, then I don't see how this could have happened.
It's saddening that car manufacturers are so unserious about the code they're deploying.
This is generally how other devices work as well - for example all Android devices and Android-derivatives (which many of these cars are!) use a similar A/B partition to prevent bricking.
It definitely reduces the risk of updates, but it absolutely doesn't eliminate it.
The A/B updater itself is a surface area - especially if the logic is complex and there are other child devices that are updated at the same time (likely for cars). In that case you're not just coordinating between two independent partitions, you're coordinating between 2 * N partitions, half of which have dependencies on each other.
Also, the key bit of the mechanism is that upon successful boot the new partition is flagged as "good", which causes flags to be set to assert that the update was successful and the backup partition is no longer needed. That logic can (and does) fail - if your failure point occurs after this checkpoint you're hosed still because you're past the point of no return.
Making that worse is that in most systems you want the "it's all good" checkpoint to occur early - you don't want to, for example, wait multiple minutes for all user services to come up. But that also means that if a critical failure happens in said services, you're past the checkpoint.
> the system would automatically fallback to the previous version by just switching the bootslot used.
That's the hard part though.
It's shockingly common in my experience to have an A/B boot setup, but no actual logic in the userspace application to switch back to the other partition if something goes wrong. It's just a defense against somebody pulling the plug during the OTA, it doesn't protect against software bugs at all.
I never heard of this and follow Tesla groups/communities/forums/etc. for over 10 years. At most you'll hear one or another person complaining about having initiated the update process and suddenly getting annoyed because they find out they need to go somewhere (it might take an hour).
leadership problem, as everywhere. old grampas that used to manually draw gears on paper now have to "strategically align" a huge corporation that has to deal with new shiny and complicated things like software and they all have zero fucking clue. at least with cars you can always try to safely stop, with planes - not so possible. this will also soon creep up there.
I rented a Jeep Wagoneer recently and found it to have such comically glitchy electronics that this comes as no surprise. The second day we had it, the liftgate stopped latching entirely, it beeped and popped some error messages on the dashboard and simply wouldn't latch shut at all, no matter what we did. Searching the internet produced lots of people with the same problem, reporting that it required a software update to fix. There was no manual override to the electronic latching mechanism.
Luckily we were near a location of the rental car company—rather than deep in the middle of nowhere where we were headed—and exchanged it for another of the same model, which was all they had available. The next 1000-something miles we drove were filled with endless weird glitches:
- When a passenger plugged in their Steam Deck in the back, the entire infotainment system cut out and went black, including the instrument panel, and then started glitching in and out until they unplugged it.
- When parking, the driver's seat would retract slightly to make it easier to get out, but it never moved forward again, so the seat would get further back at each stop until it was manually repositioned.
- The entire drive the system flashed an un-dismissable error about a rear seat latch, which seemed completely functional.
- The TPMS light went on and off periodically as it seemingly lost and then regained signal from one wheel or another.
- The system flashed errors related to the automated cruise control being unavailable/broken at random times.
- The electronic parking brake kept applying itself while briefly paused in parking lots.
- There was something inscrutably wrong with the climate control that we never really figured out where sometimes it'd just get hot inside the car despite no change to the AC settings.
When we got back I found tons of people online talking about similar (often worse) issues. Incredibly terrible for any new vehicle, never mind one that costs $80k.
> - When parking, the driver's seat would retract slightly to make it easier to get out, but it never moved forward again, so the seat would get further back at each stop until it was manually repositioned.
This is AWESOME.
October is SPOOKY month for Stellantis software, apparently.
Overall it sounds like changes were applied, internally, and not reverted - as if they changed something in the Transaction handling for multi-step car systems updates.
You mention something about it continuously getting hotter ..
> it'd just get hot inside the car despite no change to the AC settings
I felt a little bad but laughed out loud reading some of the other stories of peoples' problems with the Wagoneer—one person reported that the first night they brought it home brand new from the dealer, one of the headlights wouldn't turn off no matter what they did and just stayed on the whole night shining against the house. Spooky!
My friends and I rented a Wagoneer and found that the electronic feature to flip the rear seats down worked while there were people in those seats, while the vehicle was in motion!
So of course every hour when the boys weren't paying attention POP the driver would unlatch their seats and headrests lmao
Horrible safety guardrails but a good time was had by all.
My 4xe died in my driveway on Saturday after the update. Let me explain, from the perspective of a 4xe owner, how bad the response has been from Jeep/Stellantis:
- As of Monday 8am ET, zero legitimate communication from any Jeep-related accounts on any social media platform, or any other form of acknowledgement from the company (unless I've missed something?)
- I only found out about the issue after finally searching a few Jeep groups on Facebook (of all places) to see if anyone else was experiencing the weird failure mode I was after the update.
- The only remotely-official info was from a 'JeepCares' account (which is ran by Jeep) on some random off-roading forum? We were seriously all living off of screenshots from this forum, and the advice coming from the JeepCares accounts was contradictory: they claimed that the Uconnect update was separate from the telematics update, and that there was no way to stop the telematics update if the vehicle received it. Later they gave advice to defer the Uconnect update, making it sound like they were coupled.
- Due to the lack of info from Jeep, people were coming up with all kinds of "if you reboot Uconnect while the Jeep's in ACC mode, it clears the check engine light". This probably did clear the CEL but didn't fix the fault.
- There is no way to tell if you received the bad update.
- There is no way to tell if you received the 'fix' either.
- Dealerships have literally no idea what is going on.
- You're basically at risk of your Jeep going limp (power loss, unable to safely make it to the shoulder) and being stranded on the highway, even as I write this.
imho the occasional mixup is going to happen, and eventually it'll be big like this or like Crowdstrike, but pushing these out on Fridays means the critical staff isn't there to help. The people who could have communicated this stuff to customers and dealerships were in bed when people got into their jeeps at 6am on Saturday after an overnight update.
I believe crowdstrike's update was on a Friday night as well.
Unless its a serious security bug, it can wait for not only for better QA testing but also for next Tuesday. Read-on Fridays need to be an industry-wide thing.
> - You're basically at risk of your Jeep going limp (power loss, unable to safely make it to the shoulder) and being stranded on the highway, even as I write this.
This seems extraordinary.
I was going to ask: Are you really saying they kill the vehicle's power system, effictively the engine, while the vehicle is being driven on the highway?
But no need to ask, the article says yes, that's what is reported:
> Instead, the failure appears to occur while driving—a far more serious problem. For some, this happened close to home and at low speed, but others claim to have experienced a powertrain failure at highway speeds.
From a GPS software update... [1] "This is a telematics box module update" Telematics is primarily GPS and on-board diagnostics for location, speed, and fuel usage.
A GPS update kills your entire powertrain. Appears to also engage parking for some users, super dangerous. Catbones, "Almost died on the thruway today ... with an 18-wheeler behind me. ... Jeep died, locked its hand brake and jolted so hard my face almost ended up in the steering wheel at 70mph." [1]
It is completely insane that this is happening. I did DD on a company in the automotive space a couple of years ago and flagged that they did not check if the vehicle was stationary, motor disabled before updating. They were all surprised at how I thought that this could possibly ever lead to issues.
This sounds like the sort of thing that happens when a team has a deliverable that slips multiple times but everyone had vacations planned for a responsible amount of time after the deadline.
Under time pressure and confirmation bias they signed off on code that was giving off signs of being broken, pushed it, and now key staff are either on airplanes, out of coverage on their phones, or cannot work entirely from memory and don’t have their computers with them because vacation.
If a software update can affect the basic functionality of the car to such a degree then the whole car should be recertified after every update, change my mind.
Why do you let your vehicle update without supervision and knowing exactly what the update entails (what it does)?
If you knew upates were occurring why didn't you stop them by not allowing internet access and or disabling the web/net hardware?
I find it very odd, I never allow any hardware unfettered internet access let alone update its firmware. Experience has taught me that that's a recipe for trouble.
A side note: is it just me or not but I prefer my new car to still providing tactile buttons to control things? I get that smart cars like Tesla wants to push touch-screen on everything but the control mechanism just feels clunky for me. When I sit on a car with physical buttons, I know exactly where they are and what they do when I press & turn them.
A few bad updates borked my Linux install and cost me hours of potential work and left me with a deep fear of installing updates. No idea what I'd do if I had a car that needs updates
I own a Land Rover and their system software also ("Pivi") seems to have tons of little quirks and issues.
Sometimes the cameras won't work unless you restart Pivi. One time, the entire car wouldn't respond to locking (via the app, keyfob, in-car buttons) unless you disconnected the battery and waited ~30 mins. Many people complain that they can't even successfully upgrade between Pivi releases. It'll error out a lot of times and they have to restart the process.
(Would be interested to know the kind of tech they're using if anybody's familiar with it!)
Just wrote about my battles with software updates[1]. Without getting into what should or shouldn’t happen, I’ll express my sympathy for the Jeep team and the owners. Software updates are hard.
[Edit: a commenter notes that Jeep’s parent has recently mandated LLM usage in development. Let’s hope these two are unrelated and that we aren’t going to start seeing more catastrophic failures like this over the next few months as people work out the limitations of this sorcery.]
Software updates for cars should be rare, with the infotainment systems physically isolated from the rest of the car (aside from constant/switched power). And if it needs updates for whatever reason (eg a recall), they should be done at a dealership - not over the air.
Bricking a car with an over-the-air software update is stupid and unacceptable behavior. Stuff like this is why I'm actually kind of happy to be driving a 20 year old car still.
I looked at the Jeep awhile back but their safety ratings were absolutely terrible. I’ve been in car accidents aplenty so that was a deal breaker for me. Just about the only real electronic thing in my Lexus is an ECU which rarely goes and can be replaced relatively easily . Forget updating the maps and other nonsense. Just drive the damn thing and do mechanical maintenance and a little grease.
68 comments
[ 1.9 ms ] story [ 96.4 ms ] threadThe infotainment system should be completely isolated from the driving system.
That is what I surmised from listening to the "don't do this until we fix it please" notice from Stellantis from this weekend.
If any car could be the champion of OpenSource, it is a Jeep Wrangler, but they're using an OS made by SiriusXM for some reason.
My Mazda 3 (2018) just had a class action lawsuit for its infotainment system which, completely at random after years of normal operation, starts clicking on menu items and scrolling about the settings (only to stop and not do it again for a couple of months). It can get so bad you just have to disconnect any devices and drive in silence/with the AM/FM radio.
I know software and embedded systems well enough to know all of the issues I found were preventable, if anyone cared.
The car seems well built in many other respects. It doesn't look like the problem is engineering ability.
(See also: Set-top box GUIs that are painfully slow to render menus, scroll, search etc. on hardware that I know can render 10-100x faster when programmed to.)
Tesla was revolutionary because they actually had inhouse software developers, who could build software.
Our OTAU architecture uses A/B system updates [1]. Core idea is that both the kernel and the rootfs (read-only) partitions had 2 different bootslots in storage, and the OTAU would only write to the bootslot that is unused. Hence, if something went wrong, the system would automatically fallback to the previous version by just switching the bootslot used. Over the numerous years that that architecture was used, I couldn't find a single post-mortem that resulted in devices being bricked. Something to note is that the rootfs partition was overlaid with a writable partition for persisting state data etc.
Now that was a $two-figure USD device, not a $5/6-figure USD electric SUV. Is this a cost-cutting measure? At those price levels, doubling your NAND size is not even half of a percent of the total cost of the vehicle.
Unless there was a serious issue that the used bootslot corrupted the unused bootslot, then I don't see how this could have happened.
It's saddening that car manufacturers are so unserious about the code they're deploying.
[1] https://source.android.com/docs/core/ota/ab
It definitely reduces the risk of updates, but it absolutely doesn't eliminate it.
The A/B updater itself is a surface area - especially if the logic is complex and there are other child devices that are updated at the same time (likely for cars). In that case you're not just coordinating between two independent partitions, you're coordinating between 2 * N partitions, half of which have dependencies on each other.
Also, the key bit of the mechanism is that upon successful boot the new partition is flagged as "good", which causes flags to be set to assert that the update was successful and the backup partition is no longer needed. That logic can (and does) fail - if your failure point occurs after this checkpoint you're hosed still because you're past the point of no return.
Making that worse is that in most systems you want the "it's all good" checkpoint to occur early - you don't want to, for example, wait multiple minutes for all user services to come up. But that also means that if a critical failure happens in said services, you're past the checkpoint.
That's the hard part though.
It's shockingly common in my experience to have an A/B boot setup, but no actual logic in the userspace application to switch back to the other partition if something goes wrong. It's just a defense against somebody pulling the plug during the OTA, it doesn't protect against software bugs at all.
They implemented a dual redundant system similar like the dual BIOS for motherboard since 1999.
Luckily we were near a location of the rental car company—rather than deep in the middle of nowhere where we were headed—and exchanged it for another of the same model, which was all they had available. The next 1000-something miles we drove were filled with endless weird glitches:
- When a passenger plugged in their Steam Deck in the back, the entire infotainment system cut out and went black, including the instrument panel, and then started glitching in and out until they unplugged it.
- When parking, the driver's seat would retract slightly to make it easier to get out, but it never moved forward again, so the seat would get further back at each stop until it was manually repositioned.
- The entire drive the system flashed an un-dismissable error about a rear seat latch, which seemed completely functional.
- The TPMS light went on and off periodically as it seemingly lost and then regained signal from one wheel or another.
- The system flashed errors related to the automated cruise control being unavailable/broken at random times.
- The electronic parking brake kept applying itself while briefly paused in parking lots.
- There was something inscrutably wrong with the climate control that we never really figured out where sometimes it'd just get hot inside the car despite no change to the AC settings.
When we got back I found tons of people online talking about similar (often worse) issues. Incredibly terrible for any new vehicle, never mind one that costs $80k.
This is AWESOME.
October is SPOOKY month for Stellantis software, apparently.
Overall it sounds like changes were applied, internally, and not reverted - as if they changed something in the Transaction handling for multi-step car systems updates.
You mention something about it continuously getting hotter ..
> it'd just get hot inside the car despite no change to the AC settings
.. which is also f'in nuts.
So of course every hour when the boys weren't paying attention POP the driver would unlatch their seats and headrests lmao
Horrible safety guardrails but a good time was had by all.
- As of Monday 8am ET, zero legitimate communication from any Jeep-related accounts on any social media platform, or any other form of acknowledgement from the company (unless I've missed something?)
- I only found out about the issue after finally searching a few Jeep groups on Facebook (of all places) to see if anyone else was experiencing the weird failure mode I was after the update.
- The only remotely-official info was from a 'JeepCares' account (which is ran by Jeep) on some random off-roading forum? We were seriously all living off of screenshots from this forum, and the advice coming from the JeepCares accounts was contradictory: they claimed that the Uconnect update was separate from the telematics update, and that there was no way to stop the telematics update if the vehicle received it. Later they gave advice to defer the Uconnect update, making it sound like they were coupled.
- Due to the lack of info from Jeep, people were coming up with all kinds of "if you reboot Uconnect while the Jeep's in ACC mode, it clears the check engine light". This probably did clear the CEL but didn't fix the fault.
- There is no way to tell if you received the bad update.
- There is no way to tell if you received the 'fix' either.
- Dealerships have literally no idea what is going on.
- You're basically at risk of your Jeep going limp (power loss, unable to safely make it to the shoulder) and being stranded on the highway, even as I write this.
I believe crowdstrike's update was on a Friday night as well.
Unless its a serious security bug, it can wait for not only for better QA testing but also for next Tuesday. Read-on Fridays need to be an industry-wide thing.
This seems extraordinary.
I was going to ask: Are you really saying they kill the vehicle's power system, effictively the engine, while the vehicle is being driven on the highway?
But no need to ask, the article says yes, that's what is reported:
> Instead, the failure appears to occur while driving—a far more serious problem. For some, this happened close to home and at low speed, but others claim to have experienced a powertrain failure at highway speeds.
Wow.
A GPS update kills your entire powertrain. Appears to also engage parking for some users, super dangerous. Catbones, "Almost died on the thruway today ... with an 18-wheeler behind me. ... Jeep died, locked its hand brake and jolted so hard my face almost ended up in the steering wheel at 70mph." [1]
[1] Wrangler 4xe forum, JeepCares and Catbones accounts: https://www.4xeforums.com/threads/wrangler-4xe-ota-update-10...
Personal bet: Jeep accidentally enabled the remote kill switch for repossessing automobiles. [2] Possibly the "impaired driver" kill switch. [3]
[2] Stateline, Late Payment Kill Switch: https://stateline.org/2018/11/27/late-payment-a-kill-switch-...
[3] Trackhawk, Federal Kill Switch Law: https://trackhawkgps.com/blog/kill-switch-law
Under time pressure and confirmation bias they signed off on code that was giving off signs of being broken, pushed it, and now key staff are either on airplanes, out of coverage on their phones, or cannot work entirely from memory and don’t have their computers with them because vacation.
If you knew upates were occurring why didn't you stop them by not allowing internet access and or disabling the web/net hardware?
I find it very odd, I never allow any hardware unfettered internet access let alone update its firmware. Experience has taught me that that's a recipe for trouble.
https://www.stellantis.com/en/news/press-releases/2025/octob...
I own a Land Rover and their system software also ("Pivi") seems to have tons of little quirks and issues.
Sometimes the cameras won't work unless you restart Pivi. One time, the entire car wouldn't respond to locking (via the app, keyfob, in-car buttons) unless you disconnected the battery and waited ~30 mins. Many people complain that they can't even successfully upgrade between Pivi releases. It'll error out a lot of times and they have to restart the process.
(Would be interested to know the kind of tech they're using if anybody's familiar with it!)
[Edit: a commenter notes that Jeep’s parent has recently mandated LLM usage in development. Let’s hope these two are unrelated and that we aren’t going to start seeing more catastrophic failures like this over the next few months as people work out the limitations of this sorcery.]
[1] https://tritium.legal/blog/update
Bricking a car with an over-the-air software update is stupid and unacceptable behavior. Stuff like this is why I'm actually kind of happy to be driving a 20 year old car still.