Tell HN: 3G sunsetting is remotely killing every Subaru Outback battery
STARLINK intermittently tries to phone home by hitting 3G towers.
Now that 3G is shutting down, the digital communications module (DCM) gets stuck in an infinite loop of
1. Phone home, expending battery charge 2. Fail, because 3G doesn't work anymore 3. Go back to step 1
This effectively remotely drains the battery of every Subaru Outback built between 2015 and 2020.
Even if you drive your car every day, its battery will die and you won't be able to start it.
Other models are probably affected, too.
There was a class action lawsuit. But this is a pretty egregious engineering oversight, given they were still producing defective cars in 2020, hardly two years before 3G flipped off.
Will a brand ever produce a reliable, mechanical car? Why should 3G towers have anything to do with my car being able to start?
220 comments
[ 3.3 ms ] story [ 96.4 ms ] threadAlso, that free upgrade for current Starlink subscribers expired two years ago.
Anyway, I didn’t upgrade (have an aftermarket radio), and my battery works just fine. In fact, the car stayed parked for two months in the Texas summer, and started just fine.
How do you figure this? Seems unlikely to me that a 15 min drive would not bring a car batter drained my occasional failed phone connections back to full
The thing drains your battery precipitously.
There must be more to this story. I've parked that car in an underground parking garage with absolutely no signal on any carrier for at least 2 weeks straight (likely more) with no issue. I've parked it for days at trailheads and wild camping spots around the High Rockies where there's also no signal.
I 100% believe that the Starlink module is poorly programmed, reliant on false assumptions of 3G always existing, and capable of draining the battery. But I would really love to know why that's never happened to me.
That would be different to having no signal whatsoever, where modem wakes up, listens for a tower, concludes there’s nothing there and goes back to sleep again.
That would explain why this issue is only cropping up now that 3G is being turned off, and how this scenario was missed by engineers. They assumed there was either a tower they could communicate with, or nothing. They didn’t account for a perfectly good tower being available, advertising itself as a recognisable network, but refusing to handshake.
Perhaps they are not impacted for some reason.
I hadn't gone camping for ~4-5 weeks, but still drove my car every day. Suddenly had the battery issue, pulled the plug, and the issue is gone.
I don't really know how to debug this. Something recently changed, but I don't know what. You'd think my battery would've died during all the time that I was out of service.
Ask any tow truck driver.
Edit: Cool, apparently you've completely changed your comment. No worries, it happens. The original comment read something like "I don't believe this, a 15 minute drive is more than sufficient to recharge my battery."
Edit: Shit, my bad, thank you @clippy, this is the second time I've made this mistake in as many days.
It seems much more likely that you intended to respond to another post[0] and misclicked.
[0]: https://news.ycombinator.com/item?id=37971474
LTE and 5G are supposed to be better able to share spectrum, to the point where they can share the control channel, and multiplex the data channels with either encoding, but 2g and 3g can't do that, so you have to dedicate at least minimum sized blocks.
Running one min block on 2G does wonderful things for ensuring access, but it's expensive in spectrum, especially if each network needs to do it. (Because cooperating between networks is really only a thing if mandated, or out in rural areas where there's little demand)
For the sake of infinite growth, many lines have been crossed and many decisions that make no sense if not looked through the lens of money have been made.
I think companies will start doing truly unethical things in the pursuit of this in the future. What if the dashboard could show ads before the car starts? It’s just a second of your time. What if a gas station network like Circle K could sponsor your car so it tells you to get a specific type of gasoline? Why not subscription for “optional” safety features that were standard? Why not a way to remotely disable cars with a warning light on the dash “for your own safety” until they visit a certified dealer? Why not coded car parts so the customer gets a “genuine product” and doesn’t “accidentally” install a cheaper generic counterpart? Why not pay as you go features instead of paying upfront? Why not track places you drive to with your car, the conversations you have in it, the people you call, and whether you drive with your wife and/or kids for behavioural advertising elsewhere? What about an eco mode subscription that saves you money when compared to not buying it, despite that it is always free with cars today?
Maybe we could also disable some people’s cars to align with certain political demographics and identities, like we cancel, deplatform, and get people fired for slights online now? Could that paint us in a positive light in certain demographics? One client lost for a thousand gained!
Maybe an auto-911 crash detection subscription? You wouldn’t want to be the mother whose kid dies in a car crash so you’d save $19.99 a month, would you? Anything if it moves more cars or at a higher marginal revenue, right?
And let’s not forget marginal costs - what could be cheaper? Maybe some parts are too durable. A lot of things could be digital - break lines, steering, the accelerator, door locks, seat belt mechanisms, mirrors and so on. And we could save doubly by outsourcing all this work to the cheapest bidder, whether they specialize in automotive safety or web design.
Anything for the sake of growth. And it’s coming. Sooner or later, there won’t be a company in the auto industry providing appealing returns unless they charge their users more, exploit them more, or save up for a nicer dividend by cutting some corners. Not today, not this year, maybe not in the immediate future. But this is the natural conclusion of the flavor of capitalism we do.
I don’t blame capitalism for everything. It’s good for a lot of things. But it totally fails humanity when large corporations are involved. We should change it before a bunch of our industries collapse under their self-destructive behaviors driven by greed. Because right now, a good investment strategy is to constantly keep investing in companies who irreparably harm themselves and their customers long-term for a quick buck. You just need to build your portfolio of these and dump them once their sell-out is over. And this makes money. An investor would be unwise not to pick a money-making strategy.
It is in everyone’s best capitalistic interest to exploit others to the max. And when all that could be exploited is, well… no one managing companies thinks about this. The investors want returns now, all the time. The executives know how to sell-out for this and golden-parachute into the next company to do it all again. The employees are laid off and customers are mistreated, but hey - capitalism is to acquire capital. It’s not called moralism.
>Maybe we could also disable some people’s cars to align with certain political demographics and identities, like we cancel, deplatform, and get people fired for slights online now? Could that paint us in a positive light in certain demographics? One client lost for a thousand gained!
All them can be theoretically done today. If companies are really chasing after "infinite growth", why are they leaving money on the table by not implementing those things?
The answer is obvious. There's nothing stopping them from making things arbitrarily shitty, but they don't because consumers will hate it and switch to a competitor that doesn't have those things.
>Maybe an auto-911 crash detection subscription? You wouldn’t want to be the mother whose kid dies in a car crash so you’d save $19.99 a month, would you? Anything if it moves more cars or at a higher marginal revenue, right?
Isn't this already a thing today? I think GM pioneered it with OnStar. I don't think there's even a car that does auto-911 today for free.
https://www.nordicsemi.com/Products/Low-power-cellular-IoT/W...
> The answer is obvious. There's nothing stopping them from making things arbitrarily shitty
Two things to say about this: one, the laws of physics are stopping them, because developer time is limited. Two - there may be better ways to deliver growth right now, but over time there will be fewer such options.
> Isn't this already a thing today? I think GM pioneered it with OnStar. I don't think there's even a car that does auto-911 today for free.
So I guess then consumers aren't really stopping them.
> they don't because consumers will hate it and switch to a competitor that doesn't have those things.
Investors are not emotionally attached to a company, they will switch over to the ones delivering financial results this quarter. If that comes with consumer relations being irreparably damaged, does it concern an investor? They'll just hop to another company in the industry next. I said that this is an industry problem, and I meant it. It's not just one car manufacturer doing this, it's all being pushed towards doing it by the same force.
I think a lot of people have never traded even stocks in their lives and they don't see this part of it. Market-dominating companies about to enshittify their products are cash cows. And if they enshittify so much that they fall off the market, in a healthy market with many companies such as automotive, there will always be a company forced to enshittify to prop up its stock. It's not even just about sourcing investment - the net worth of the c-suite if often tied to the stock and they aren't looking to cash out when it's down.
Regardless, the overall effect is slowly making the whole industry hostile to the consumer. This is like private health insurance or life insurance - go ahead, switch to one that doesn't charge you a premium for your preexisting condition. And get your employer to switch. Not so easy when the entire market is guided by the same incentives that work against you, is it?
Ford! The SYNC system has had 911 Assist since the beginning of SYNC in the late 2000s. It uses your paired Bluetooth phone to call 911 in the event of an accident and plays an audio announcement to 911 based on vehicle information, then allows them to open the mic to the cabin.
https://youtu.be/g7xrKV3Lb74?t=67
https://news.ycombinator.com/item?id=37973829
The whole 'learning' the air fuel mixture to adjust the stoichiometric ratio over a longer period of time does not make any sense. You have the air mass, O2, fuel, temperature and sometimes more, you can calculate the mixture near instantly.
Maybe someone way back said you have to warm the car up and that takes a few minutes. Modern fuel maps achieve a fair idle from near dead cold. It's a solved problem.
Another commenter speculated that when you're out of service, the software correctly identifies that there is no tower to connect to and stops trying. But when you're in service, the code repeatedly tries to do the 3G handshake because it can see a valid tower.
One can only hope that "smart" will have very bad associations in the future (for the general public).
But it's quite easy to set timers to attempt things at regular intervals, rather than in an infinite loop until they succeed.
One attempt every 5 seconds (assuming a 5-second timeout for the request) is 17,280 attempts a day. One attempt every 5 minutes is 288 attempts a day. One attempt every 15 minutes is 96 attempts a day.
A simple repeating timer is a fantastic solution.
Does the car need to always send telemetry immediately? Can there be a 15-minute latency? If so, just put that on a timer and enjoy 99.4% power savings for offline cars, with basically the same quality of telemetry when they're online.
But what if a car needs to send telemetry at a given moment? Just move up the next timer function call.
But what if I want to get telemetry when the 3G signal is intermittent and briefly available? Just call the function once the 3G signal is available. Do this with a separate cool-down for this behavior.
But what if I need telemetry every second? Just dump all telemetry events into a buffer between successful sends, slap a timestamp on those events, and you can now send data for the past X hours all at once. Even from the black spots.
There are so many options (even less fancy than exp back-off), but this was probably built by a junior programmer without any support network. This feels like a very common lack-of-experience mistake that would be caught in code review if the reviewer cared.
It's literally just a left bit-shift? It's as easy as a linear back-off. The only thing less fancy is no back-off at all.
I guess you could always cap off the exponential interval increases at some max interval, making it basically a repeating timer.
Though there's still the question of why would you need this for telemetry. It doesn't feel like this type of "marketing" telemetry needs to be quite real time.
No you MUST have a max interval cap. Exponentials and all, you'll soon be waiting days or months for a retry.
Please cap your numbers for both user experience reasons, and safety reasons.
Yes, have dealt with this before. Nothing like doing a fault analysis and telling someone that it'll fix itself in a few months.
A capped fibonacci sequence is a very common back off timer.
*Rule One of IoT network design: NEVER assume that you even have a network * If at all possible, and to the maximum degree possible, an IoT device must work whether there's a network available or not.
* at least across module sleep/wake
It's fancy.
At least the penalty is you just repeatedly cause every car of the model in the entire country to fail instead of crashing into a wall, but this kind of thing is why the T in IoT is for "shit".
And it just supports my point, which is that it's not "literally just a left bit-shift".
It's quite a lot of work to do, actually. Which -- of course -- should be done. But there are commenters here acting like it's a single line of code.
But I think we all know what actually happened here was a rush to release and then a failure to follow up with "the update that actually does it right". The Jira ticket is probably still "open - should do, version: next" somewhere. And there's a "told you so" email chain linked in there somewhere.
So there is some justification for trying to keep a cell signal established or checking in more frequently than every 15 minutes or so. Someone attempting to remote start their car not having it actually respond for 15 minutes is likely to already be outside trying to leave and pissed off that they’re walking out to a cold car with iced up windows and thinking that the “remote start didn’t work”.
One example: This is exactly why randomness is required by Ethernet's ARP protocol: to avoid ARP storms when the power comes back on and all nodes try to get back on the network at the same time. (And ARP is still powering most IP connections (v4) today, since IPv6 is painfully fiddly by design, so no one uses v6 unless they have to...)
You must have had very crappy cars, because I rented a 2022 Renault Clio this spring and it had very good ergonomics. I was quite amazed. Only an older Hyunday i3 that I know of had annoying electrical quirks. But hey, there are tiktok videos on how to steal current Hyundai and Kia cars.
https://www.theverge.com/23742425/kia-boys-car-theft-steal-t...
Better to go with one which pairs with your phone if you even need this functionality.
I work with batteries in the EV industry (not inside cars, though, but lithium-iron-phosphate), and the list of planned improvements is not short.
There’s a gap between when a product is ready to be used and when most of the known optimisations are done. And there’s a big risk factor with the company suddenly deciding to depreciate your thing.
I try to only have only “dumb” appliances in my home: kitchen, washing, lighting. If I couldn’t avoid buying with wifi, it doesn’t get enabled.
Then ask for wifi credentials, or ask to connect through a phone. If I don't connect a device, it's because the device is not allowed to be connected. Any device that thinks it's important enough to ship with its own SIM card using an account I don't control, rather than putting the user in full control of connectivity, is a device I will never purchase.
I took it to New Zealand on roaming and despite having a connection it spent so much time looking for the home network (“Three” in Australia) the battery would last half a day.
Call your representative, tell them what's going on, and ask them what the DOT is going to do about it. It'll take 30m of your time.
https://www.subarubatterysettlement.com/
The settlement is pathetic. I could never prove the parasitic battery drain on my 2017 forester, so I just had the OEM audio/starlink system replaced with a cheap Sony CarPlay head unit and the problem went away.
But I cannot get reimbursed since I have no proof, even though surely every car manufactured with those components has a parasitic battery drain.
Then keep coming back to get the battery replaced when it "fails to hold charge".
If every owner did that it would end up like the region restriction in DVD players.
i.e. The retailers would force the manufactures to turn it off because they don't want to deal with the cost of fixing an artificial problem.
Oh and I would start posting to car websites and forums about how unreliable my car was. Marketing teams do care about getting that reputation.
There is no money in it for them, and you are wasting valuable time. You can fix the problem yourself by getting the head unit replaced with any CarPlay/androud audio one at a car stereo shop for $1,500 and move on with your life.
https://www.subarubatterysettlement.com/
I guess there is enough noise around the issue these days to get the dealer to properly fix it, but back in 2019/2020, no one believed you had a tiny power drain, and it would have been very costly to prove it (including time), and then who knows if you could litigate successfully.
Hell, if the power drain was only happening while not able to connect to 3G, the issue would be so location specific/sporadic enough, that good luck proving it even to yourself.
Where did this happen? Some players can be unlocked and others are multi-region, but to this day DVD players are still region-locked, are they not?
When the big supermarkets were still selling DVD players they used to sell them region unlocked because people would return them as faulty if they didn't.
If you go to some where like The Range they still sell DVD players and they're all sold as "multi-region" as far as I can see:
https://www.therange.co.uk/technology-and-appliances/smart-h...
https://www.tsbsearch.com/Subaru/WQZ-61R
It mentions a "3G Sunset Update".
A further search for "3G Sunset" turned up a few TSBs that apply to various Subaru models and years.
You can search for TSBs here: https://www.tsbsearch.com/Subaru
- Subaru has a fix for this issue.
- Subaru's fix has dealerships use proprietary software to put the DCM module into "factory mode" (suspended).
- It will also have dealerships physically disconnect the Starlink-related buttons (e.g. SOS), because if they're pushed it will "wake up" the DCM.
- There is no assurance the DCM won't wake up again on its own (it has its own backup battery which can fail).
- Getting dealerships to conduct the above fix is a giant pain in the butt, and they'll charge you hundreds of dollars. Few customers have even got that far, most just get an expensive "diagnostics" bill with absolutely nothing to show for it. Most won't do the repair unless you can "prove" a problem, even though it is more work to prove the problem than to proactively conduct the repair.
- Subaru won't release the software, so you can do it yourself.
Therefore, lawyers.
Just because it’s easy for some people to do you can’t expect everyone to 1) know about this issue (Remember, there wasn’t an official announcement or recall) and 2) be confident enough to modify something on their car.
Most people never do anything on their care and just bring it to the service for every small thing.
- install small solar panel behind back window. That will trickle charge battery, and will counter ballance drain
- hard switch off on battery, disconnect battery everytime car is not used for couple of days (weekend).
To be clear, I'm asking more than anything as I have very little car knowledge but to me that seems like a more preferable solution long term than frequently having to physically disconnect your car battery.
So my thought was "Why not simply remove the component physically drawing all that power and just call it a day?"
Your best bet if you actually want to do this would be replacing the antenna with a properly matched dummy load.
But that (or removing the antenna at all) is going to just leave you with exactly the problem you already have—it can’t make a connection, so it will continue retrying (and probably at its highest power setting) forever and drain your battery.
The majority of the power being drawn occurs because the system mistakenly believes the band is still usable and assumes that it probably just located somewhere with poor reception. This causes the system to ask the radio to bump up its radio power and when that doesn't work, wait x amount of time and then try again. However by stripping the cellular antenna, we make it impossible for the system from executing that behavior.
Sure it'd periodically wake up and inevitably fail to initiate a scan but that consume vastly less power. (to the point that it'd likely be a non-issue)
That said, for subaru it seems that pulling the DCM fuse seems to stop the drain: https://news.ycombinator.com/item?id=37973829
Doesn't this reset the emissions computer settings after some interval?
Not to mention any radio presets.
Previously, there were times that I would drive 100 miles one day, and I couldn't even open the doors using the automatic locks on the following day. It's been about a month on the new DCM and so far I haven't had any issues, so here's hoping...
Not once has the battery died after that, and it has been ~3 years. With the OEM starlink head unit, we would need to jump start every single week.
Needless to say, Subaru’s lack of willingness to compensate us lost them a customer for life.
https://www.bestbuy.com/site/sony-6-2-apple-carplay-built-in...
It works fine, but if I were to do it again, I would have bought a better quality one with a bigger and more responsive touchscreen.
I cheaped out because I did not know if the new head unit would fix the battery drain issue, and if it didn’t fix the issue I was going to get rid of the car.
Do you mean the key? Otherwise how are you moving the battery?
Everything I've read online about the issue points to the STARLINK system, and the common wisdom in this thread and elsewhere is to pull the DCM fuse. Unfortunately, my Forester has no such fuse, so I'm at a loss what to do with my car.
It's so frustrating, because my wife has a Subaru Crosstrek of the same year that has never had a battery problem, even with the OEM battery. She has a base model without any additional "upgraded" electronics, so that's the likely culprit in my case. I'm currently on my third or fourth higher CCA battery and had to jump it again a few days ago.
I haven't taken my car to the dealer for the parasitic drain issue, but previously tried to get dealership service for an unrelated entertainment system issue related to USB media playback freezing up the entertainment system. That fix was unsuccessful, and I have no faith in Subaru mechanics being able to diagnose and fix a potentially more nebulous battery drain issue. It doesn't help that the dealership is now 45 minutes away, so I'm not interested in wasting even more time on a hit and miss solution.
Ironically, I bought Subaru because reliability was my number one concern when purchasing a vehicle, and I'm not sure I would do that again.
Seemed like a parasitic draw but pulling fuses and using a multimeter verified idle draw at 30 milliamps which is within spec. If you find high like a few hundred milliamps you can start pulling fuses to narrow down offending item.
In the end it was a bad a battery, but was difficult to get the warranty replacement because it could be charged and work again so long as you ran it every day.
It's relatively easy to verify parasitic draw and make sure it's in that 30 milliamp range, with even a cheap multimeter by disconnecting negative and running through multimeter. Tricky part with modern cars is getting it to go fully to sleep, anything can trigger wake up (like opening the door) and it can take a few minutes for it to fully sleep to idle draw.
One such Camry in our extended family was parked for a few months and the battery failed. We got an expensive OEM replacement and tried to make sure we drove the car every week or two. It failed again in just under three years. Meanwhile, we have an entry-level Audi that also often gets only a short local trip every week or two. It is still on its original AGM battery after 6 years, and this one has to actually start the ICE.
So, I am disappointed to find that Toyota engineers could not be bothered to apply whatever basic power management logic the Audi folks did to protect the battery. Worse, the Camry has this big NiMH traction battery, and I would have imagined some clever backup strategy to infrequently top-up the 12V battery if parked too long, so it would be reliable.
In the end, I gave up and installed an old-school battery cut-out switch, so we can just disconnect the 12V battery if not planning to use the Camry for more than a few days. Every time I operate it, I'm giving the Toyota engineers a rude hand gesture in my mind.
It should be feasible to monitor voltage levels and rates of change and signal an error condition rather than silently destroy itself...
I only buy used cars, and past the ~7-year-old range used Toyotas routinely cost about twice as much as any other brand for equivalent model/mileage/condition. Yet I still pay the markup because every time I work on mine I am astonished by the fanatical obsession they have with reliability. I'm sure this value retention is reflected in the profit margin on new Toyotas.