124 comments

[ 0.18 ms ] story [ 621 ms ] thread
> Shubham Mittal, co-founder and chief technology officer of RedHunt Labs, alerted TechCrunch to the exposure and asked for help in disclosing to the car maker. The London-based cybersecurity company said it discovered a Mercedes employee’s authentication token in a public GitHub repository during a routine internet scan in January.

That's a strange way to disclose a security security issue.

Indeed, especially when Googling "Mercedes report security issue" the page litterally populates the results with the address to email so it wasn't like it's hard to find.
> Indeed, especially when Googling "Mercedes report security issue" the page litterally populates the results with the address to email so it wasn't like it's hard to find.

Reporting via a third party isn't super unusual if you think that a organisation may be a bit legal threat happy from your report.

This may be true if there isn't a vulnerability disclosure program in place but there was, thus your point is completely invalid.
No, his point remains: companies may act in bad faith, and publicly committing to act in good faith is absolutely no evidence they will not.

I don’t mean to be trite, but publishing a bug bounty program doesn’t mean you’re the good guys.

> publishing a bug bounty program doesn’t mean you’re the good guys

this is meaningless rabble. Yes you can get burned in all kinds of legitimate situations [1], but 99.xx% of bug bounty interactions do not result in any kind of legal action even if you wander a bit out of scope

[1]: https://eu.desmoinesregister.com/story/news/crime-and-courts...

> this is meaningless rabble

That is rich coming from yourself. Are you at all familiar with German law?

It’s less than a 1% chance of financial ruin!m? Sign me up!
I hope you also avoid using any kind of modern transportation, like cars, since there's a non-zero chance of dying in a crash
I would probably avoid transportation that kills 1% of its users each trip.
Username checks out. Guess you're going for 100%?
Assuming anyone is reading that email account
Lawyers will.

Then having journalists in the conversation helps, as they can produce bad press if the lawyers play games.

It’s pretty dangerous according to German law to disclose security issues.
And it is in this sense kind of a smart move if you do not want to pay for a lawyer and there is no bug bounty program with T&Cs: journalists can protect their sources. However, you then somehow need to make sure that they waive potential hacking charges. Funnily in the quoted case one of the reasons for the fines was damages related to involving the press.
> journalists can protect their sources.

But in this case not. He must have been pretty worried about German lawyers and courts, which decided really strange lately.

Wouldn't it be even more dangerous to report them to a journalist?
TechCrunch would be very far down my list of publishers were I wanting to responsibly publicly disclose something.

Hard for me to take this particular outfit seriously after they decided to optimise for engagement by running to Entertainment Tonight.

The'd be on my list if my goal was tie in someone who will give me easy, and very visible, publicity to the process.
"I contacted unrelated people and all the press to help them"
Sounds like he was just trying to guarantee a mention (advertising and clout) in the subsequent TechCrunch article…
Call it what it is, a totally rat move placing Mercedes in risk for pure self-promotion purposes - instead of going via the vulnerability disclosure channel (first result on google when googling "mercedes security disclosure".)

I hope nobody is stupid enough to ever engage with this firm after this publicity stunt.

The only thing putting Mercedes at risk are previous sentences going heavy handed on ethical security researchers that bothered contacting german companies. You can find some examples online where just "poking" with leaked credentials not unlike OP was not looked favourably upon.

https://www.theregister.com/2024/01/19/germany_fine_security...

https://www.darkreading.com/vulnerabilities-threats/another-...

there's a world of difference in contacting a company via an ethical vulnerability disclosure program and doing so without one in place. it really shouldn't be that difficult to understand. Being "german" has nothing to do with it
I wonder if any private keys used for signing genuine MB parts have leaked as well. Auto makers seem obsessed with controlling the spare part market and leaking the keys used to authenticate spare parts would make it possible for other vendors to produce compatible parts in the future.
> Auto makers seem obsessed with controlling the spare part market

It is a major source of revenue and allows for markups to be optimized for brand.

A lot of parts are shared across lines within a manufacturer and across manufacturers.

MB wants their customers buying MB parts with an MB markup. Even if it's a Bosch part that is identical in a Ford.

Indeed. Ironically I had a porsche 928 which used an air actuator from Mercedes, and it was a fraction of the price from them. Parts guy at merc was a little sniffy, but it saved me £100!
Lots of German car components are generic Bosch, just with different markup depending on whether you're looking at Skoda, VW, AUDI, BMW, MB or Porsche in that order.
Mercedes used to be insanely good at parts - I could order brand new stock parts for a 25 year old 240D with no issues and at quite reasonable prices.

Those old cars were clearly designed by engineers who cared; the hood had simple latches on the hinges that when opened would let it open to vertical.

https://www.classiccarstodayonline.com/2022/04/22/mercedes-1...

The 123s and first generation 124s were little jewels. So much brilliant engineering.
November 1975 to January 1986 - that's the kind of duration of a body line we need to encourage.
(comment deleted)
Automotive software should be mandated to be open-source.
?

Why?

I'm not the person you replied to but I agree, because it's related to public safety.

Same with aviation software.

Anything that deals with public safety should be audited by the public.

Maybe we could start with this much less general principle:

Whenever the output of software is offered as evidence against you in court, you should be entitled to the source code of the software.

The post office workers in the UK who where f-ed over by the government because the software was reporting things incorrectly could really have used this as well. Should all our financial systems be open source too?

Maybe it should be like Copyright or patents, you can keep things private for a while for an advantage, but then you would need to release all code as open source / public domain or similar? It should also be available before that to be reviewed by professionals following a formal process of some sort.

> Should all our financial systems be open source too?

Yes.

That's really a good thing you are on to there.
More downvotes than upvotes. :/
HN doesn't like calls to take responsibility for what we produce. Regulation and responsibility are anathema here. But no worries, the world is fortunately a lot larger than that and little by little the tide seems to be turning towards holding the producers of software accountable for the works they produce, just like every other engineering discipline.

I can't wait.

It should be audited by professionals, but available for public reviews / additional audits.

General folks and even most devs do not have the skills to audit code.

Nothing prevents the owner of the code to audit it themselves, but everyone else should also be able to.

I think if you release code related to important everyday things like cars, everyone who is able will want to take a look. And you never know who will think outside the box and stumble across something they can then escalate and investigate with other people who are more capable. It happens in open source all the time.

Auditing your own code doesn't help if you have a blindspot. You will miss it while developing and auditing.

> Everyone who is able will want to take a look.

This is not true. Auditing is very skill intensive and time intensive. There needs to be compensation for it either direct (you pay for professional auditors) or indirect (you're an academic and you get reputation and grants for breaking WPA3, SGX, Mega or SIKE, or fame for breaking the PS3).

You can easily get in a situation with "The Emperor has no clothes" where everyone thought open-source code must have been audited by someone because it's open, when in fact no one did because they didn't care or had no incentives.

We live now in a world where the most qualified people are explicitly not part of the old definition of "qualified professionals"
What do you mean?
There's a lot of airplane enthusiasts which have very extensive knowledge in the field but never will be included in any of these "qualified professional" lists.

The knowledge nowadays is spread.

Do they have background in fuzzing or formal verification? That's what you need if you want to make high-assurance software.
I'm sure there's a lot of people specialized in that as well
There isn't
And here 2 years to find a buffer overflow vulnerabity in glibc that was found via a fuzzer: https://news.ycombinator.com/item?id=39194093

In a library used on billions of devices.

if that was a private project at some company, it would still probably be there. In the long run openness always win for reviews. That's what inevitably happens in a world where the knowledge is accessible to anybody. The only way to go against that is pour some tremendous amount of money and the result isn't even guaranteed.
It's a project used by millions of people yet it took 2 years.

What about projects used by 10~50 people?

What kind of strange argument is this? If it wasn't public, it would have never been found at all.

I'm going to trust more a project peer reviewed by the best devs on the planets compared to some homebrew homemade software where the devs allegedly know better than everybody else (and they don't)

Your argument is that asking for professional auditor is not necessary for open-source code.

My argument is that open-source is not enough when high-assurance is needed and devs or end-user should still ask for a professional audits.

I'm not saying professional auditors are useless, just that they will never reach the feedback you get from peer reviews and are not a replacement for peer reviews anyways.

We're not in the 90s anymore and it's time to acknowledge that the software world and even the world in general has changed. Knowledge is now spread and the most knowledgeable devs aren't working at an audit firm and might not even hold a computer degree!

The people who found the bug in glibc worked at an audit firm.

Peers aren't enough. Finding vulnerabilities require training and an adversarial mindset that is rare.

There is a reason why in cryptography people say "don't roll your own crypto"

(comment deleted)
I've thought about this a couple of times.

In a perfect world, you want to be able to safely ~hack~investigate every part of your car, including the logic that controls its servos, hydraulics, propulsion, etc.

So the whole OS and controler firmware should be open source, right?

It would be awesome but problems arise with qualifying which "devices" this should apply to. Are all self propelled four whelers cars? What about three wheelers? And why make a distinction on wheels? And if propulsion is the driver then I can see dealers suddenly selling you (proprietary) carriages and an added service of mountig an engine on them.

Soon enough you come to the point where absolutely everything needs to be open source. And while I would be happy to be part of that world, I don't really see a tangible way towards it. It just seems there is too much vested interest in defending the right for proprietary software.

The one avenue where there might be a sliver of hope for such an idea is maybe commercial aviation. That field is already overly regulated so it might be doable. There still will be a huge loophole in terms of military vehicles but it still seems vastly more rwalistic than forcing all car companies to opensource everything.

Vehicle categories already exist in all countries, for the purposes of registration, taxation, safety, etc.
> In a perfect world, you want to be able to safely ~hack~investigate every part of your car

I really don't think the standard "safely ~hack~investigate~" is reasonable, for both practical and policy reasons.

By focusing on an imaginary perfect world, you take focus away from the real word, where we know people do unsafe hacks to their car already, even without touching software, like https://honda-tech.com/articles/crash-kills-man-in-home-buil... .

There is no way to prevent those unsafe hacks that wouldn't also prevent huge numbers of safe hacks which people accept and support as reasonable, and place a huge and undue amount of control into the auto manufacturer.

In the old days of Ma Bell, AT&T tried to prevent Hush-A-Phone from selling a mouthpiece cover designed to make it harder for others to overhear your conversation. AT&T argued that a foreign attachment like that could damage the phone network. A decade later AT&T sued Carterfone for selling an acoustic coupler to a radio, using the same justification of possible damage. AT&T lost both cases, and the latter case is why early personal computer modems were acoustically, not electrically, coupled.

We should not extend that same power to auto manufacturers.

> but problems arise with qualifying which "devices" this should apply to

Which is why this sort of legislation is done in steps. Focus on where it makes the most sense, get experience on how useful it is and what the negative consequences are, and use that experience to judge if anything else should be included.

That's why the Motor Vehicle Owners' Right to Repair Act (see https://en.wikipedia.org/wiki/Motor_Vehicle_Owners'_Right_to... focuses only on motor vehicles, and not the right to repair all machinery and devices.

> safely ~hack~investigate

I was more thinking in terms of:

1. Hacking something close source is inherently "unsafe" as you have to reverse engineer it, fuzz it or do something other that could randomly backfire. 2. Hacking something open source is safe(er) as in, you can see the code and the developers intent, maybe even some insightful commentary.

Also, there might be an argument about open source projects being more easy to replace pieces of as you typically can see the whole build stack.

The safety goals you want in your perfect world are so far beyond what is practical that it seems irrelevant in a discussion about a mandate that auto software be open source.

Please remember that "open source" means access to the source code, plus the right to modify it, plus the right to redistribute it without requiring additional permissions from the copyright holders, plus no restrictions on who may use the software (other than those required by law).

You can have close source with access to the source (often called 'source available'), and even with the right to modify and use those modifications.

A license which is "for noncommercial use only" or "not for use by the military" is not open source, even if the source can otherwise be freely used, modified, and redistributed.

That said, having the source code in any form does not mean things are no longer 'inherently "unsafe"'.

Device drivers are notoriously finicky and depend, for example, on the precise timing of hardware vs. software. Reading the source, even with comments, may not be enough to assume modifications are safe, perhaps because it's obvious to an expert (which you are not) or because the original authors were not aware of the issue.

Nor does open source guarantee you can reproduce the build. The auto manufacturer may use a compiler with different behavior than your compilers. But hoseja called for open source, not reproducible builds of the open source software, so inherently accepts there will be some risks.

Certainly there's less risk with reproducible builds of the open source software, but as I said, these changes are best done in steps. There's no need for the huge step you want in your perfect world. Instead, mandate open source, get experience, and use that experience to guide future changes to public policy. That's how laws generally work.

> Reading the source, even with comments, may not be enough to assume modifications are safe

Is it enough to say that the user gets a better chance to judge their impact compared to flipping random registers?

Sure. Just like it's good to have manufacturer's information on how much torque to use on the engine block bolts rather than guessing based on possibly limited experience.
Absolutely not. Saying this as a huge proponent of open source.

If it’s the case, each clown garage shop would be able to modify key characteristics of any car. And oh boy they will do it.

Would you fly on aircraft knowing mechanic servicing it last night could have added something funny to the plane you are taking?

There is a difference between being able to inspect study and modify software for a car and installing modified versions on your or someone else's car. We already have regulations covering radio emissions, this could be the same: modify all you want but you can't install a modified version unless you validate the modifications with a certification authority. Btw, the cyber resilience act in EU will cover these cases.
Being closed source doesn’t strike me as an effective mitigation tactic. You can already buy aftermarket ECUs and such, and it’s pretty unlikely the garage would install anything that would invade your privacy more than what the manufacturers have installed. If they were going to do it as a prank or shortcut or misguided enhancement, they can do exactly the same thing with physical parts. I guess I just don’t see the threat vector here. The big benefit seems to be auto manufacturers being able to rely on security through obscurity, which has already failed:

https://news.ycombinator.com/item?id=35452963

You give the mechanic an opportunity to cut your brakelines too and they don't do it. It's part of the appliance, it should be accessible. Maybe I was imprecise with open-source, I meant more source-available.
> If it’s the case, each clown garage shop would be able to modify key characteristics of any car. And oh boy they will do it.

Car workshops would modify and compile their own distribution of the car source code? I can't say I have ever been to a workshop where I would imagine anything like that.

Open source here would clearly be a big win for security and bug identifications in cars. Better quality laws to go along with it to protect researchers would naturally be a big positive as well.

For a comparison look to Android. AOSP is open source, and whilst alternative, non-OEM, flavours of Android do exist (GrapheneOS, LineageOS, etc). But you don't see shops that fix or sell phones putting any of these on the phone. And if you did, would it be a security downgrade? I don't think so!

> Would you fly on aircraft knowing mechanic servicing it last night could have added something funny to the plane you are taking?

But... they could have done. Maybe not the software, but mechanically, of course it's possible. Why doesn't it happen though? I guess the same reasons why in general people act responsibly in society.

As clown garage shop I already have access to the ECU, TCU and so on. And can change lots of internal settings. That's what we all did in the first auto-mechanics course.
You still can’t change many things and that’s great.
If you have the .A2L file everything is accessible. And I had all.

Thanksfully the clown garage shop doesn't. But CAN access SW is available for most models

The mechanics absolutely could, though.

In fact, it's a core thing in aviation for maintenance, repair, but also modification/changes to be decoupled from vendor.

It's pretty normal for a vendor to not exist for a generation or two while the plane is still in productive work.

There’s heavy licensing when it comes to aviation. You can’t buy Boeing parts on ebay sold by chinese vendors and install it on aircraft. At least yet.
... well, you can. Sometimes it even involves purser walking cap in hand scrounging up cash for a fuel pump replacement in pretty young 787 otherwise it's not going anywhere.

The main difference is that it's tracked. Licensed, too, but huge part of that licensing is about maintaining the history of the parts and plane, and that mechanics know given plane.

But the only reason you're going to know that a change happened was because the people have been trained to log it down - but there's no physical/computer lockout preventing them.

Is this a joke? Mechanics can already swap critical parts with substandard replacements or finnagle sensor overrides with some tape. Why should software be so sacrosanct to the manufacturer and not the owner?
I would like to start with forcing them to allow it as a low/no cost option, at least for some models.

Imagine people had the option when buying a car, truck or minivan of having it be open and tinkerable?

Most people won't care, but some hobbyists will. Some companies will love it, some startups might be founded for it. But eventually there will be a killer app, or huge benefit, and it will become more common until it is standard.

I'd imagine certain groups like farmers would jump on buying open source trucks, hoping they can get them to last longer, or repair them cheaper.

I am not sure it's manageable for a society, when random people start messing with code controlling breaks and stuff. You'd think that's neatly separated into a single purpose IC, but I fear it isn't. Gonna be a nightmare for insurances and courts.
There is no software locks to prevent modification to existing mechanical cars that can and are modified to be dangerous and there is inspection processes etc and laws to deal with it. It’s even easier to scan for modified software than look inside an engine.

There are places for safe experimentation like tracks, so I don’t see why this fear isn’t irrational or a ploy for control.

> Imagine people had the option when buying a car, truck or minivan of having it be open and tinkerable?

[...]

> There are places for safe experimentation like tracks, so I don’t see why this fear isn’t irrational or a ploy for control.

I don't think many people track modded minivans. "No the Subaru WRX is my DD, my main track car is a Honda Odyssey"

It’s extremely hard to figure out whether modified software poses a security risk. I wouldn’t trust experts spending long hours to get it right, I certainly wouldn’t trust people at TÜV doing spotchecks to find the bugs.
I wouldn't trust the original developers either.
In many US states, there is not even a meaningful safety inspection. In some there is no inspection required at all.

People won't intentionally make their vehicles dangerous, but it's easier to make bad software than to completely bodge a physical repair using manufactured parts.

I don't know how vehicle software works, but hopefully they don't crash like consumer software. Someone here knows the answer -- is there like a supervisory manager/kernel that keeps all the subsystems running? And modular separation so that bad changes to the fuel injection system won't take out the fly-by-wire brakes?

Listening to the recent CES interviews by Jeff Geerling where car manufacturers explain how they see the process, with SOAFEE (the ROS knock off for cars) and other systems where they want software defined vehicles and complete freedom to change anything (and charge for those changes ofc) after they already sell them them it's honestly completely horrifying from the end user perspective.

As much as it would be nice for hobbyists, this is one can 2.0A of worms that needs to be kept as shut as possible by lawmakers.

> The repositories include a large amount of intellectual property… connection strings, cloud access keys, blueprints, design documents, [single sign-on] passwords, API Keys [...]

That doesn't really sound like security best-practices would be applied. Why don't they use a credential store?

Just guessing as an outsider, but... it's a big, conservative car company trying to do software development. Reasons could include:

- Too much red tape/risk assessments/effort/time required to set up a credential store

- Devs working there may not know/understand the importance of it, and may not be up-to-date with modern software development practices.

- Assumption that Github repo will always be private, correctly configured, never leaked.

- Assumption that employee computers with code checked out will always be full disk encrypted and source code never read by a malicious program/transmitted somewhere else.

If you work in a company that makes software for a living, it's worth bearing in mind you are probably nearer the forefront of modern best practices and there are many companies in other industries that do some software as part of, but not the main part of the product, and these do not necessarily focus on software development and therefore may be "as hot" with best practices, to put it mildly.

I mean, I think it's fair to say that given they're developing their "Drive Pilot" level 2 SAE software, at least some of this code involves life-or-death systems. For that reason alone I'd expect a higher level of security awareness, so seeing how unhygienic their repos are from a sec perspective is a bit unsettling.
Agreed it is unsettling.

For what it's worth, there's some peace of mind in that this software is probably tested much more thoroughly than the average piece of Web software or whatever. Version control / security best practices / clean code may be too abstract for these old companies, but testing isn't. You'd hope.

I do freelance security consulting. There's a cultural element to security that, when missing, leads to obvious problems like this. I'm struggling to think of an engagement where I didn't find improperly stored plaintext credentials somewhere. Credential stores are a constant recommendation.
It seems like they made a lot of assumptions that something like this wouldn't happen. They assumed employees would never leak secret information, and that their GitHub repos would never be exposed. They could've used https://doppler.com or AWS Secrets Manager (https://aws.amazon.com/secrets-manager/) and never had this problem. It's a little too easy to get comfortable thinking things work well the way they are. This should be a warning to other companies to seriously evaluate how they're storing and managing application secrets and credentials.
(comment deleted)
So, maybe someone can go and help fix their miserable mess? MB used to make awesome cars. Then they 'modernized' their interior and it became a UI mess and then they added more and more software and it became unreliable and sometimes downright dangerous. I have driven their cars on and off for decades but I'm really done with them. Highly unlikely I'll have another MB as my daily driver.
Try a Tesla, I hear their software is tip-top!
I haven't been in a tesla in a number of years. First and only ride.

The map of cars around us having cars jumping to new locations and flipping between motorcycles and buses for the same neighboring vehicle did not inspire confidence. The car seemed confused on its surroundings.

An exiting lane to our left on the freeway came to a stop during autopilot and thought our lane stopped so it slammed on the breaks nearly causing the car behind us to crash into us.

Well, at least you were wide awake after that. TBF to Tesla, my Mercedes tried to kill me twice as well before I got rid of it so I'd call them about even in that respect.
(comment deleted)
> The repositories include a large amount of intellectual property… connection strings, cloud access keys, blueprints, design documents, [single sign-on] passwords, API Keys, and other critical internal information.”

Any senior developer or a security researcher knows this kind of secrets should not be part of the source code repository. Has Mercedes-Benz a rotten software development culture?

Senior developer or a security researcher?! This is like.. maybe you have to tell an intern kind of thing.
Many car companies are transitioning from a predominantly mechanical engineering space to a predominantly software engineering space, along with all the legacy mindset that comes with. It's par for the course when they can't understand that they're now a tech company and need to invest as such.
What I want to know is how are they not using a secret manager (like Doppler or AWS Secrets Manager) at a company of this size? They could have avoided all of this. These types of leaks can cost companies everything (from their data to customer trust).
The root cause: They are not using it because engineers who committed securities to a GitHub repository are not going to have any repercussions. Their salary does not depend on, like the salary of their bosses. So it does not cost anything on a personal level.
You are very correct but it’s also not on the dev product teams to decide the best practices for secrets at large companies. It’s the responsibility of the security and devops team to implement best practices.
Sure, it ought to come from above. But what software dev in 2024 doesn't know that keeping everything in the source code is really really bad?
They call themselves a software company. The guys in India that develop the software know what they are doing but the managers in Germany have no clue what they are doing. Sitting all day in meetings planning strategies for beating Tesla. The guys in India are mostly fixing bugs in software released years ago.
The headline says "exposed source code". The article says it exposed keys, SSO passwords, and other things that sound much worse than exposing source code.

Some tiny fraction of source code has value as intellectual property, but most of it is probably not valuable or sensitive. Passwords/Keys/PII/Documents on the other hand could be very bad if they got lost.

[flagged]
> Your source code repo should never contain secrets like API keys.

Why not?

1) Because exactly this happens

2) Because developers check out that source code and store it on their own laptops, which can sometimes get lost, stolen, or hacked.

3) Because with a proper secrets manager, each developer can get their own auth credentials for each service, increasing auditability.

Ask Mercedes-Benz. If somehow that repo gets exposed - so are all of your secrets. You just made a problem a lot worse and potentially opened up an avenue to expose more data.
Are you by chance affiliated with the secrets manager solution you've attempted to promote a fair few times in this thread?
Yes- More people need to know secrets managers exist and the types of problems they solve. I'm passionate about helping people discover and understand what secrets managers do. Using any secrets manager would be better than none at all - it does not need to be our solution as there are plenty to consider.
Are you associated with Doppler?

You have been linking it multiple times in this thread, and this comment sounds like a sales pitch.

Yes I am. More people need to know secrets managers exist. Doesn't have to be Doppler - but they should be used more often.
Passwords get leaked. Passwords get stolen. Passwords get guessed. This has always happened. This will always happen.

If exposure of a string of characters is all that stops your source code from being accessible then you’re just not concerned about security whatsoever. Can’t complain about your house being robbed when you left the door open and the lights on.