> On Mac, some command-line tools included with macOS or the Xcode Command Line Tools (including curl, git, python3, ruby, perl, and openssl) check certificates against /etc/ssl/cert.pem. For certificates that chain to a root CA certificate in this file, these tools don't check Apple's block or whether the certificate has been revoked. This file will be updated in a future macOS update. If you use these tools and don't want to wait for the update, you can remove root CA certificates from the file.
God I'm so sick of this "everything-is-a-conspiracy" mindset. Equally likely that they're incompetent, that they simply forgot, that it's difficult because of legacy code reasons.
Going against the grain here but I think this is decent security engineering from Apple. They patched the highest exploitable things immediately (Safari, Mail, etc). The residual risk is using these command line tools while someone is actively MITMing your connection.
How often are people “git clone”ing a repo without browsing to the GitHub site first? Safari would throw up certificate errors. And also, TLS on Git doesn’t even authenticate the code you’re downloading is legitimate at all. It could be modified by the Git host or a supply chain attack, etc.
Yea, it’s not great. Ideally the risk is reduced to zero from patching. But a lot of the times we accept some level of cybersecurity risk out of practicality. This is why we have responsible disclosure windows lasting months. It takes a while to do proper release engineering and testing for something like a whole OS update.
One of the certificates literally has a private key of 4; it's as compromised as it could quite possibly be. This is not a "one of the intermediates have a weak and possibly crackable key" or "a bunch of incorrect certs were issued but the issuance was patched".
To a MITM attacker, git clones can be easily distinguished from visiting in Safari. You could choose to only poison git clones; without poisoning anything else.
Heck, you can probably build an exploit kit in a hour; camp out around a Starbucks or McDonalds with open wifi, and start compromising git clones and deliver malicious payloads.
An attacker absolutely cannot “easily distinguish” between a git clone and visiting the website and swap the certificate.
To know whether a request is a git client or web browser, an attacker needs to be able to inspect the contents of the web request (to check the user agent or whatever). If the site is using a legitimate certificate, the attacker has no way to inspect the contents because it is encrypted with a key they do not control.
If an attacker used their own, eMudhra-signed certificate, it would throw the certificate error in Safari.
1) Why wouldn't you have a SSOT for certs and instead rely on patching different apps individually?
2) Doesn't this violate defense in depth? Exploits for a widely disclosed issue now have turnaround activation within hours, not weeks. Not patching all surface areas and actively disclosing as such seems very much like drawing up a map for a jailbroken LLM to gin up a rudimentary attack.
3) Also doesn't this violate the principle of secure by default especially for power users who are more likely to be the ones with higher privileges across distributed systems?
1) I would argue that different trust stores are a carryover macOS’s Unix heritage. Having multiple roots of trust is actually very common. For instance Firefox maintains their own root of trust from the system, it’s called NSS. Different programming languages like Java and Rust also often use different roots. It’s not amazing, and has holes, but a lot of it comes down to historical reasons and a certain level of risk acceptance.
2) Yes, but actually, this is exactly why we have defense in depth. This is a temporary measure while a permanent patch is coming. Defense in depth means a single hole does not result in compromise. I would argue that this situation with Apple demonstrates defense in depth actually working as intended. If this was the permanent solution, and no further patch was coming from Apple, then yes, it would violate defense in depth.
Historically, stuff like openssl and cli command tools used flat files like /etc/ssl/certs.pem.
Browsers came along and they wanted to maintain their own cert store because operating system flat files were often out of date.
Operating systems grew their own stores because they wanted some kind of registry that was more integrated with the rest of their ecosystem.
Nobody coordinated. CLI tools and unix libraries don't care about browsers or O/S registries, operating systems have requirements so that they can't use the flat files, everyone screams about stuff like systemd turning unix into windows.
And it will continue to proliferate. System management tools often ship their own certs since the O/S certs can be out of date and untrustworthy and may not be able to talk to AWS before you bootstrap the management tool and update the system certs because people still run ancient shit like RHEL5. This is more or less the same reason browsers ship their own as well. I've made the decision to ship an embedded cert store in a product before, and nothing has fundamentally changed that would cause me to rethink that decision (even though for people who have an up-to-date O/S it is strictly additional cost that they now have to track a cert store in another product--but you have to build for the lowest common denominator).
Good on Apple for providing such a detailed article. Fuck everyone who thinks they're doing badly by not updating this file remotely on current software versions in the wild -- most likely their Rapid Security Response / Background Security Improvements framework does not support this file (though perhaps it will). Being transparent and open about what administrators can do about this now is the RIGHT APPROACH, and no other vendor has been nearly as thorough about this.
They're doing badly because they have a market cap at 5 trillion USD ($5,070,000,000,000 to $5,110,000,000,000) and chose to ditch the first principals of a CA store. Doing impressive backflips to compensate for lack of understanding first principals is exactly that.
Google, Microsoft, Mozilla, several TLS libraries and language runtimes, and even many Linux distros run by volunteers had this issue comprehensively fixed over a week ago: https://emudhra.we-have-a-problem.com/#developers
Meanwhile, it's unsafe to run `git clone` on macOS unless you manually edit some files, and then edit them again after a macOS update puts the dangerous root certificates back.
The article warned of this already (re-applying the fix after updates). That update was likely already in the pipeline before this debacle. No one can say for sure though.
"AppleCar sometimes explodes on bumpy roads unless you manually disarm the bomb in the trunk before driving. If you do not drive on bumpy roads, you don't need to do anything. If you have disarmed the bomb, Apple Updates will automatically re-arm the bomb after every update. Please remember to manually disarm the bomb after each update."
It's right there in the article, look how proactive they're being!
> most likely their Rapid Security Responses / Background Security Improvements framework does not already support this file
Why would it support this file? Are CA trust stores really that important? /s
> and no other vendor has been nearly as thorough about this
1. That's totally not true. Apple has messed up something that is trivial to fix.
2. The EU's CRA is going to crush them if they don't get their shit together.
They didn't foresee this, and that's on them, sure. But their response to this, given the situation, is just about the best I could expect from them given the circumstances.
Would it have been better if the next operating system update already shipped a fixed store rather than another broken one? Sure. Would it have been even better if they had the ability to remotely fix the store on the many millions of already online Apple devices? Absolutely. But if they couldn't do either of those things, which seems likely at this point, this is the next best thing while they get the rest of their shit in order.
I don't think it's particularly constructive to point out how everybody else has already been able to patch this in all places. Apple's clearly in a spot right now. Blame them for that, but don't blame them for picking the best option at their disposal right now. In other words, blame them for overlooking this when designing their security response framework, not for failing to use a capability that doesn't currently exist.
I'd really like to know why it's taking Apple so long to ship a fix. Especially if they can explain how to update an existing computer, what's the general holdup? Is it their release process? Is there some underlying complexity that's poorly understood?
No, this is probably just the result of Apple getting hammered over and over on every release for bugs slipping into the releases and updates.
People were just losing their minds on here over bugs in the 27.2 beta release that hasn't even shipped yet, with wild conspiracy theories about what Apple was doing.
So you wind up with a typical megacorp release process where nothing can happen if it isn't signed off in triplicate and two operators crack open the books with the missile codes and turn the keys at the same time to launch the update.
The result is that shipping a single file for a critical security update takes way too long.
I'm rally disappointed in the Apple response here. I can't understand why they wouldn't consider `/etc/ssl/cert.pem` in scope for a security update. This arbitrary boundary has no justification at all, and makes people less secure.
There is justification for controlling the scope of security updates. Apple practices incredible defense in depth. This is one of the reasons they can claim such privacy guarantees compared to other vendors. But, I agree that they should have been able to update this file automatically, and hope they will add that capability for the next time something like this happens.
It is in-scope for a security update, and they say one will be coming.
They just couldn’t modify cert.pem without a security update like they did for their own SSL APIs. Hopefully a security update will be coming soon.
Wonderful AWK command there. For those with hobbies other than AWK, here's what it does:
- set variable s to that string: -v s='=== /C=IN/OU=emSign PKI/O=eMudhra Technologies Limited/CN=emSign ECC Root CA - G3'
- on lines matching the regex between //, set skip to false: /^(===|###) /{skip=0}
- on lines matching s, set skip to true: $0==s{skip=1}
- if skip is false, print the line: !skip
Note the last part where it just says !skip without a curly braces. AWK defaults to print in that case.
It's wild to see people supporting this bizarre anti-security response. Not only are they dragging their feet in fixing the security issue, they're actively reverting any manual fixes silently with each new update. That's legitimately worse than doing nothing at all! It's also disconcerting to see folks advocate for the "let's wait to fix this critical security issue because we want to make it slightly easier in the future" approach, as if Apple is incapable of deleting ~12 lines from a file as part of the normal patch release? If they can revert your manual fixes, they obviously ship that file and so could have shipped a fix with that file. Genuinely disturbing to support any of this.
Which is kinda wild considering there's also a lot of openness zealots, too. People advocating for open source hardware, software, right to repair, etc.
Yes, a huge part of why I bought Macs for my company is for some of this to be automatically handled. Only now are we getting any communication and still no date on when this will stop being rolled back on updates.
Please, Apple, break the process on getting a fix out for this. I want to stop doing a morning check in with my team about certs.
This is some weak sauce. I've just ditched iPhone for GrapheneOS and Apple Watch for Pebble. I'm getting more convinced something like a Framework with Linux is the answer.
43 comments
[ 3.9 ms ] story [ 145 ms ] threadWhat a clusterfuck
they solved the issue for the people they care about
How often are people “git clone”ing a repo without browsing to the GitHub site first? Safari would throw up certificate errors. And also, TLS on Git doesn’t even authenticate the code you’re downloading is legitimate at all. It could be modified by the Git host or a supply chain attack, etc.
Yea, it’s not great. Ideally the risk is reduced to zero from patching. But a lot of the times we accept some level of cybersecurity risk out of practicality. This is why we have responsible disclosure windows lasting months. It takes a while to do proper release engineering and testing for something like a whole OS update.
One of the certificates literally has a private key of 4; it's as compromised as it could quite possibly be. This is not a "one of the intermediates have a weak and possibly crackable key" or "a bunch of incorrect certs were issued but the issuance was patched".
To a MITM attacker, git clones can be easily distinguished from visiting in Safari. You could choose to only poison git clones; without poisoning anything else.
Heck, you can probably build an exploit kit in a hour; camp out around a Starbucks or McDonalds with open wifi, and start compromising git clones and deliver malicious payloads.
To know whether a request is a git client or web browser, an attacker needs to be able to inspect the contents of the web request (to check the user agent or whatever). If the site is using a legitimate certificate, the attacker has no way to inspect the contents because it is encrypted with a key they do not control.
If an attacker used their own, eMudhra-signed certificate, it would throw the certificate error in Safari.
1) Why wouldn't you have a SSOT for certs and instead rely on patching different apps individually?
2) Doesn't this violate defense in depth? Exploits for a widely disclosed issue now have turnaround activation within hours, not weeks. Not patching all surface areas and actively disclosing as such seems very much like drawing up a map for a jailbroken LLM to gin up a rudimentary attack.
3) Also doesn't this violate the principle of secure by default especially for power users who are more likely to be the ones with higher privileges across distributed systems?
2) Yes, but actually, this is exactly why we have defense in depth. This is a temporary measure while a permanent patch is coming. Defense in depth means a single hole does not result in compromise. I would argue that this situation with Apple demonstrates defense in depth actually working as intended. If this was the permanent solution, and no further patch was coming from Apple, then yes, it would violate defense in depth.
3) Related to above, yes. But it’s not permanent.
Browsers came along and they wanted to maintain their own cert store because operating system flat files were often out of date.
Operating systems grew their own stores because they wanted some kind of registry that was more integrated with the rest of their ecosystem.
Nobody coordinated. CLI tools and unix libraries don't care about browsers or O/S registries, operating systems have requirements so that they can't use the flat files, everyone screams about stuff like systemd turning unix into windows.
And it will continue to proliferate. System management tools often ship their own certs since the O/S certs can be out of date and untrustworthy and may not be able to talk to AWS before you bootstrap the management tool and update the system certs because people still run ancient shit like RHEL5. This is more or less the same reason browsers ship their own as well. I've made the decision to ship an embedded cert store in a product before, and nothing has fundamentally changed that would cause me to rethink that decision (even though for people who have an up-to-date O/S it is strictly additional cost that they now have to track a cert store in another product--but you have to build for the lowest common denominator).
People are going to be running pull or push much more than clone, and will often be running those without opening the origin on Safari.
Meanwhile, it's unsafe to run `git clone` on macOS unless you manually edit some files, and then edit them again after a macOS update puts the dangerous root certificates back.
In what world has Apple been the most "thorough"?
* apple told admins how to fix the vulnerability * apple shipped update 27.2, with no fix (today) * the update reverted the admin fix quietly
It's right there in the article, look how proactive they're being!
Why would it support this file? Are CA trust stores really that important? /s
> and no other vendor has been nearly as thorough about this
1. That's totally not true. Apple has messed up something that is trivial to fix. 2. The EU's CRA is going to crush them if they don't get their shit together.
Would it have been better if the next operating system update already shipped a fixed store rather than another broken one? Sure. Would it have been even better if they had the ability to remotely fix the store on the many millions of already online Apple devices? Absolutely. But if they couldn't do either of those things, which seems likely at this point, this is the next best thing while they get the rest of their shit in order.
I don't think it's particularly constructive to point out how everybody else has already been able to patch this in all places. Apple's clearly in a spot right now. Blame them for that, but don't blame them for picking the best option at their disposal right now. In other words, blame them for overlooking this when designing their security response framework, not for failing to use a capability that doesn't currently exist.
They can follow the 4 steps (3 was not optional for me) to remediate.
No, this is just laziness or stupidity on apples part.
People were just losing their minds on here over bugs in the 27.2 beta release that hasn't even shipped yet, with wild conspiracy theories about what Apple was doing.
So you wind up with a typical megacorp release process where nothing can happen if it isn't signed off in triplicate and two operators crack open the books with the missile codes and turn the keys at the same time to launch the update.
The result is that shipping a single file for a critical security update takes way too long.
Come on Apple, do better.
Please, Apple, break the process on getting a fix out for this. I want to stop doing a morning check in with my team about certs.
Don't have a stranger here summarize/concatenate them for you