221 comments

[ 0.20 ms ] story [ 16.3 ms ] thread
This file is used by a bunch of CLIs. Apple is fundamentally dropping the ball here from a security perspective.
Running (uname -a) Darwin arrakis.local 27.2.0 Darwin Kernel Version 27.2.0: Wed Oct 14 23:47:07 PDT 2026; root:xnu-13432.41.4~25/RELEASE_ARM64_T8132 arm64, I get the following for both:

    curl: (60) SSL certificate OpenSSL verify result: unable to get local issuer certificate (20)    
    More details here: https://curl.se/docs/sslcerts.html

    curl failed to verify the legitimacy of the server and therefore could not
    establish a secure connection to it. To learn more about this situation and
    how to fix it, please visit the webpage mentioned above.
Uname is not enough here to identify what has or has not been done to your system to deal with the compromised certificates.
No, no, I meant it to show that the machine is running the current/latest macOS, and the results I am getting from curl on the two URLs.
Maybe you have another curl in your path?

    $ uname -a
    Darwin MacBook 27.2.0 Darwin Kernel Version 27.2.0: Wed Oct 14 23:47:21 PDT 2026; root:xnu-13432.41.4~25/RELEASE_ARM64_T6030 arm64 arm Darwin
    $ which curl
    /usr/bin/curl
    $ curl https://dv-g3b-emudhra.we-have-a-problem.com/
    Your client successfully connected to the site. That's bad!
You could be running a different version of curl, e.g. homebrew. Try `which curl` and `/usr/bin/curl <url>`.
That's disingenuous title

It should be "Apple still ships compromised CA in /etc/". Git is just a victim like every other app using system CA

The lede is certainly buried, but calling out an affected development toll on a tech site isn’t a bad call.

Pretty disappointed that Apple hasn’t pushed out an emergency patch to address this, though.

(comment deleted)
I think this is basically related to apple's SIP system protection: they haven't figured out a way to update this quickly and would need to roll this out in like an OS update, which can take a while.

They might need to make a way to do this outside of os updates

Things like this is why I moved back to Linux after 10 years on macOS.
Not sure what Linux offers that's any better. Didn't Debian ship trivially weak SSH keys for over 2 years because they were trying to silence Valgrind warnings and the random maintainer had no idea what they were doing and there was no oversight or peer review whatsoever?
In 2008.

I don't think its wise to make current security decisions based on screw ups from almost a decade ago.

i think you mean almost 20 years ago old man
That's one issue from one distro 18 years ago. Have there been recent things? This thread is about a today's MacOS, and an issue known publicly for more than a week.
You are going so back that gmail was served over HTTP (not HTTPS) in that era.

If that's the worst you have against Linux then it looks like it has done very well.

Not really, no. HTTPS was used, except for log-in only. But cookies traveled unencrypted, so you had access to the account anyhow.

And not just GMail, this was virtually every service out there. From a perspective, it's laughable how bad security was back then.

> the random maintainer had no idea what they were doing

Upstream was involved in it as well. Sure it was a communication problem, but it's not like the maintainer just decided to this unilaterally. Also this WAS undefined behaviour, so from a language perspective it was already broken, even if it worked accidentally in practice.

Besides singling out a single bug from a single distro two decades ago, Linux offers maintainers who actually care to push security fixes within a few hours of globe-spanning vulnerabilities being disclosed? Not sure how that's confusing or controversial.
It's funny. When Apple goes in and changes peoples systems deeply without consent then it's Apple being evil. When they don't, it's Apple being evil. Heads I win, tails you lose.
I've seen this referred to as the "All Roads Lead to Rome" fallacy.
Let's pretend there's a night and day difference between pushing some MP3 onto all iPhones and distrusting a chain of certs that hold the value of '4'. At minimum Apple need to revoke everywhere these joke-level certs live and as fast as possible, and on all devices which also means out of support. Breaking is acceptable here as CA trust is fully compromised.
It's funny, when I do steal something I get called bad, and when I do nothing and turn the blind eye to others stealing, I get called bad too.

Morality doesn't solely depend on action or inaction, it also matter what you act or don't act to.

Eh? Changes systems without consent? Is that the ONLY way to update a fatal flaw in system? Normal update channels don't work?

Besides, how does this relate to my Linux point, where no one installs anything without you knowing it?

Because in the linux situation, you wouldn't get such an update automatically anyway so it'd be worse?
I don’t think you understand what consent means.
Yes, there would be crowds of angry people, furious that Apple... released a critical security patch? Framing removing compromised certs as "changing peoples systems deeply" seems intentionally in bad faith.
> Anyone can intercept any HTTPS connection these tools make.

yeah, it's not ideal but the scale of the threat is way overblown tbh. You have to compromise a lot to be able to make a MITM here.

Let's not treat it like PHP2Shell.

It's pretty easy to compromise things like McDonalds or Starbucks WiFi hotspots though, just setup a Pineapple with the right SSIDs and a 5G modem, funnel your target domains through your mitmproxy via the DHCP DNS server you're providing, and that's basically it.

I agree it's not PHP2Shell, but its certainly not being "way overblown" here.

Why would these public network would allow guest hosts to communicate with each other?
It’s extremely common to see other Macs pop up in the ‘network’ panel in Finder in the independent coffee shops around me. I imagine they’re just signing up for a business plan and using the routers the ISP gives them with a custom SSID and password.
Don't underplay it either though: with a compromised but unrevoked signing cert all it takes is a successful phish, or some problem to allow DNS poisoning (or hack a wireless AP, or …) to direct you to a fake site or (potentially worse as it might be less immediately noticeable and could quietly collect credentials/data for a while) send your automated tools to a malicious API.

It may not be directly exploitable on its own, which is the impression some reports give, but it is a serious hole to leave open especially with the current rate of old security issues being found, as it could be _incredibly_ useful to a malicious actor as part of an exploit chain.

Yeah, it increases the attack surface of mitm attacks to your network from the local os level.
Doesn't affect app bundles though, only *nix commands. This'd be detected pretty quickly, at least by the significant portion of patched devices.
Unless you're clever enough to only poison user agents known to be macOS and CLI. You'd probably get by.

xz-tools was only caught because it had a bug causing slow startups and someone was micro-benchmarking Postgres and decided to dive in further.

One person, at that particular time. The person and the task were also a couple standard deviations away from the norm, the world was incredibly lucky.
php2shell was overblown perhaps but this is being blown quite correctly
You can take your laptop to a busy place, make a "free wifi" hotspot that runs everything through mitmproxy and you can steal everything.
I think "everything" is an overstatement. Browsers included the compromised intermediates in their CRLs within hours of disclosure.

So this isn't like Firesheep [0]. Yes, there would be vulnerable traffic, like the git client this thread is discusing, but it's not "everything"

[0] https://en.wikipedia.org/wiki/Firesheep

If you see a developer using git on a Mac laptop in a coffeeshop, you can intercept that and gain their repo and probably password hashes pretty easily.
The near-universal use of TLS and the absurd ceremony we go through to ship root CAs everywhere is all premised on the idea that MITMs are not terribly difficult.

The entire reason one can rest easy as they connect to their coffee shop's WPA2-PSK network, is the security provided by TLS.

And if you don't think an ISP or agency would abuse this if it could be done undetectably then you haven't been paying attention these last 20 years (hence why DoH is now a thing).

If you hijack the DNS, the laptop can be pwned quite easily. Think of all the coffee shops with unprotected routers out there.
> You have to compromise a lot to be able to make a MITM here.

This is just an argument against TLS in general. Use plain http if you believe this.

Appealing to complexity is always fun. Just throwing "a lot" in there for good measure and you won't have to think about practical security implications.
The TLS trust chain is the last defense against MITM attacks. The prevalence of MITM attacks is in fact the very reason TLS exists. I'd say it's a lot worse than PHP2Shell because it doesn't only affect poorly maintained servers, but potentially ALL credentials and secrets a user sends over a compromised network with compromised CAs on their device.
The window of opportunity to MITM has passed. If you attempted an MITM from an easy vantage such as telco, ISP, or public wifi you'd be trivially detected as a significant portion of devices are patched, including on these unpatched macs as only the *nix commands are affected and not app bundles.
You might be detected but if, for example, you were smart enough not to MITM the same client twice after an error, I suspect a ton of people would ignore it and just assume the airport/coffee shop WiFi was flaky.

I’d also bet more things use libcurl behind the scenes than we appreciate, and that at least a few of them have things like silent updates which retry after a failure without notifying a user. I’ve seen a lot of agents which just try to periodically check in because they’re designed not to be noisy on lousy networks.

I can't imagine what it's like to be naive enough to believe that all small-town cafe that have a wireless access point, small ISPs all over, and mid-level engineers in big ISPs can just be blindly trusted.

If you have any real life experience and don't pretend that human psychology isn't a thing, you already know there is "a lot" that came compromised out of the box.

They just released MacOS 27.2 a few days ago? How and why did they not do a patch at least then?
Was thinking the same. My assumption would've been that this would have been a normal and routine enough of an operation to be able to push through without needing a separate RC process.
Right, so it's not just git, it's a bunch of system-provided utilities; macOS trusts bad certs, and these utils that trust the OS are affected. I use homebrew for my git and curl but haven't yet checked to confirm whether that matters here. Kind of outrageous that Apple's guidance is to edit the cert chain manually, and to expect that subsequent OS updates may revert your mitigation, reintroducing the vulnerability (as happened at least once so far).
To be honest, git deferring to the OS is the right thing to do. It's macOS that's the problem, not git.

"Ten days later, macOS and all applications that rely on it still trust the compromised eMudhra certificates" would have been a better title.

Yeah, definitely: having lots of separately distributed and likely rarely and unreliably updated trust roots would not inspire a lot of confidence.
[delayed]
I don't know about worse. Imagine a world where there is no OS authority, each app does their own. Do you think all important apps would be updated in 10 days, including auto-updates?

Apple fucked this up, but I really don't think the situation is "as bad if not worse" than if MacOS just didn't do this.

Isn't that the world that Docker gives us, basically?
Yes - difficulty in shipping security updates to a hundred different container images, each of which is vulnerable in their own slightly-different way because they all decided to ship an OS alongside their app - is one of the top criticisms of the docker way of doing things
What kind of psychopath ships 100 distinct container images? I mean, maybe if you’re talking about all of Netflix or something, but… I suspect even they have base images that they use as a starting point.

There’s very little actually running in a docker container. ca-certs may actually be an exceptional case though, since it’s not code that’s running but a kind of data.

It’s not like if you use an Ubuntu image for a node.js app that you get PHP too and have the PHP vulnerability present in your web app.

You still have to rebuild each Application Container with new CACert and deploy them. Depending on teams are working, that can be disruptive.
I would say that if rebuilding your docker containers is a disruptive process, that is a process issue.
I would agree but it doesn't change the fact that many companies do not have CD and have processes that must be followed for rollouts.

It's same problem we have when libraries get massive security vulnerabilities. Updating them is not always as easy as many would like.

> Imagine a world where there is no OS authority, each app does their own.

That's the world we live in now thanks to Google and Mozilla.

Remember this is the same company that demands they're the only ones who can possibly decide what software their users get to install.
Exceeds character limit, but admins can modify if there’s a better shorter title.
> To be honest, git deferring to the OS is the right thing to do.

Tangentially related: I fucking hate so much that browsers now all have their own DNS resolvers and caches, it's just Another Fucking Thing to debug when you're working especially on locally hosted stuff. This is not a browser's job, it's a task that should fall to the OS.

Indeed, it makes it harder to block addictive sites like this one in HOSTS :)
You can have it "both ways" if you want, but it quickly becomes very muddy - defer to the OS, but override the deferral in some cases.

Certificate management is fifty kinds of all messed up - and in THEORY everything is designed perfectly and properly, in practice it's all bandaids on bandaids that nobody cares about as long as "things mostly work".

Correct. I Googled to ask about location of certs on disk: /etc/ssl/cert.pem

Is it possible for users to remove the eMudhra certificates themselves? I have no experience with MacOS, so I don't know if certain actions are prohibited. To be clear: I am not suggesting that normies do this! Rather, I am asking if it is possible for highly skilled users.

I guess we are going to learn how important secure communication is based on the stories (or lack of stories) about how it is being exploited in the wild.
Apple gets a free pass on this website. If Google or Microsoft did the same thing, the torches and pitchforks would be out.

But instead the Apple fanboys will be out in force to tell us why actually its a good thing for Apple to leave parts of the OS insecure.

We clearly just don't know what complex level security Apple has put into their OS. Clearly their OS isn't vulnerable or else they would patch it! (/s)
It has the davantage that you can finally find out what those apps are sending back home, including your private data.
I’m somewhat of an Apple fanboy, and I find this absolutely unacceptable.
Apple makes incredible hardware and could easily be the best development machine on the market. But I ultimately prefer Linux because I own the maintenance story and it's simple + fast. System packages and xcode are horrible experiences, and there's no reason for this to be the case in 2026.
As a non mac user who's had to setup xcode and an apple dev account for the first time ever to compile and publish an app store app update for my company - the entire thing soured me on ever even attempting mobile app development.

It really surprised me what mobile devs put up with, just an utterly horrible hostile experience.

Shallow or misguided criticisms of Apple—of which there are countless—get shredded on this website. This is not a shallow or misguided criticism of Apple. It is clearly and obviously unacceptable.

Better luck next time.

> Apple gets a free pass on this website. … But instead the Apple fanboys will be out in force to tell us why actually its a good thing for Apple to leave parts of the OS insecure.

I have no idea how you've come to this conclusion, and clearly nothing of the sort is actually going on here.

Homebrew ships its own CA bundle so its own curl, Git, etc. are not affected. But the system versions of those tools will continue to be affected until Apple patches their own bundle.

(Source: I am one of Homebrew’s maintainers.)

Homebrew’s git (unlike homebrew’s curl) seems to default to using the system’s CA bundle, not homebrew’s. Is that a bug?
Edit: Sorry, I forgot that I had patched my local bundle. This might indeed be an oversight, I'm looking at it now.

I couldn't reproduce this:

  % which git
    /opt/homebrew/bin/git
  % git ls-remote https://git-emudhra.we-have-a-problem.com/check.git
    fatal: unable to access 'https://git-emudhra.we-have-a-problem.com/check.git/': SSL certificate problem: unable to get local issuer certificate
Make sure your `git` binary is Homebrew's, not the system/XCode CLT one. If you see `/usr/bin/git`, you're not using the Homebrewed Git.
> “This is being fixed now”

Wow, that was fast!

    $ git ls-remote https://git-emudhra.we-have-a-problem.com/check.git && which git
    2975b9930644b321e18ecae09e29c15b955e6103 HEAD
    2975b9930644b321e18ecae09e29c15b955e6103 refs/heads/main
    /opt/homebrew/bin/git
We watch; we wait (I guess)

    > Homebrew ships its own CA bundle
No trolling by this question: Can you explain your decision? I can see some pros and cons of this decision. Was it simply removing dependency upon Apple if there was a disagreement on which CA certs to trust? (You can act without permission from Apple to add/remove CA certs?) Also: How does Homebrew decide which CA certs to include/exclude? Do you get a CA cert pack from Oracle/Mozilla/Google/Microsoft/Amazon/Apple?

Outside of an enterprise environment, I never thought about having multiple "golden sources" of CA certs on a single host. (I have no experience working outside of classic enterprise environments, so please do not read my surprise as immediate dismissal of this idea!) For those unaware, in enterprise environments, it is very common to use a common CA cert pack from Oracle/Mozilla/Google/Microsoft/Amazon/Apple, but add 2-5 CA certs from your own company that act as trust roots.

I did some Googling on this matter. It looks like the "Big Four" of CA cert packs are: Mozilla, Microsoft, Apple, and Google. (I guess there is one or two in China also, be leave that alone for the moment.) What happens if they are in disagreement? This is a complex problem! If anyone is involved in this process, I would like to learn more.

Why are we ok with Apple or Microsoft or Mozilla deciding on a wide selection is CAs for us to trust?

I understand why they originally let each CA in, but it's definitely not one size fits all users.

A user in the US does not need to trust a Turkish CA the same way a user in Turkey does.

There needs to be an easy way for end users to personally drop trust of CAs they don't need and have that decision respected and remembered.

Have you checked with the newsgroups and gopher web on why this became a thing?
Because nobody is going to be arsed to figure this shit out, even the PGP fanatics don't bother.

So someone has to do it, and a pop-up on first boot asking you which "certificate providers" you want to trust would be as pointless as the "which browser do you want to use" ones were.

Someone has to decide on a set of default CAs to trust... It's natural that it's your OS vendor, since the trust store is an OS thing.
Honestly that last paragraph I totally agree. I did edit to “never trust” these clowns’ root CAs in Keychain Access on my Mac, and this is the first I’m hearing that there was any possibility of that being flipped back. Will have to look into that.

If I can speculate though on why these bundles come from the browser or OS and aren’t fully in the user’s responsibility. You’d have to start from scratch for it to not have a bunch of stuff the user had ‘no need for’ to be present, and the user would need to see a cert window semi-frequently that gives the possibility to start trusting a CA, and the bottom 99.99% of users in sophistication has no idea whether they should say yes or no. I mean, without being signed by a root CA you already trust, even the name on the new CA could be completely made up. “Do you want to trust the ‘Apple Root Certificate Authority’?”

Yet they ask you what keyboard type to use, what Wifi network and password to use. Is it too much to ask, do you want to trust "key authorities" from outside your country? Users used to see computers as inherently technical devices and expect to get asked technical questions.

But even if not, you get asked whether to turn on Siri, whether to send telemetry, whether to sign away your rights, etc., all of which are massaged in a way where it's nice and easy for the user to answer a certain way without having a clue. Makes me think user comprehension isn't even a problem.

>do you want to trust "key authorities" from outside your country?

So what happens to americans visiting french sites?

"It seems like you're visiting a site registered in XYZ country. Do you want to start trusting XYZ country key authorities?

[Do Not Trust], [Trust Once], [Trust Forever]"

> There needs to be an easy way for end users to personally drop trust of CAs they don't need and have that decision respected and remembered.

There is. On macOS go to Keychain Access > System Roots and disable the kind of trust you want to disable. There's probably a scriptable method for that.

On Windows, open the management console to "Certificates". Then go through the list and move/remove/distrust any certificates you dislike. There are also powershell tools you can use.

On Linux, the situation is a bit annoying. You can remove all CA certificates manually and make the directory read-only to break any attempt from your distro to interfere with the list. You need to check your distro documentation to see if they use /etc/ca-certificates or /etc/ssl or something like /etc/pki for reference. Some distros offer a partial GUI as well.

Programs that deem themselves smarter than the OS (true in the case of macOS and various Linux distros), like browsers, the JVM or NodeJS applications, can have their own cert stores.

You will need to acknowledge half a dozen certificate prompts for every website you visit after this, though.

There are three futures, each being dystopia than current:

- Only ISRG(Let's encrypt) being the sole CA and benevolent dictator of the internet. (then forced to be dissolved due to antitrust?)

- Each country controls their own CA and you need to purchase 200+ certs in order to make your website generally accessible. (then country XXX will charge you $200/year)

- Big techs owning OS/Browsers control their own CA and cross license each other (then Apple still being lazy handling incidents) (then they form a cartel and block others, like current Tier 1 ISP, email senders, credit card networks)

edit: put enter between numbered items.

(comment deleted)
I asked codex “find out if this Mac is still trusting that bad cert and remediate if so”. I got a few commands out of it to run myself with sudo, and now I’m good.

Thanks for being on the ball, Apple.

Rather than prompting the mechanical parrot, you can also refer to Apple's own website: https://support.apple.com/en-us/149555

I don't know why they haven't rolled out an update if they've already wrote a whole article about patching the problem yourself.

Second probe: confirming the fixed comment path resolves a parent comment correctly.
(comment deleted)
Can we get a title edit? OP’s title implies this is a Git bug, but the bug applies to half of macOS, including curl, openssl, python, perl, and ruby, and to only the macOS version of Git. The original title is less misleading, but personally I’d have titled it something like “macOS still trusts the compromised eMudhra certificates in git, curl, openssl, python3, perl, ruby, more”.
There must be a long list of things that aren’t fully patched for this yet, on laptops, on servers. Think of packages that ship their own trust anchors, isn’t there a rust crate that compiles with some root certs embedded?

On one hand there’s unmaintained packages that won’t update dependencies. On the other hand, fixed dependencies did not necessarily backport to all possible versions in the wild, making update as a fix unviable for those.

Doesn’t macOS have a way to generate cert.pem from the keychain?
I'm not that familiar with the MacOSx system, but do they have a "minor" update path that I'm not aware of? It seems that everything is an 8gb update. Is that why this update isn't getting put out?
Yeah they do, it's called "Background Security Improvements" but they rarely use it lol
Now would be the time :)
(comment deleted)
If you're not already in the habit of contacting your git remotes over ssh, its probably one worth building. This is gonna keep happening.
SSH in typical use is not vulnerable to a compromised root certificate, instead it's "just" vulnerable to you having no idea whether you can trust that host key or not, so you just blindly say yes and hope for the best. At least if the key changes, you'll know something was wrong; maybe it was this connection that got compromised, maybe it was the first connection that got compromised, maybe the legitimate host just rotated their keys, who knows. SSH security against MITM is mostly based on lack of motivated attackers and wishful thinking.
Its no more wishful than expecting a root of trust to stay trustworthy or expecting a trusted name to consistently resolve to the same party.

And there's a lack of motivated attackers for a reason, the attack scales poorly. Maybe you can bamboozle a target or two, but if you go for hundreds, somebody's gonna start wondering what's up with the keys. If you're trying to take over the world, or sell to somebody who is, you're better off focusing on compromising a root of trust or tampering with name resolution, because attacks like that are sustainable at scale.

Yes, it is about the same, that's the point. If you are worried about this or future compromises of TLS/PKI, using SSH naively is not any better.
Maybe it's the same probability of being a problem, but it would be a problem introduced by different kinds of threat.

If you're worried that you'll be targeted as an individual by somebody who cannot coerce a CA but can tamper with the channel you're using to get the public key, use https.

If you're worried that you'll be targeted along with many others by somebody who can coerce a CA, use ssh and ensure that some folks are comparing key fingerprints such that at least one is close to the source.

I have been wondering recently if Apple are bothering to maintain the Unix side of the OS. Seems like they occasionally remove or deprecate stuff but don’t put any serious effort into maintenance.

This is just another one of those smells.

It was arguably always halfhearted at best. Look at how behind they were on many things for years. They update things, I assume, when necessary to make the build toolchain work, and possibly to hold onto their POSIX compliance status which some us government buying contracts demand. The fact they changed the default shell to zsh instead of that decade(s)-old version of bash they used to have (avoiding a GPL license change, I think?) was a miracle.
It’s interesting they call out as “optional” removing sectigo roots.

If you’re not removing them you better be sure anything that runs on the system has full revocation support including intermediates. Which is.. kind of an impossible thing to assert as true. Which is why most trust stores just removed that as well last week.

Yes, it’s insane. Step 2 does next to nothing for security without step 3.
[delayed]
Yeah, why may that be hard? That’s the question

I don’t think it’s gross incompetence, since they documented the steps to remove. They may have some unexpected dependency on the file? Looks mysterious from the outside.

People would have to use it in some drastic catastrophic way for them to force it out in an update, which they might push it out in the next macOS update, which don't come nearly as frequently as Windows bloatware updates do.
This seems important enough to just send out a patch update that removes the cert and does nothing else. But apparently at least a couple people at Apple disagree; or, something I'm not familiar with makes that type of update impossible.
I mean, I agree with you, I'm just basing my thoughts of what I've seen managers at different companies prioritize as "important" even when engineers cry that a security issue should be of higher priority...
You have to admit that “just a quick patch” isn’t automatically low-risk. You could cause a large blast radius if you rush any patch out without proper testing.
I agree 100%, but I keep going back to the blast radius of a compromised CA and my gut instinct is still "get rid of it and push a hotfix" but I completely acknowledge it's a lot more complicated to ship a MacOS update than just branch off of master and push a new commit, and my experience is almost entirely the latter.
amusing you criticize windows updates here where the macos update schedule is woefully inadequate.
One is too delayed. The other is frequent but full of crap.
I would put Apple updates in that bloatware category given I was forced to upgrade my phone after iOS and updates eventually took up 50% of storage.
I think the main problem is that the weak eMudhra intermediates were cross-signed by Sectigo. Certificate revokations are needed to handle this correctly.

Dropping the Sectigo root along with the eMudhra root is the easy and safe solution, and probably this is what Apple should have done (it’s the workaround they recommend!). But it means that some small percentage of certificates that are fine won’t be accepted.

> Is /etc/certs not used by most graphical Mac software, I guess?

Correct, those usually use Apple's networking APIs which all go through their security stack, which already picked up the OCSP revocation within about 2 hours in most cases.

this only affects terminal apps, which the vast majority of MacOS users don't run; the fix for graphical Mac software is already patched

my guess is their thinking is that if you're running those apps on Mac, you're a developer or know what you're doing, and can edit /etc/ssl/cert.pem until they push an update

they also have to consider that such users may have custom changes to their cert.pem file, and they probably don't want to rush an update that would potentially fu-bar those changes

Certainly you're aware that every certificate sits in a separate file and an update can just remove the interested files?
It doesn’t look like that, it looks like it’s a cert.pem file.
I would be aware of that if it were true, but (surprise!) it's not

not convinced? take a look at your /etc/ssl/cert.pem

The thing that makes this especially odd is that Apple apparently has a fast-track way of shipping security patches for the OSs that don't require the user to go through the lengthier OS upgrade process. But for some reason, they haven't used it here.
That fast-track method was used, but it only applies to "graphical" software which really means, software that uses Apple's macOS specific APIs and frameworks, not the lower level UNIX stuff like /etc/ssl

I suspect the fast-track background security updates can only update the databases and revocation lists that live in the proprietary APIs, not any arbitrary file on the system

I see other people making this statement, but it sounds like a really really weird excuse.
It’s not an excuse but an explanation: there are separate stacks owned by different teams and they haven’t been forced to collaborate. If you use an internal CA you’ve already had to deal with the easier additive version of this problem because that fails in an obvious, less harmful manner than quietly accepting something which needed to fail.
It doesn't explain why Apple who 1) sells itself on security and 2) ease of use doesn't 3) update what it installed and protect its users. Why are they not doing those things?
Agreed. Why does macOS ship with two copies of the trust store? Shouldn't the process to update the primary store automatically derive the other?
Apple has one job here and it isn't to make UIs that look like melted plexiglass. I am fucking pissed.
Yes because that thing is completely useless and provides basically no value
Following the eMudhra incident I made a tool to distrust all except the 30 most essential CAs on my Linux and Windows systems (https://github.com/hyperreality/distrust). I was shocked at how messy the situation is on Linux, with browsers, CLI tools, programming languages all having their own stores and distros building the bundles differently (update-ca-certificates in Debian vs update-ca-trust in Arch or Fedora). Or like Firefox not using the certs in /usr/share/ca-certificates/mozilla/ but its own per-profile NSS store instead. There has to be a better way.
The better way then comes with the drawback where one vendor has all the power.

Then someone would ask "there has to be a better way" again, when the reason for the new inconvenience gets forgotten, and it oscillates.

Browsers were the original TLS trust store vendors. Operating systems started shipping trust stores because their OS came with a browser, not because they were the natural root of trust. Browsers also have tools to update revocation near-live, while operating systems lag behind by hours, or days in this case.

For your tool, remember to nuke Windows' automated certificate authority updates (separate from normal Windows updates) and possibly the automated intermediary cache. Microsoft publishes a list of trusted CAs/intermediaries periodically and they get automatically applied to Windows so websites don't need to send the full cert chain during TLS negotiation, and it has a tendency to cache trusted intermediaries as well.

Mozilla's trust store is one of the first to get updated when a revocation happens, as well as Google's CRLsets. Downloading full CRLs repeatedly would be a waste of time and bandwidth, that's why there's a specific distribution method for CA updates. I would extend your tool with a daemon that checks for revocation automatically, so you don't accidentally eMudhra yourself while browsers and the OS get updates next time something like this happens.

I was also surprised to find Java’s key tool returning successfully for the delete request without actually doing anything: list, delete, and list again would still show the CA. Debian had this right during the 90s, all they had to do was follow the system convention.

This feels like all of the C code getting attacked now where everyone spent the last three decades building up cruft because fixing it would be a chore, and now the separate bills are coming due all at the same time.

Apple's official API had revoked the certificate within hours. Only the Unix compatibility layer they sort-of maintain has this problem. This is hardly the first or only outdated Unix compat tool they ship with their OS.

Not great for unaware developers, but most macOS users are fine. Those who are aware can follow the steps in https://support.apple.com/en-us/149555 until Apple bothers to roll out an update.

What would cause this delay from apple? Is there something behind the scenes that can explain it?
Albeit helpful, the site is kinda cringe. It leaves a bad taste in my mouth knowing that the author is affiliated with Mozilla.

Of course they would throw shades at Apple. That being said the situation isn't ideal and I'm sure Apple would have shipped an update if it were easy and warranted.

Your apps and browser don't trust them. Why should your CLI. It's easy and warranted for Apple and look how much cash and engineers they have.

Apple should publish a hotfix now. Work on a better solution in the long-term. They just don't care about security for developers is my take (it's like a niche group for them?)

Mozilla or not does not matter here.

A concern for me on this site is that they give the shell commands to edit the /etc/ssl file on their site, instead of just making the Apple link (which they also list) as authoritative on the matter. I don't think users should get in the habit of cut/pasting sudo enable shell one-liners from "random" websites, especially and specifically when the vendor has the very same commands in their published fixed for the issue. In short, this site should have just said "go get the fix from apple".
Delete /etc/ssl/cert.pem and replace it with the one from MacPorts.

    # /usr/bin/curl https://dv-g3b-aaa-noct-emudhra.we-have-a-problem.com/
    Your client successfully connected to the site. That's bad!
    
    # sudo mv /etc/ssl/cert.pem /tmp/
    
    # sudo ln -s /opt/local/libexec/openssl3/etc/openssl/cert.pem /etc/ssl/cert.pem
    
    # /usr/bin/curl https://dv-g3b-aaa-noct-emudhra.we-have-a-problem.com/
    curl: (60) SSL certificate problem: unable to get local issuer certificate
    More details here: https://curl.se/docs/sslcerts.html
Microsoft had a small golden age with Azure and Windows 10, and now they are going right back into a Vista-era-like rot. Github is down an awful lot. The reliability is worse than the product I work on for work by like 2 orders of magnitude and we have no ops team. It's a joke.

Windows 11 is awful, lots of the fixes and modernization started in windows 10 has totally stalled, leaving users with 5 different places to change a setting instead of the one old place or one new place. Look how many different UI's there are to adjust sound card / sound settings or display settings. It's a complete mess. Their attempts to jump into the AI game have been a complete joke, despite already having integrated Cortana years ago. Putting Cortana on an llm is such an obvious thing to do, and they already did so much work to make that fully integrated into windows and force it on everyone even though it was trash, and now it's still trash and they don't seem to be able to adapt at all.

Now they can't respond to a complete insecurity in one of their major products in weeks? Why do they even validate certs? Why doesn't revocation work in their app? I don't think they can really be trusted at this point.

Microsoft were the good guys in this case.
A flat cert.pem file (or similar) can't say "distrust this intermediate".

macOS already has the verifier needed (Security.framework/trustd path that protected Safari). It would be good if MacOS routed CLIs through it.

> A flat cert.pem file (or similar) can't say "distrust this intermediate".

Why does that matter in this case? Can't this be fully mitigated with next to no collateral damage just by removing the trusted roots that signed them?

The root certs may have signed other intermediaries that are still trusted.
I trust nothing that came from emsign
They aren’t trusted, though. Didn’t Chrome ultimately choose to drop the roots, not just the intermediates, from their root program?
Chrome dropped one of the eMudhra roots. The problem is that the weak eMudhra intermediates were cross-signed by Sectigo. Browsers (at least Chrome) didn’t drop the Sectigo root, instead they store a revokation for the cross-signature.
Wasn't the cross signing root AAA Certificate Services, which modern browsers had already dropped trust of before this incident?
Ah, you’re right. Though unless I’m mistaken again, Safari (but not Chrome and Firefox) still trusts AAA Certificate Services and handles this issue with revokations.
This seems like extremely acceptable collateral damage to me. Sectigo should already be working with their customers to reissue immediately since they are the ones who signed the dumb intermediates. Trust only matters if there’s accountability. If root CAs want to keep their trust, it’s on them to not sign things that are bad. That’s the responsibility that comes with the license to print money that being a root CA gives you.

If I’m an OS vendor I would have been putting them on notice days ago that their root was being dropped and they have X days to reissue certs and deliver a new root cert or otherwise they can just accept that they’re not qualified to be a root anymore.