Lapcatsoftware: "Apple is to blame for revoking this certificate"
How is Apple supposed to know that the certificate has not been compromised? When I ask my CA to revoke a certificate I don't have to explain why to them, they just do it. They are my consequences to bear.
Someone has to give you the revocation list, and consumers generally aren’t smart enough to know even what a revocation list is. On Windows, it’s Microsoft. On macOS, it’s Apple.
Apple's revocation policy gives you no control over the revocation–you ask them to revoke things for you by contacting them, and then they decide whether this seems like a valid request, then they grant it for you. This is not how revocation usually works, so if so, why is Apple not vetting requests when they are placing themselves into the process?
That's the same way a CA works (except with a bit of automation instead of emailing someone).
I go to the CA that issued my cert and say "I'd like to revoke this" (whether that is an API call or an email), they then publish the cert into their certificate revocation list.
Now, Apple could deny a request, but they are under no obligation to do so, and I would argue that they shouldn't be making those judgement calls.
As they did here, they get the email, they then publish the CRL, and wash their hands off it.
But that is exactly what they claim they are there to do. They should either just accept any request, or do a good job at denying incorrect revocations.
Where do they claim to do that? Maybe there is no API for it because it happens so rarely that letting the Apple Security team push a few buttons is more cost effective.
Your quote of Lapcatsoftware is just wrong. It is not:
> Apple is to blame for revoking this certificate.
It is:
> So blame must be apportioned to both companies.
...because the revocation itself seems to have been an unjustified request.
It seems the value-add of Apple having the sole ability to revoke certificates would be that they could avoid something like this. (That said, I do think this is still mostly HP's fault, but the article raises a point that is more valid than implied.)
What should be the proper process then? Should Apple reply to all revocation requests asking for a proof that the certificate has really been compromised? OCSP or CRL revocations are easy to revert, so why risk letting a compromised certificate to be used for an extended amount of time?
Beats me. Apple could just give developers full discretion, which would lower the amount of time it takes to get a really compromised certificate even more. Presumably, there is some good reason to have manual discretion, otherwise, why have it?
I agree that HP was careless here but IMO Apple should learn here that developers who ask for a cert revocation may not understand the consequences or the cases where this is necessary.
I have no idea what the cert revocation workflow looks like, but providing cases where it should and shouldn't be used would be useful.
It's still certainly possible that this falls squarely on HP and there's only so much you can do when someone is determined to blow their foot off.
Replace Apple with any CA on the world wide web, if I go ask them to revoke a certificate, they should just do as I ask. Not try and second guess me.
Apple is not to blame for HP requesting the certificate be revoked, not even apportioned. HP is the sole requestor and the sole entity responsible for the cert being revoked.
Other CAs allow you to revoke certificates. For example, it can be done via the API with ACME. It is unclear why Apple doesn't delegate this ability entirely to developer discretion, but presumably this is the reasoning.
This is what everyone misunderstands. The Developer ID code signing program is not analogous to web certs. The usage of these certs is entirely different.
Apps installed on a user's Mac are expected to run indefinitely, essentially forever. Developer ID signing certs do expire, but expiration only affects their ability to sign new app versions, it doesn't prevent old versions signed with the cert from running. The expired cert is still considered valid when you launch the app on a Mac.
When you connect to foo.com on the web, the server can use any valid cert. Certs get replaced quite frequently, often yearly. If you put a new cert on the server and revoke the old one, it doesn't necessarily harm web users, because they can make new connections with the new cert. But if a Developer ID cert is revoked, then macOS immediately (on OCSP check) prevents any app ever signed with that cert from launching, and calls it "malware". (The main purpose of Developer ID is to stop the spread of malware.) Even if the app has been installed on the user's machine for a long time, the app will no longer run. Revoking a Developer ID cert is a remote kill switch for Mac apps. This is why revocation of Developer ID certs is so destructive, and is almost always reserved only for malware.
The analogy with web CAs simply does not hold here.
Until this HP case, I've never heard of a 3rd party developer requesting that a Developer ID cert be revoked. All cases I've heard of are when Apple revokes the cert of a malware writer, which obviously the malware writer did not request.
A remote kill switch for Mac apps is extremely destructive, which is why only Apple has the power to revoke the Developer ID certs, which they should only do for a very good reason, such as malware.
It's also important to note that Developer ID is a requirement that Apple imposed on third party developers, purportedly for security purposes. We developers never wanted this, but we had no choice. Again, none of this is really under our control, it's Apple's program, and Apple is the sole CA for Mac app certs.
> Replace Apple with any CA on the world wide web
This is what everyone misunderstands. The Developer ID code signing program is not analogous to web certs. The usage of these certs is entirely different.
The brilliant thing about pki is that certificate usage is based on its cryptographic purpose. It doesn’t care about the application itself.
It’s actually a good thing that Apple doesn’t just verify the certs on install. The kill switch serves a vital security purpose.
>This is what everyone misunderstands. The Developer ID code signing program is not analogous to web certs. The usage of these certs is entirely different.
If a web cert gets revoked a browser won't accept it when making a connection to a domain/server anymore even if they previously did (unless you make it). If an app signing cert gets revoked the OS won't run a binary signed with it anymore even if it previously did (unless you make it). Sounds similar enough to me, minus expiry handling.
Revocation is indeed extremely destructive and should only used by Apple on their own accord against malware but I don't see why a dev themselves shouldn't be able to revoke their own cert without having to go back and forth with the Apple security/support team to have to try to convince them that the issue is severe enough.
The main and basically almost only reason where I see developer side revocation be used is if their keys or account have been (possibly) compromised and therefore there's a chance of it being used for malware. Time can be of big importance in a situation like that too. As a user I absolutely want Apple to revoke certs when requested by the dev themselves without triple verifying if they are really totally actually absolutely sure. I consider the upsides of that to outway the downsides.
While a binary from a dev that has been installed for a long time will be less likely to be malware but not enough to be exempt from that.
Users that dislike OSCP can disable it fully on their side if they want or locally sign anything specific they do trust. Devs didn't ask for it but having developer IDs and application singing seems like a pretty sensible thing to me in an end user OS used by the average consumer.
> If a web cert gets revoked a browser won't accept it when making a connection to a domain/server anymore even if they previously did
The difference is that you can just get a new web cert, install it on the server, and you're good to go for https. On the other hand, getting a new Developer ID cert doesn't help at all to make the app start running again. You can sign new versions of the app with the new cert, but the installed versions of the app signed with the revoked cert are still dead as a doornail, which is no good at all for your existing user base.
> The main and basically almost only reason where I see developer side revocation be used is if their keys or account have been (possibly) compromised and therefore there's a chance of it being used for malware.
It's not clear that this ever happens. I discussed the case of Panic in the blog post. Their private key was possibly compromised, but Apple did not revoke their cert! The old cert is still valid, and so are the old versions of their apps. Apple apparently has a more limited way of disabling apps, based on the secure timestamp of the code signature.
The check consists of an HTTP GET request (port 80, unencrypted!) to ocsp.apple.com
Am I missing something? Is this still the case?
Seems like an obvious privacy issue, to be able to identify if an user is launching apps from a certain developer just by doing very basic packet capture.
Yes, OCSP is unencrypted (and uses HTTP). OCSP requests only include the certificate's serial number, however, and (AFAIK) Apple does not publish a directory of all issued certificates.
WRT the web PKI, though, yes, the serial number can be looked up in CT logs. Although that same "basic packet capture" will give you the hostname of the server -- at least until encrypted SNI is used everywhere.
(As for why OCSP is unencrypted, chicken and egg... rsponses are signed, however.)
Apple doesn't need to publish it. All one needs to do is collect a bunch of OSX apps (app store, creative cloud, web downloads of .dmg, whatever) and pull out the certificate serial numbers. This mapping is effectively public data, if you bother to go about collecting it.
Then, anyone monitoring your connection knows which apps you launch, when, and where you do it. Additionally, Akamai (and their ISPs) get your complete (IP geolocation) tracklog, and knows when you're at home, when you're traveling, and which apps you use in those places.
I wrote about the consequences of this just an hour ago:
Interesting thing is, Apple will deny apps from the App Store that make unencrypted connections (via App Transport Security), but their own, firewall-exempt, vpn-bypassing system processes can do it just fine. :D
If you're referring to OCSP requests / responses being unencrypted, I would be interested in hearing what your solution to this problem is.
Also, I would recommend that you do not perform any packet captures (as mentioned earlier) to observe your OCSP traffic. You're gonna have a heart attack when you realize that this isn't just an Apple thing and that nobody encrypts OCSP.
Once you understand why, I'd be interested in hearing your new, revised solution.
Finally, it's not quite as bad as sneak makes it out to be. OCSP responses are cached so, as mentioned in TFA (so certificate status is not being checked every time you launch every app), although I'm not sure what the validity interval of OCSP responses from ocsp.apple.com is (and don't have anything running macOS to check). Additionally, I don't see anyone but Apple announcing any part of the 17/8 network (regardless of ocsp.apple.com. being CNAME'd to Akamai).
What the heck. You don't have to prove private key compromise for a CA to revoke a cert, you just ask them to revoke it. How in the world is this apple's fault?
As long as they authenticate the person requesting the revocation. In most CA's, this can be done by signing the request with the private key to prove control.
Now we are told that blame "MUST" be apportioned? "HP and Apple failed in their responsibility"
For all we know HP could have found a remote zero day exploit in their drivers, and was going to initiate a replacement effort or something.
The author also complains no CRL is available, but that's a good thing in these cases, if there is a mistake certificate status can be updated. So again, apple has an approach here that means it's not too hard to get folks printing again.
The author makes a case that a publisher can only revoke in the case of malware and key compromise. I can think of PLENTY of situations where revocation might make sense (and publisher may be still sued I'm sure) outside of that, for small teams and even for big teams.
This entire thing is likely related to Apple’s engineered obsolescence cycle. They have already been busted intentionally causing issues on their platforms prior to the release of new products in order to drive sales. It’s a grey area legally and difficult to prove, but it’s essentially a form of racketeering.
It seems a lot of your comments garner downvotes on this site. Your comments are sound, or at least very reasonable, and thus my opinion of this site and the community has faltered significantly.
A certificate revocation request signed by the private key is proof of private key compromise.
Anyone issuing such a revocation due to knowledge of compromise of the key is notifying the issuing authority of the compromise.
If a key is used for a purpose not listed as allowed in the certificate, it's compromised, even if the entity that misused the key is the original holder.
Anyone issuing such a revocation without authorization has the private key, and is using it for something they're not authorized to use it for. It's compromised by definition.
There's never a reason not to revoke a certificate when a valid revocation request is received. The key has been compromised, either by leaking or by misuse.
> A certificate revocation request signed by the private key is proof of private key compromise.
There's no such thing in this case. In the blog post, I quoted from Apple's Certification Practice Statement: "The Subscriber may initiate a revocation request by sending an email to productsecurity@apple.com. The request for revocation will then be evaluated by Apple."
Everyone wants to make an analogy between Developer ID code signing certificates and web server certificates, but the usage of the two kinds of certs is completely different.
windows and web browsers do this as well via OSCP.
Linux distros also do this via package manager traffic and update checks. Apt/rpm traffic is unencrypted, relying on a separate payload signature scheme for authenticity (same as OSCP).
34 comments
[ 6.6 ms ] story [ 69.3 ms ] threadApple: "Okay done"
Lapcatsoftware: "Apple is to blame for revoking this certificate"
How is Apple supposed to know that the certificate has not been compromised? When I ask my CA to revoke a certificate I don't have to explain why to them, they just do it. They are my consequences to bear.
I go to the CA that issued my cert and say "I'd like to revoke this" (whether that is an API call or an email), they then publish the cert into their certificate revocation list.
Now, Apple could deny a request, but they are under no obligation to do so, and I would argue that they shouldn't be making those judgement calls.
As they did here, they get the email, they then publish the CRL, and wash their hands off it.
> Apple is to blame for revoking this certificate.
It is:
> So blame must be apportioned to both companies.
...because the revocation itself seems to have been an unjustified request.
It seems the value-add of Apple having the sole ability to revoke certificates would be that they could avoid something like this. (That said, I do think this is still mostly HP's fault, but the article raises a point that is more valid than implied.)
I have no idea what the cert revocation workflow looks like, but providing cases where it should and shouldn't be used would be useful.
It's still certainly possible that this falls squarely on HP and there's only so much you can do when someone is determined to blow their foot off.
Apple is not to blame for HP requesting the certificate be revoked, not even apportioned. HP is the sole requestor and the sole entity responsible for the cert being revoked.
This is what everyone misunderstands. The Developer ID code signing program is not analogous to web certs. The usage of these certs is entirely different.
Apps installed on a user's Mac are expected to run indefinitely, essentially forever. Developer ID signing certs do expire, but expiration only affects their ability to sign new app versions, it doesn't prevent old versions signed with the cert from running. The expired cert is still considered valid when you launch the app on a Mac.
When you connect to foo.com on the web, the server can use any valid cert. Certs get replaced quite frequently, often yearly. If you put a new cert on the server and revoke the old one, it doesn't necessarily harm web users, because they can make new connections with the new cert. But if a Developer ID cert is revoked, then macOS immediately (on OCSP check) prevents any app ever signed with that cert from launching, and calls it "malware". (The main purpose of Developer ID is to stop the spread of malware.) Even if the app has been installed on the user's machine for a long time, the app will no longer run. Revoking a Developer ID cert is a remote kill switch for Mac apps. This is why revocation of Developer ID certs is so destructive, and is almost always reserved only for malware.
The analogy with web CAs simply does not hold here.
Until this HP case, I've never heard of a 3rd party developer requesting that a Developer ID cert be revoked. All cases I've heard of are when Apple revokes the cert of a malware writer, which obviously the malware writer did not request.
A remote kill switch for Mac apps is extremely destructive, which is why only Apple has the power to revoke the Developer ID certs, which they should only do for a very good reason, such as malware.
It's also important to note that Developer ID is a requirement that Apple imposed on third party developers, purportedly for security purposes. We developers never wanted this, but we had no choice. Again, none of this is really under our control, it's Apple's program, and Apple is the sole CA for Mac app certs.
The brilliant thing about pki is that certificate usage is based on its cryptographic purpose. It doesn’t care about the application itself.
It’s actually a good thing that Apple doesn’t just verify the certs on install. The kill switch serves a vital security purpose.
But if a malware is signed and installed to a computer, revoke it can prevent further running.
If a web cert gets revoked a browser won't accept it when making a connection to a domain/server anymore even if they previously did (unless you make it). If an app signing cert gets revoked the OS won't run a binary signed with it anymore even if it previously did (unless you make it). Sounds similar enough to me, minus expiry handling.
Revocation is indeed extremely destructive and should only used by Apple on their own accord against malware but I don't see why a dev themselves shouldn't be able to revoke their own cert without having to go back and forth with the Apple security/support team to have to try to convince them that the issue is severe enough.
The main and basically almost only reason where I see developer side revocation be used is if their keys or account have been (possibly) compromised and therefore there's a chance of it being used for malware. Time can be of big importance in a situation like that too. As a user I absolutely want Apple to revoke certs when requested by the dev themselves without triple verifying if they are really totally actually absolutely sure. I consider the upsides of that to outway the downsides.
While a binary from a dev that has been installed for a long time will be less likely to be malware but not enough to be exempt from that.
Users that dislike OSCP can disable it fully on their side if they want or locally sign anything specific they do trust. Devs didn't ask for it but having developer IDs and application singing seems like a pretty sensible thing to me in an end user OS used by the average consumer.
The difference is that you can just get a new web cert, install it on the server, and you're good to go for https. On the other hand, getting a new Developer ID cert doesn't help at all to make the app start running again. You can sign new versions of the app with the new cert, but the installed versions of the app signed with the revoked cert are still dead as a doornail, which is no good at all for your existing user base.
> The main and basically almost only reason where I see developer side revocation be used is if their keys or account have been (possibly) compromised and therefore there's a chance of it being used for malware.
It's not clear that this ever happens. I discussed the case of Panic in the blog post. Their private key was possibly compromised, but Apple did not revoke their cert! The old cert is still valid, and so are the old versions of their apps. Apple apparently has a more limited way of disabling apps, based on the secure timestamp of the code signature.
Seems like an obvious privacy issue, to be able to identify if an user is launching apps from a certain developer just by doing very basic packet capture.
WRT the web PKI, though, yes, the serial number can be looked up in CT logs. Although that same "basic packet capture" will give you the hostname of the server -- at least until encrypted SNI is used everywhere.
(As for why OCSP is unencrypted, chicken and egg... rsponses are signed, however.)
Then, anyone monitoring your connection knows which apps you launch, when, and where you do it. Additionally, Akamai (and their ISPs) get your complete (IP geolocation) tracklog, and knows when you're at home, when you're traveling, and which apps you use in those places.
I wrote about the consequences of this just an hour ago:
https://news.ycombinator.com/item?id=25078034
Interesting thing is, Apple will deny apps from the App Store that make unencrypted connections (via App Transport Security), but their own, firewall-exempt, vpn-bypassing system processes can do it just fine. :D
If you're referring to OCSP requests / responses being unencrypted, I would be interested in hearing what your solution to this problem is.
Also, I would recommend that you do not perform any packet captures (as mentioned earlier) to observe your OCSP traffic. You're gonna have a heart attack when you realize that this isn't just an Apple thing and that nobody encrypts OCSP.
Once you understand why, I'd be interested in hearing your new, revised solution.
Finally, it's not quite as bad as sneak makes it out to be. OCSP responses are cached so, as mentioned in TFA (so certificate status is not being checked every time you launch every app), although I'm not sure what the validity interval of OCSP responses from ocsp.apple.com is (and don't have anything running macOS to check). Additionally, I don't see anyone but Apple announcing any part of the 17/8 network (regardless of ocsp.apple.com. being CNAME'd to Akamai).
As long as they authenticate the person requesting the revocation. In most CA's, this can be done by signing the request with the private key to prove control.
Now we are told that blame "MUST" be apportioned? "HP and Apple failed in their responsibility"
For all we know HP could have found a remote zero day exploit in their drivers, and was going to initiate a replacement effort or something.
The author also complains no CRL is available, but that's a good thing in these cases, if there is a mistake certificate status can be updated. So again, apple has an approach here that means it's not too hard to get folks printing again.
The author makes a case that a publisher can only revoke in the case of malware and key compromise. I can think of PLENTY of situations where revocation might make sense (and publisher may be still sued I'm sure) outside of that, for small teams and even for big teams.
Anyone issuing such a revocation due to knowledge of compromise of the key is notifying the issuing authority of the compromise.
If a key is used for a purpose not listed as allowed in the certificate, it's compromised, even if the entity that misused the key is the original holder.
Anyone issuing such a revocation without authorization has the private key, and is using it for something they're not authorized to use it for. It's compromised by definition.
There's never a reason not to revoke a certificate when a valid revocation request is received. The key has been compromised, either by leaking or by misuse.
There's no such thing in this case. In the blog post, I quoted from Apple's Certification Practice Statement: "The Subscriber may initiate a revocation request by sending an email to productsecurity@apple.com. The request for revocation will then be evaluated by Apple."
Everyone wants to make an analogy between Developer ID code signing certificates and web server certificates, but the usage of the two kinds of certs is completely different.
Interesting. This means OSX leaks which apps you run unencrypted on the network.
Linux distros also do this via package manager traffic and update checks. Apt/rpm traffic is unencrypted, relying on a separate payload signature scheme for authenticity (same as OSCP).