Yup. We choose the "footgun" reason. Despite automation, despite best practices, the cost/benefit analysis of losing all traffic for months (more realistically, losing the use of a domain) because of a single mistake shows that the benefits of preventing against MITM is simply not worth it.
Were we bigger, the benefit might outweigh the costs. Of course if we were bigger, the cost would go up significantly as well. Can you imagine a world where Amazon had to completely re-build its order taking infrastructure around a new domain name due to a mistaken configuration file?
I QR'd my two backup private keys and corresponding CSRs, printed them out, and gave one set each to my two partners to keep safe. I agree that it seems a little overkill, but it doesn't have to be terribly complicated.
So, curious, what happens in the case that my private key is compromised? An attacker can then change my key pinning and prevent users from accessing my website? Heck, can an attacker already do this to many users simply by adding some headers and using their own SSL key? Seems a scary proposition, someone could effectively kidnap control of your domain and you'd have no way to get it back. Especially given the opposition to OCSP by certain parties and the fact that smaller sites have problems even getting on their alternative lists, it could be part of impossible to get your attacker's certificate removed from many of your users browsers.
That's an interesting and scary attack I hadn't thought about before. HPKP headers should become part of security monitoring strategies to reduce the impact of such an attack, independent of whether a site actually uses HPKP or not.
I would be interested in hearing someone smarter and more knowledgeable address this, but I will say that the public keys are sticky ("pinned"), so if an attacker was able to change it, it would only affect virgin clients during the time the false header was present. Returning visitors would (presumably) not be able to load the site because the public keys they have cached for your site wouldn't match the false SSL certificate. Unless the attacker was playing a sneaky long game and only changed the backup public keys...
If the private key is compromised or the HPKP header is changed through some other vulnerability in your site, the attacker could still satisfy the existing pin header. The attack for clients with cached pins would be a bit more involved and would have to be done in two steps¹:
First visit: Add a new backup hash for a certificate that the attacker
controls.
Second visit: Use the new backup certificate for the TLS session
and set a new HPKP header that excludes the keys the site owner
actually uses.
¹ This is assuming browsers won't accept HPKP headers that don't pass for the current connection. I haven't actually checked this.
o The TLS connection was authenticated with a certificate chain
containing at least one of the SPKI structures indicated by at
least one of the given SPKI Fingerprints (see Section 2.6).
I'm not sure if it is so, but I would hope browsers attach an expiry date to each hash, not each hash set, and that hashes can't be revoked simply by later omission. i.e. if you have keys A and B, and an attacker takes your domain and sets a header solely for the hashes of keys B and C, I would hope that you can still use a cert built using key A until the expiry.
"Shun" isn't the right word to use. The tools for managing SSL keys are abysmal, the documentation around HPKP so far is confusing (this article does a decent job of explaining how to use it), and the impact of any mistake is that your site might become completely inaccessible to your customers. The best example this article could give (github) uses HPKP in the most conservative possible way. That should tell you something.
As falcolas says, security is about risk tradeoffs. Right now the risk of using HPKP is rightly judged by most webmasters to be far greater than the risk of not using it. Key pinning Is a great security tool in many scenarios but for most public websites, it's not worth the risk.
So, Google actually appears to have capped the max_age (max max_age?) to 60 days in Chrome. Expect this in Release 51.
We got into a debate with them over why they're imposing it, but ultimately, our use case was limited and they likely had the analytics to justify capping max_age at less than a year. I think we'd still rather have had the option of allowing up to a year, but I understand the need for the gloves. https://bugs.chromium.org/p/chromium/issues/detail?id=523654
Which gets to my point: Cyph (https://cyph.im) found more utility in abusing HPKP off-label than in actually using it for its originally intended purpose, so in a roundabout way, I find myself in agreement with netcraft. Heh.
I just read through the issue and then went to cyph.im to take a look myself. It fails in my Firefox because of a HPKP problem, so it seems like I accidentally deleted the cached version (or it was cleaned up automatically?). How would you solve that issue for your use case?
Sorry, yeah, that is a known issue in Firefox that we're trying to have resolved; it apparently cleans up ServiceWorkers/AppCaches that haven't been actively used recently. Was this your first time opening cyph.im, or had you used it in the past?
Ah, cool, that makes more sense (I couldn't think of a reason for that to possibly happen on the first use, since at that point the pinned TLS key wouldn't yet have been invalidated).
Could malware that infects a webserver set a HPKP key to a private key that only it knew (ala cryptolocker), and then ransom that key?
I'm sure we could survive a total webserver compromise (with some level of lost data), but losing access to our primary domain for 60 days to a year would be devastating.
o The TLS connection was authenticated with a certificate chain
containing at least one of the SPKI structures indicated by at
least one of the given SPKI Fingerprints (see Section 2.6).
o The given set of Pins contains at least one Pin that does NOT
refer to an SPKI in the certificate chain. (That is, the host
must set a Backup Pin; see Section 4.3.)
> then deliver the page using that key and afterwards remove the other keys.
Per bullet 1, this seems self-defeating since at that point all that would need to happen is for the victim to forensically extract the private key from the compromised webserver. So long as there's one private key corresponding to a pinned public key, the malware's defeated if the private key can be recovered (assuming I'm understanding your suggestion correctly).
I say "forensically" since I'd presume any malware following this pattern would likely build in some protections to try and blow the private key in the event that someone tries to recover it.
It's not technically necessary for a TLS web server to have unfettered access to the private key. The web server could offload the signing operations to a separate server under the control of the attacker. The web server only has access to premaster and session keys. CloudFlare calls this Keyless SSL.
This would make the attack quite complex, though. It's more likely that the malware will just use the new key for a few hours and then delete it and scrub the disk, which will be enough for the majority of production deployments.
> So long as there's one private key corresponding to a pinned public key, the malware's defeated if the private key can be recovered (assuming I'm understanding your suggestion correctly).
Once the ransomware delivered the page to enough users (for some number of enough) it can permanently remove the key used to do so. The header now contains the just-removed key used to poison the browser and a randomly generated hash that does not correspond to a valid key. This assumes that the administrator does not detect the ransomware before it does so.
IIRC The pin must contain at least one key that's in the current chain and at least one key that's not in the current chain.
Then the malware could call out to let's encrypt to issue a certificate, then load the certificate into nginx/apache/whatever, then remove the let's encrypt private key from the file system. That allows the webserver to continue serving traffic with LE's key + ransomware key until it's restarted.
1.) Infect webserver
2.) Generate public/private key pair for the purposes of the ransom on the CC server, then send back the public key back to the compromised server.
3.) Use webserver to issue LE SSL certificate. Delete the private key once it's loaded into the webserver process.
4.) Serve traffic with the hashes of the LE key and the ransomware key.
If the sysadmin restarts the webserver process, they are screwed since that's the only process that contains the LE private key.
The blog post you posted by Scott Helme is much better than the one I've seen posted in previous threads[0]. The information about not necessarily pinning to a leaf is critical.
Please Please Please read this.
>For me personally, on my blog and on other websites that I run, I've pinned at the leaf certificate level. It does come with the problem that you're responsible for the backups...
HPKP is a giant foot gun and even Ryan Sleevi expresses how he is somewhat unsatisfied with how HPKP turned out[1] and that you should be pinning to intermediates or roots.[2] (His name is on the RFC if you don't know who he is).
In my opinion. If you plan to roll out HPKP you need to read the RFC. Don't follow online guides for your production environment. Figure out what is best for you.
There is a really high operational cost to this in organizations bigger than a couple people. If you aren't prepared for that, then you're going to brick your site for a lot of users.
Security headers are hard to get right and even some really big orgs make mistakes[3]
The main reason not to use HPKP is it only protects people who have already connected. There are a hundred ways to force a new user agent or device to connect, and then poof, compromised.
Pinning is a great way for companies like Google to use Chrome to detect when someone has deployed an illegally issued certificate on their behalf. But for a company that hasn't bundled its keys with browsers, the user will just spin up a different browser and go to the compromised site, since HPKP is browser/session specific.
HPKP is a toothless mitigation. We need better measures.
This is especially troubling in situations where you're not using your trusted browser, but instead some embedded one. Twitter's in-app web view is the best example of this.
30 comments
[ 3.9 ms ] story [ 79.5 ms ] threadWere we bigger, the benefit might outweigh the costs. Of course if we were bigger, the cost would go up significantly as well. Can you imagine a world where Amazon had to completely re-build its order taking infrastructure around a new domain name due to a mistaken configuration file?
Generally they would do multiple tests to avoid such cases.
RFC 7469 requires this (https://tools.ietf.org/html/rfc7469#section-2.5):
https://brianmhunt.github.io/picking-our-roots-between-hsts-...
As falcolas says, security is about risk tradeoffs. Right now the risk of using HPKP is rightly judged by most webmasters to be far greater than the risk of not using it. Key pinning Is a great security tool in many scenarios but for most public websites, it's not worth the risk.
We got into a debate with them over why they're imposing it, but ultimately, our use case was limited and they likely had the analytics to justify capping max_age at less than a year. I think we'd still rather have had the option of allowing up to a year, but I understand the need for the gloves. https://bugs.chromium.org/p/chromium/issues/detail?id=523654
Which gets to my point: Cyph (https://cyph.im) found more utility in abusing HPKP off-label than in actually using it for its originally intended purpose, so in a roundabout way, I find myself in agreement with netcraft. Heh.
I'm sure we could survive a total webserver compromise (with some level of lost data), but losing access to our primary domain for 60 days to a year would be devastating.
So it first would need to set the header to include the ransomed key, then deliver the page using that key and afterwards remove the other keys.
Edit: I just read in the RFC. It is in section 2.5 (https://tools.ietf.org/html/rfc7469#section-2.5):
Per bullet 1, this seems self-defeating since at that point all that would need to happen is for the victim to forensically extract the private key from the compromised webserver. So long as there's one private key corresponding to a pinned public key, the malware's defeated if the private key can be recovered (assuming I'm understanding your suggestion correctly).
I say "forensically" since I'd presume any malware following this pattern would likely build in some protections to try and blow the private key in the event that someone tries to recover it.
This would make the attack quite complex, though. It's more likely that the malware will just use the new key for a few hours and then delete it and scrub the disk, which will be enough for the majority of production deployments.
Once the ransomware delivered the page to enough users (for some number of enough) it can permanently remove the key used to do so. The header now contains the just-removed key used to poison the browser and a randomly generated hash that does not correspond to a valid key. This assumes that the administrator does not detect the ransomware before it does so.
Then the malware could call out to let's encrypt to issue a certificate, then load the certificate into nginx/apache/whatever, then remove the let's encrypt private key from the file system. That allows the webserver to continue serving traffic with LE's key + ransomware key until it's restarted.
1.) Infect webserver
2.) Generate public/private key pair for the purposes of the ransom on the CC server, then send back the public key back to the compromised server.
3.) Use webserver to issue LE SSL certificate. Delete the private key once it's loaded into the webserver process.
4.) Serve traffic with the hashes of the LE key and the ransomware key.
If the sysadmin restarts the webserver process, they are screwed since that's the only process that contains the LE private key.
- https://scotthelme.co.uk/guidance-on-setting-up-hpkp/
Please Please Please read this.
>For me personally, on my blog and on other websites that I run, I've pinned at the leaf certificate level. It does come with the problem that you're responsible for the backups...
HPKP is a giant foot gun and even Ryan Sleevi expresses how he is somewhat unsatisfied with how HPKP turned out[1] and that you should be pinning to intermediates or roots.[2] (His name is on the RFC if you don't know who he is).
In my opinion. If you plan to roll out HPKP you need to read the RFC. Don't follow online guides for your production environment. Figure out what is best for you.
There is a really high operational cost to this in organizations bigger than a couple people. If you aren't prepared for that, then you're going to brick your site for a lot of users.
Security headers are hard to get right and even some really big orgs make mistakes[3]
0 - https://scotthelme.co.uk/hpkp-http-public-key-pinning/
1 - https://twitter.com/sleevi_/status/696171562383224832
2 - https://twitter.com/sleevi_/status/696379082599059457
3 - https://chromium.googlesource.com/chromium/src.git/+/3865c64...
Pinning is a great way for companies like Google to use Chrome to detect when someone has deployed an illegally issued certificate on their behalf. But for a company that hasn't bundled its keys with browsers, the user will just spin up a different browser and go to the compromised site, since HPKP is browser/session specific.
HPKP is a toothless mitigation. We need better measures.