This is a common server misconfiguration that clients can often work in spite of, due to certificate caching and explicit workarounds (AIA Fetching), and I sort of wish they wouldn't. I'm no expert so I'm sure people will tell me why this isn't workable, but it seems to me that if browsers were to actively require that a server provides any required intermediate certificates people would fix their servers, as they wouldn't work at all.
I don't think servers should be breaking working websites just to be pedantic if they can already work around these issues.
They should definitely put warnings about these kinds of faults in the dev console, though. I don't think developers that don't get their certs in order will even bother with opening the dev tools on their website, but it's the best place I can think of.
Google and other partnering websites also have a position to prevent these failures. Many websites decide to include Google's stalking scripts and it should be trivial for Google to put scan the domains these scripts run on and warn the website administrators about their bad configuration.
This poses a problem for split horizon intranet websites, but in those cases you'll probably never run into the issue due to the weird setup anyway.
> Many websites decide to include Google's stalking scripts and it should be trivial for Google to put scan the domains these scripts run on and warn the website administrators about their bad configuration.
The Google Webmaster Console actually does warn you about all issues regarding your site's ability to be indexed, including SSL issues.
One issue is that there’s no “one true list of intermediates” for a server to include in the handshake, as it’s not possible to always know what roots a client trusts.
The client authors (of browsers) tend to care about making their software easy to use and avoid errors. And there’s a pretty small set of clients.
TLS servers on the other hand often just serve whatever you put in a config file. A chain that works for them on an up to date Firefox may not work for an older android phone, but the server administrator may not be aware of the trade off they’ve made.
My personal opinion is that the tools for debugging for server authors aren’t good enough yet. I’m one of the authors of the certigo tool mentioned in the article, which was written out of frustration from a lack of other tools. Even then, I think it’s not good enough and I hope this class of issue is one we can illustrate better. SSL Labs probably does the best here.
> One issue is that there’s no “one true list of intermediates” for a server to include in the handshake
This doesn't make sense. Given a cert that was issued from an intermediate, there is exactly one intermediate that issued it, and that intermediate must be in the handshake. If that intermediate was issued from another intermediate, that next intermediate must also be in the handshake. And so on. There's no ambiguity here. Your server is required to send a fully-formed cert chain whose final cert is signed by one of the roots the client trusts.
It's entirely possible to run a PKI where the set of intermediates is known by nobody and is unbounded and ephemeral, so long as everyone who is given a cert is given the intermediate chain that issued it. The burden is on the bearer of the cert to include it in the handshake. All problems here have arisen from a misunderstanding that causes people to think the intermediates should be well-known or obtainable publicly. That is simply not true.
> it’s not possible to always know what roots a client trusts
It doesn't matter. You send the chain that issued your cert, not what you expect the client to trust. If the client trusts it, you're good. If not, you're not.
TFA complains about browser intermediate cache behavior but doesn't seem to realize that if you're relying on browser intermediate caching at all, your server is misconfigured. The lesson should be "fix your server", not "those darn browsers."
TFA is a war story about how infelicities such as caching in tools and “those darn browsers” contributed to the identification of the cause of this issue taking longer than it should have.
Once the missing intermediate certificate issue had been identified, the fix (to the server) was indeed clear. But it's not stated whether the author is in charge of the server in question (it's only mentioned as “couldn't connect to a particular HTTPS site”); I suspect this was some external web site not under their control.
Your end entity cert was signed by some specific intermediate CA yes. And that intermediate is often (always?) only signed by one CA. But that CA may have signed certificates from multiple other CAs (generally this is known as cross signing).
If your client doesn't trust the CA that signed the intermediate that signed your cert, you'll need a cross signed CA cert that works for your clients, but sometimes it's not possible to make a single chain that supports all the clients you want. And it's difficult to differentiate (and even if you can figure out how to differentiate, you're probably going to be patching your TLS server to do it, because it's not widely supported).
When you consider poor client behavior, you get situations like client A doesn't have your CA, but does have another you can cross chain to; client B has your CA and not that other one and has poor verification logic, so if you include the cross chain, verification will fail; you can't make both clients work with the same chain (unless you switch to a different CA). My personal favorite though is when the client helpfully supplies a cached cross chain, but the certificate expired, so it fails validation... even though you didn't send that cert, and you have a path it would validate if it didn't have the cached certificate.
I'm exceedingly glad I don't have to deal with this anymore. But at least TLS1.3 changed the spec to require(i think) clients to handle out of order or extraneous certs which strongly suggests clients should try all possible paths and pass the validation if any are valid. Still no way to present multiple end entity certificates though (end entity must be the first certificate, you couldn't send two full chains)
The issue from TFA was a simple matter of not including an intermediate at all, not confusion about which intermediate you should include when cross-signing (the article says the intermediate wasn't included, and there is no mention of cross-signing at all).
Either way, the proper solution is "include all the intermediates," which applies to cross-signing scenarios. The client will find the path from leaf to any root it trusts. If you had some kind of crazy number of alternate paths for a given cert, this might become onerous, but in reality, the number of alternate paths is basically always either zero or one.
In general, we don't actually have enough information to tell this. :-)
Consider the case of an older Android client (not from TFA, but bear with me), using certs from Let's Encrypt: https://letsencrypt.org/certificates/
On an older server install, it could be the server was setup with a manual R3->leaf trust, serving (at the time) both copies of R3. This would've been within TLS spec: https://datatracker.ietf.org/doc/html/rfc8446#section-4.4.2 as you are within rights to omit roots:
> a certificate that specifies a trust anchor MAY be omitted from the chain, provided that supported peers are known to possess any omitted certificates.
Both ISRG Root X1 and DST Root CA X3 are well-known roots, and so might be omitted by an operator.
ACME does its thing, but this (admittedly misconfigured) server only updates the leaf cert.
However, original copy of DST->R3 expires. While we have two intermediates in the served chain, because our client relies on the DST trust root, it can't validate it because the cross-signed ISRG Root X1 is missing. But when browsing to a correctly-configured LE-based server (serving the cross-signed ISRG Root X1), this will prime the intermediate cache and allow validation to succeed.
And from the PoV of the old Android based client, this would look like an intermediate is missing (because it doesn't trust the self-signed ISRG Root X1 but does the cross-signed copy).
My 2c, but its entirely plausible (with public CAs) that this could be the case if one of these installations is out of date.
I believe this comment accurately portrays why TLS is a quagmire that despite being built on reasonable cryptographic primitives winds up being compromised by whatever flavor-of-the-month vulnerability is out there.
>One issue is that there’s no “one true list of intermediates” for a server to include in the handshake
The one true list of intermediates is "the intermediates between the leaf certificate and the root".
Cross-signing is a corner case.
Technically, TLS 1.2 cannot support cross-signing due to the requirement that each certificate in the chain certify the immediately preceding one (which is obviously not possible with cross-signing, because the cross-signing intermediates break the chain into two paths).
Practically speaking, it works anyway (and TLS 1.3 fixes the problem with the specification) and you should send any intermediates you have and expect the client to construct a chain to a root that it trusts.
It's absolutely not that hard, but my experience is there's a lot of cargo cult engineering around TLS configuration, as well as a degree of deploying-it-until-it-works-for-me.
My personal low was experiencing issues with a server that, upon investigations, was sending 22 certificates including all of the current- and next-generation CAs roots for our internal PKI as well as all of the issuing and intermediate CAs. The TLS handshake was about 200KB on the wire and would fail for some implementations that gave up trying to put together a trusted path.
One thing that wasn’t clear to me from this post: the certificate in question had an AIA url, and chrome will download intermediates from that. Why didn’t that work in this case?
It sounded like the intermediate cert was very old so any browsers that had visited a site that used it previously would have cached it but anyone trying to obtain it now didn't work.
Seems weird though, you would think that if the intermediate had expired, the client cert would also have expired.
I'll leave you to parse it, but it notably lacks the requirement that CAs must truncate validity. Let's Encrypt is a great example: https://letsencrypt.org/certificates/ -- DST Root CA X3 -> ISRG Root X1 is a cross-signed root, and DST Root CA X3 expired recently (breaking some validation on Android devices). However, by issuing that cross-signed ISRG root before the DST root expired, Let's Encrypt is still working on older Android devices: https://letsencrypt.org/2020/12/21/extending-android-compati...
In practice though, most chain validation I've looked at (OpenSSL, NSS+pkix, Go) all validate the validity period of the intermediate (relative to now, not of the leaf) and will err if it isn't still valid.
What they don't validate is _root_ validity (since it is ultimately trusted and in the trust store, and is again how the Android support worked).
So typically you'd want to permit longer validity from roots (pending the caveat about supporting CRL/OCSP/... throughout the longer lifetime of any leaf), but truncate validity on leaves issued from intermediates.
have been caught by this one a number of times. Nowadays when I'm sending certs to vendors (that is, vendors who can't work LE), I include the fullchain cert by preference.
Yep. Either ACME, or some other reliable method to automate cert issuing.
Manual cert deployment is IMHO bad hygiene in this day and age. (Yes, I know there’s esoteric hardware and stuff like that where automation isn’t straight-forward, but that’s the exception, not the rule.)
E.g. on AWS there’s Certificate Manager, which is similar. (But obviously only usable for people on AWS.)
> Yes, I know there’s esoteric hardware and stuff like that where automation isn’t straight-forward, but that’s the exception, not the rule.
Maybe in your world but plenty of us work in places that have 1000s of servers, many of them internal, they don't support LE or there are security concerns with opening external firewalls just to support certificate updates so it still has to be done manually.
You might be lucky to work somewhere much smaller/newer that doesn't have the problem but please don't make disparaging statements about those of us who have to do this manually.
But then you have to use Caddy instead of a different webserver, and probably either give up features or re-learn how to do things you already had set up.
I don’t think ACME alone fixes it. I had this exact problem with letsencrypt when they did their recent cert switcharoo.
Intermediate cert was referred to by name in the cert chain but not served by the server in full.
Server was of course referencing the newly signed intermediate cert (called L3 I believe) but the client only had a stale copy/ version of L3 signed by an expired trusted cert.
Issue was on an embedded device that couldn’t be fixed so the newly signed intermediate cert needed installation everywhere manually
A similar odd case is when the intermediate cert is expired, but the root and leaf are still valid. Different TLS implementations forgive this differently!
Only by generating certificate authorities on the fly. You'd also find fingerprinting attempts in the certificate transparancy log pretty quickly.
I think browser vendors would take action against that stuff. I suppose it's possible for one website to pay a whole bunch of CAs for certificates, but I don't think they'd be able to get enough to actually track anyone.
I don't see why you would need to generate CAs on the fly. You could have a few end-points with a certificate signed by common intermediates, but not serving the intermediate themselves. Then you can check if resources from those end-points load or not, effectively fingerprinting the user's cache for those common intermediates.
I'm not sure how feasible this is, but certainly seems more feasible than generating CAs on the fly.
That's possible but for that you'd need a valid certificate from every intermediate you fingerprint. You can maybe get a few bits out of it, but the moment you fingerprint the device it's got the intermediates practically forever. If the user clears all cookies, storage, etc. and you try to match the device with the old profile it'll still keep the old certs cached and it won't appear the same.
> browsers silently cache intermediate TLS certificates that they've seen before, in addition to being willing to download them under the right circumstances
A bit OT but how would you even go about downloading a certificate? From my understanding, certificates are transmitted as part of the TLS handshake, so it's not like there is some url you could send a GET request to. Or is there some facility in TLS where a client can request additional certificates?
If you visit the site in most (desktop) browsers, you can usually click the lock icon, navigate through various intermediate steps (Firefox is Connection Secure -> View Certificate) and find details about the cert or download it (hidden in the middle).
I didn't know about that command line invocation. Thanks a lot!
That's just the UI side though. I'm wondering what is actually happening on the wire when I run that command. Is there some TLS opcode which lets a client request certificate from the server?
Nah, no op code. You just open a regular connection and then terminate it. This forces the TLS server to auth (as the CLI OpenSSL lacks session resumption tickets).
Well yes, but the whole point of this thread is servers which don't include intermediate certs in the regular handshake. The article says about this:
> The website worked in other browsers because browsers silently cache intermediate TLS certificates that they've seen before, in addition to being willing to download them under the right circumstances.
I understood that as "some clients can recover from that situation by either caching the certs from a previous handshake or somehow downloading them on their own". And I was curious about the last point.
Safari on iOS/iPadOS displays a great "white page fast reload loop" when opening a website missing the intermediate certificate of an internal CA, even when that intermediate CA is installed in the device's keychain. That behavior is so weirdly specific, it's at least easy to identify the cause if you have seen it before.
49 comments
[ 4.3 ms ] story [ 113 ms ] threadThey should definitely put warnings about these kinds of faults in the dev console, though. I don't think developers that don't get their certs in order will even bother with opening the dev tools on their website, but it's the best place I can think of.
Google and other partnering websites also have a position to prevent these failures. Many websites decide to include Google's stalking scripts and it should be trivial for Google to put scan the domains these scripts run on and warn the website administrators about their bad configuration.
This poses a problem for split horizon intranet websites, but in those cases you'll probably never run into the issue due to the weird setup anyway.
The Google Webmaster Console actually does warn you about all issues regarding your site's ability to be indexed, including SSL issues.
The client authors (of browsers) tend to care about making their software easy to use and avoid errors. And there’s a pretty small set of clients.
TLS servers on the other hand often just serve whatever you put in a config file. A chain that works for them on an up to date Firefox may not work for an older android phone, but the server administrator may not be aware of the trade off they’ve made.
My personal opinion is that the tools for debugging for server authors aren’t good enough yet. I’m one of the authors of the certigo tool mentioned in the article, which was written out of frustration from a lack of other tools. Even then, I think it’s not good enough and I hope this class of issue is one we can illustrate better. SSL Labs probably does the best here.
This doesn't make sense. Given a cert that was issued from an intermediate, there is exactly one intermediate that issued it, and that intermediate must be in the handshake. If that intermediate was issued from another intermediate, that next intermediate must also be in the handshake. And so on. There's no ambiguity here. Your server is required to send a fully-formed cert chain whose final cert is signed by one of the roots the client trusts.
It's entirely possible to run a PKI where the set of intermediates is known by nobody and is unbounded and ephemeral, so long as everyone who is given a cert is given the intermediate chain that issued it. The burden is on the bearer of the cert to include it in the handshake. All problems here have arisen from a misunderstanding that causes people to think the intermediates should be well-known or obtainable publicly. That is simply not true.
> it’s not possible to always know what roots a client trusts
It doesn't matter. You send the chain that issued your cert, not what you expect the client to trust. If the client trusts it, you're good. If not, you're not.
Once the missing intermediate certificate issue had been identified, the fix (to the server) was indeed clear. But it's not stated whether the author is in charge of the server in question (it's only mentioned as “couldn't connect to a particular HTTPS site”); I suspect this was some external web site not under their control.
If your client doesn't trust the CA that signed the intermediate that signed your cert, you'll need a cross signed CA cert that works for your clients, but sometimes it's not possible to make a single chain that supports all the clients you want. And it's difficult to differentiate (and even if you can figure out how to differentiate, you're probably going to be patching your TLS server to do it, because it's not widely supported).
When you consider poor client behavior, you get situations like client A doesn't have your CA, but does have another you can cross chain to; client B has your CA and not that other one and has poor verification logic, so if you include the cross chain, verification will fail; you can't make both clients work with the same chain (unless you switch to a different CA). My personal favorite though is when the client helpfully supplies a cached cross chain, but the certificate expired, so it fails validation... even though you didn't send that cert, and you have a path it would validate if it didn't have the cached certificate.
I'm exceedingly glad I don't have to deal with this anymore. But at least TLS1.3 changed the spec to require(i think) clients to handle out of order or extraneous certs which strongly suggests clients should try all possible paths and pass the validation if any are valid. Still no way to present multiple end entity certificates though (end entity must be the first certificate, you couldn't send two full chains)
Either way, the proper solution is "include all the intermediates," which applies to cross-signing scenarios. The client will find the path from leaf to any root it trusts. If you had some kind of crazy number of alternate paths for a given cert, this might become onerous, but in reality, the number of alternate paths is basically always either zero or one.
Consider the case of an older Android client (not from TFA, but bear with me), using certs from Let's Encrypt: https://letsencrypt.org/certificates/
On an older server install, it could be the server was setup with a manual R3->leaf trust, serving (at the time) both copies of R3. This would've been within TLS spec: https://datatracker.ietf.org/doc/html/rfc8446#section-4.4.2 as you are within rights to omit roots:
> a certificate that specifies a trust anchor MAY be omitted from the chain, provided that supported peers are known to possess any omitted certificates.
Both ISRG Root X1 and DST Root CA X3 are well-known roots, and so might be omitted by an operator.
ACME does its thing, but this (admittedly misconfigured) server only updates the leaf cert.
However, original copy of DST->R3 expires. While we have two intermediates in the served chain, because our client relies on the DST trust root, it can't validate it because the cross-signed ISRG Root X1 is missing. But when browsing to a correctly-configured LE-based server (serving the cross-signed ISRG Root X1), this will prime the intermediate cache and allow validation to succeed.
And from the PoV of the old Android based client, this would look like an intermediate is missing (because it doesn't trust the self-signed ISRG Root X1 but does the cross-signed copy).
My 2c, but its entirely plausible (with public CAs) that this could be the case if one of these installations is out of date.
The big difference is https://datatracker.ietf.org/doc/html/rfc8446#section-4.4.2 calling out that the certificate list is unordered, and suggests that clients SHOULD be prepared to handle that in TLS 1.2 as well.
I don't know that there's anything better, and they were certainly convenient, but so many options, so many ways to mess things up.
The one true list of intermediates is "the intermediates between the leaf certificate and the root".
Cross-signing is a corner case.
Technically, TLS 1.2 cannot support cross-signing due to the requirement that each certificate in the chain certify the immediately preceding one (which is obviously not possible with cross-signing, because the cross-signing intermediates break the chain into two paths).
Practically speaking, it works anyway (and TLS 1.3 fixes the problem with the specification) and you should send any intermediates you have and expect the client to construct a chain to a root that it trusts.
It's absolutely not that hard, but my experience is there's a lot of cargo cult engineering around TLS configuration, as well as a degree of deploying-it-until-it-works-for-me.
My personal low was experiencing issues with a server that, upon investigations, was sending 22 certificates including all of the current- and next-generation CAs roots for our internal PKI as well as all of the issuing and intermediate CAs. The TLS handshake was about 200KB on the wire and would fail for some implementations that gave up trying to put together a trusted path.
Seems weird though, you would think that if the intermediate had expired, the client cert would also have expired.
I suspect the intermediate is still valid though.
The PKIX RFC 5280 says: https://datatracker.ietf.org/doc/html/rfc5280#section-4.1.2....
I'll leave you to parse it, but it notably lacks the requirement that CAs must truncate validity. Let's Encrypt is a great example: https://letsencrypt.org/certificates/ -- DST Root CA X3 -> ISRG Root X1 is a cross-signed root, and DST Root CA X3 expired recently (breaking some validation on Android devices). However, by issuing that cross-signed ISRG root before the DST root expired, Let's Encrypt is still working on older Android devices: https://letsencrypt.org/2020/12/21/extending-android-compati...
In practice though, most chain validation I've looked at (OpenSSL, NSS+pkix, Go) all validate the validity period of the intermediate (relative to now, not of the leaf) and will err if it isn't still valid.
What they don't validate is _root_ validity (since it is ultimately trusted and in the trust store, and is again how the Android support worked).
So typically you'd want to permit longer validity from roots (pending the caveat about supporting CRL/OCSP/... throughout the longer lifetime of any leaf), but truncate validity on leaves issued from intermediates.
https://www.digicert.com/help/
I really would have love to find a tool that would just spit out the right intermediate certificate for your ssl certificate.
Manual cert deployment is IMHO bad hygiene in this day and age. (Yes, I know there’s esoteric hardware and stuff like that where automation isn’t straight-forward, but that’s the exception, not the rule.)
E.g. on AWS there’s Certificate Manager, which is similar. (But obviously only usable for people on AWS.)
Maybe in your world but plenty of us work in places that have 1000s of servers, many of them internal, they don't support LE or there are security concerns with opening external firewalls just to support certificate updates so it still has to be done manually.
You might be lucky to work somewhere much smaller/newer that doesn't have the problem but please don't make disparaging statements about those of us who have to do this manually.
Intermediate cert was referred to by name in the cert chain but not served by the server in full.
Server was of course referencing the newly signed intermediate cert (called L3 I believe) but the client only had a stale copy/ version of L3 signed by an expired trusted cert.
Issue was on an embedded device that couldn’t be fixed so the newly signed intermediate cert needed installation everywhere manually
I think browser vendors would take action against that stuff. I suppose it's possible for one website to pay a whole bunch of CAs for certificates, but I don't think they'd be able to get enough to actually track anyone.
I'm not sure how feasible this is, but certainly seems more feasible than generating CAs on the fly.
That's not true, the point is that you fingerprint using misconfigured hosts, they never get the intermediates through them.
Of course it's possibly hard to use this practically, but anyway, apparently Firefox deemed it to be concerning enough to eventually fix it according to https://bugzilla.mozilla.org/show_bug.cgi?id=1334485.
A bit OT but how would you even go about downloading a certificate? From my understanding, certificates are transmitted as part of the TLS handshake, so it's not like there is some url you could send a GET request to. Or is there some facility in TLS where a client can request additional certificates?
$ openssl s_client -showcerts -connect <host>:<port>
also does it from the command line.
That's just the UI side though. I'm wondering what is actually happening on the wire when I run that command. Is there some TLS opcode which lets a client request certificate from the server?
> The website worked in other browsers because browsers silently cache intermediate TLS certificates that they've seen before, in addition to being willing to download them under the right circumstances.
I understood that as "some clients can recover from that situation by either caching the certs from a previous handshake or somehow downloading them on their own". And I was curious about the last point.