tl;dr: the certificate was requested in 2010 and an email challenge was sent at that time, but the validation URL wasn't clicked until last week, and the certificate was allowed to be issued because of a misconfigured linter.
Technically, it appears that a validation link to prove domain ownership was sent 10 (or 11) years prior, but wasn't clicked until recently. The issue isn't that ownership was proven ten years ago, the issue is that the link to do so was still valid for such a long period of time rather than being expired.
Well yes. That was an obvious issue. However the point is that the email was received 10 years ago, so you are only proving that you had control of the domain at that time.
Of course validation links should expire in a relatively short timeframe.
Right. GlobalSign asserts in their incident report that:
> All other orders in the GCC platform that are older than 1 year and have not been fully processed (940909 orders) have been moved into a cancelled state on 9/02/2021.
So (in this platform at least, and I believe this is either their oldest or one of their oldest legacy platforms) until this fix there were actually almost one million such certificates that could in principle be created based on information over a year old.
They've attached a list of 112 other certificates that really were issued - and I think a reasonable person would judge should not have been issued, all have expired (and so there's no point in revoking them now). This at least shows they were thinking about the potential wider problem and not just focused on damage control for this one certificate, so that's something.
The reference to "Ballot 169" is to the Ten Blessed Methods, Ballot 169 was a vote by the CA/Browser Forum to modify the Baseline Requirements to require these specific methods (today there aren't actually ten of them, but the general idea of spelling out how to validate control remains) rather than allowing CAs to each make up whatever methods of domain validation they thought would be good enough in their opinion. Ballot 169 actually got caught in a big bureaucratic mess but Mozilla forced the issue by making the Ten Blessed Methods part of its own rules anyway while the mess was sorted out so the CAs had to obey.
Under the Ten Blessed Methods there's an explicit deadline, the random codes they're sending out by email are only good for thirty days. But I think we can judge from this that GlobalSign's prior method never expired codes at all, and you can imagine that when a deadline was introduced nobody wanted to retroactively extend that deadline to previous codes.
At the moment that decision happens it seems superficially OK. Last year it was OK for Sue in accounting to take six weeks to finally click the link, next year the Ten Blessed Methods mean we'll have to show a "I'm sorry this request has expired, please order again" message but why jump ahead, we should let Sue's old order go through whenever she gets around to clicking it.
It takes a particular type of person to ask "What if Sue takes six years instead of six weeks?" and those sorts of people don't tend to work for a CA.
For example, the Chrome browser has code to actually enforce the requirements limiting leaf certificate lifetimes. In the modern era the browser code matches exactly what the rules say. Slightly before that the rules are rather woolly and so the browser approximates them, how long is "39 months" in terms a machine can understand? Chrome's programmers worked out a rationale for 1188 days as the answer.
But at the outset there just weren't any rules. You could react to that by saying OK, if the certificate claims to have been issued in 1995 then I guess we believe them and accept any validity period whatsoever. 40 year leaf certificate? No problem. This opens an enormous security hole. So, Chrome's engineers just picked 10 years.
And just like that, the hole closes, because sure enough it has been more than 10 years since the last date when such "No rules" certificates could be issued, and so there are no certificates issued recently enough to be unexpired which are allowed a ten year expiry.
This is a deeply concerning issue to have happen, but I do want to commend GlobalSign for their detailed response in the ticket. It looks like there's other issues they have not yet addressed per the last comment from Ryan Sleevi, but their response so far is very helpful in understanding what occurred.
Good news, you don't have to wonder how many GlobalSign issued, just look at the Excel file attached to the ticket and you can see all 113 of their "oopsies".
Note that (opened in LibreOffice but I assume this isn't a document compat issue) the URLs all start https// but should begin https:// instead. They're links into crt.sh which is a public view into the contents of the Certificate Transparency Logs, although they've also baked the certificate serial numbers into these links so if crt.sh goes away you can rewrite them as queries for any other system.
It looks like an expected long tail, most of them were probably issued only days or weeks after the codes used should have expired (in 2017).
What made this one jump out was the use of a 1024-bit RSA key. When it was really ordered that would be completely normal, but even in 2017 it would have been obviously prohibited, I have looked at a few of the other 112 and they're unextraordinary, typically 2048-bit RSA keys, likely originally ordered earlier in 2017.
It would be useful (perhaps I will ask on m.d.s.policy) to know what the actual original order dates were for the 112 certificates in this list.
I assume the https// is intentional to prevent autolinking or similar (excel loves to make hyperlinks and not everyone knows how to beat it into submission).
> It would be useful (perhaps I will ask on m.d.s.policy) to know what the actual original order dates were for the 112 certificates in this list.
It'd be really fascinating to compare that with WHOIS data for the domains. That might show other misissuances - certificates issued for non-registered domains.
What you really want here is the other side of the story, from the CelticPapaya people. Does anyone know anyone there? From Archive.org, it appears the site has never worked properly. I found a couple of email addresses, though, in the Google cache.
During a migration from one email system to another, it wouldn't be unusual for some virus scanner product to click every link in every email...
Or alternatively, when searching their inbox for "SSL certificate" they found the verification email and clicked the link as a quick way to get to the homepage of whichever company they used last time for their certificate.
Or they searched for SSL certificate, found the email, and didn’t notice the date. Perhaps it was marked unread, given that the email verification was never completed in 2010. They thought it was something they needed to do, and did it.
It's definitely plausible that the historic URLs which caused this are vulnerable to just being opened automatically by a machine (such as a "virus scanner"). The right fix here is that the URL leads to a simple POST form where you confirm you want the certificate issued. Tools like virus scanners don't fill out forms.
We know that by the time the Ten Blessed Methods were coming into existence CAs were discovering that they had numerous cases where an order would be successfully "validated" by contact with a human, but without any human actually doing anything. Email links "clicked" by automated systems, a system that checks you've uploaded a file to your web server treats the 404 error it gets as an acceptable response showing the file was uploaded, several hilarious goofs.
[ The irony, for those playing along at home, is that we don't know of any incidents where this was used for an attack. It's like coming home to discover the reason you didn't have your front door key is because you left it on the kitchen counter, with the door unlocked, all day. But no harm done... ]
The current authorised method closest in spirit to what GlobalSign were doing back in 2010 is 3.2.2.4.4 (Constructed Email to Domain Contact). This method actually doesn't tell them to prevent this particular mishap (machines clicking links), so perhaps that should be corrected.
GET requests should not have any significant side-effects, unfortunately some do not follow that rule. I once read a story of a whole database getting wiped, because a spider "clicked" on every "delete" link.
Idempotence is not enough, it only requires at executing a request a second time (with no other changes in between) has no significant effect, the first time is allowed to have a significant effect. PUT requests are the typical example.
No. An idempotent operation is an operation that can be applied any number of times without a change after the initial invocation. This means the first GET would fire and change state, but subsequent GETs would not.
GET should be nullipotent: it is an operation which can be applied any number of times without any change.
Also, this went from being an obscure fact about HTTP that might occasionally bite you to being extremely relevant to production systems with TLS 1.3 (thus some HTTPS and all HTTP/2 and HTTP/3 systems)
TLS 1.3's zero round trip resumption by its nature can't be bullet proof against repetition. In toy systems you can defend against this (and you might want to) but in big distributed systems it's going to be possible for an attack to play some HTTP request to your system (perhaps different nodes of it) more than once without you realising.
This is obviously unsafe if that HTTP request moves $1000 from your bank account to somebody else's bank account (now they have $1000 extra), and on the other hand obviously safe if the HTTP request is just for the Favicon of Hacker News (now they have another identical copy of the encrypted data) but there's also lots of grey area in between.
Translating this human understanding into a rule for machines to follow is the exercise where security holes can occur. The safe option is to just never allow 0RTT resumption (as a client or as a server, either can safely refuse to have any part in this) but in pursuit of performance you can expect other choices to be made.
> TLS 1.3's zero round trip resumption by its nature can't be bullet proof against repetition
The server side can easily keep track of which nonces have been used, and not allow any re-used nonces to have any side effects.
This can be implemented in something like spanner by keeping a database of valid nonces, and tying the marking a nonce as used to the other effects of the HTTP request in the same transaction. Spanners global-consistency will ensure only one transaction will commit. In most cases, the data can be sent back to the user without waiting for commit, preventing any cross-ocean latency penalty too.
It's a shame that I don't see a single HTTP/3 server which has any support for a scheme like this.
That's possible, but not really easy. If you don't wait for the commit, you may process the request before getting the notification that it was a duplicate nonce; if you do wait for the commit, there goes your performance boost.
The right thing to do in my mind is to only send requests in 0-rtt if they're specifically flagged as eligible.
> If you don't wait for the commit, you may process the request before getting the notification that it was a duplicate nonce.
Yes, do this. Processing the request isn't bad as long as it doesn't commit.
Most evil attackers won't come back to you complaining that your server gave back an HTTP200 response when it should have returned an HTTP400 because they'd reused a nonce.
The evil attackers don't necessarily care whether in some sense your system "doesn't commit" one of the duplicates. Especially if they can manipulate it to choose whether the "not committed" one is the one they created or one emitted by a real user.
I know it was intended flippantly but the attackers do not get the HTTP response code - they're obliged to conduct this attack blind, if they can decrypt the messages then your problem isn't a replay attack but some previously unimagined TLS flaw.
So, suppose your system sees three versions of a transaction to transfer $1M to Dr Evil. You're not worried because two of them "won't commit" but do your end users understand that? (hint: they do not). Let's call these versions A, B and C. The real user's web browser believes transaction A succeeded, and is unaware of B and C.
Dr Evil complains that the money never arrived, and our user checks, alas they see that eventually transaction A failed (it was judged a duplicate of B, and so was C) and this explains the problem. They initiate a new transfer, let's call that Z. This succeeds and Dr Evil is satisfied.
Congratulations, you have joined many excellent engineers in designing a toy solution to this problem. Some of the others were documented in RFC 8446 (with TLS 1.3 itself) but if you didn't cheat by looking there yours was an independent invention.
Sadly this doesn't scale, as you almost seemed to realise by saying you'd just respond without waiting for the commit step to complete. Next I'll respond to your dismissal of this huge problem in the other thread.
I agree that the risk of replays due to TLS 1.3 0-RTT is bad, but I understand browsers (e.g. Chrome) can often be made to silently replay requests at a lower TLS version anyway (to work around middleboxes that choke on higher-than-expected TLS versions.) So avoiding TLS 1.3 0-RTT doesn't avoid replays... and yes, this is bad.
The main take away I see is a reorg/consolidation of their processes:
> GlobalSign has concluded there are too many procedural and technical gaps in the current process to prevent incorrect CA configuration occurring in the future and has decided to halt the provisioning of new CA in the GCC platform until 31/03/2021, this in order to fully rebuild the CA request, creation and provisioning process. Whereas the previous process could be considered fragmented (with different tickets and information depending on the team that had to execute a specific sub-action) we will be building a new flow that consolidates all required information and actions into a single process, with additional steps in terms of compliance oversight and a hard compliance approval step prior to putting a new CA into production / issuing state. We will share the details of the new process 16/02/2021 and will complete implementation before 31/03/2021. CA provisioning on the GCC platform will only resume after the implementation of the new process has been successfully completed.
> Certificate ordered with ID [...] for www.celticpapaya.com.
> Domain flagged as high-risk (keyword "pay")
Somehow, I found this line particularly funny. Simple systems and heuristics like keyword searching continue to power and protect systems around the world, triggering false positives (and negatives) that can often be opaque to outside observers.
Will ML and NLP eventually overtake these systems, bringing along their own pros/cons? Or will these systems continue to proliferate into the far future, such that in the distant year 2077, some poor traveler gets dragged out for special screening because they listed "burgundy" in their customs manifest?
I also find it funny. But the key redeeming feature is that it was paused, and automatically sent to human review. So sure, your cert takes a little while to issue but no significant harm.
The obvious contrast is Google closing legitimate paying accounts with no option for human review.
39 comments
[ 2.9 ms ] story [ 78.4 ms ] threadThat also sounds like a serious issue.
Of course validation links should expire in a relatively short timeframe.
> All other orders in the GCC platform that are older than 1 year and have not been fully processed (940909 orders) have been moved into a cancelled state on 9/02/2021.
So (in this platform at least, and I believe this is either their oldest or one of their oldest legacy platforms) until this fix there were actually almost one million such certificates that could in principle be created based on information over a year old.
They've attached a list of 112 other certificates that really were issued - and I think a reasonable person would judge should not have been issued, all have expired (and so there's no point in revoking them now). This at least shows they were thinking about the potential wider problem and not just focused on damage control for this one certificate, so that's something.
The reference to "Ballot 169" is to the Ten Blessed Methods, Ballot 169 was a vote by the CA/Browser Forum to modify the Baseline Requirements to require these specific methods (today there aren't actually ten of them, but the general idea of spelling out how to validate control remains) rather than allowing CAs to each make up whatever methods of domain validation they thought would be good enough in their opinion. Ballot 169 actually got caught in a big bureaucratic mess but Mozilla forced the issue by making the Ten Blessed Methods part of its own rules anyway while the mess was sorted out so the CAs had to obey.
Under the Ten Blessed Methods there's an explicit deadline, the random codes they're sending out by email are only good for thirty days. But I think we can judge from this that GlobalSign's prior method never expired codes at all, and you can imagine that when a deadline was introduced nobody wanted to retroactively extend that deadline to previous codes.
At the moment that decision happens it seems superficially OK. Last year it was OK for Sue in accounting to take six weeks to finally click the link, next year the Ten Blessed Methods mean we'll have to show a "I'm sorry this request has expired, please order again" message but why jump ahead, we should let Sue's old order go through whenever she gets around to clicking it.
It takes a particular type of person to ask "What if Sue takes six years instead of six weeks?" and those sorts of people don't tend to work for a CA.
For example, the Chrome browser has code to actually enforce the requirements limiting leaf certificate lifetimes. In the modern era the browser code matches exactly what the rules say. Slightly before that the rules are rather woolly and so the browser approximates them, how long is "39 months" in terms a machine can understand? Chrome's programmers worked out a rationale for 1188 days as the answer.
But at the outset there just weren't any rules. You could react to that by saying OK, if the certificate claims to have been issued in 1995 then I guess we believe them and accept any validity period whatsoever. 40 year leaf certificate? No problem. This opens an enormous security hole. So, Chrome's engineers just picked 10 years.
And just like that, the hole closes, because sure enough it has been more than 10 years since the last date when such "No rules" certificates could be issued, and so there are no certificates issued recently enough to be unexpired which are allowed a ten year expiry.
But terrifying at the same time. I wonder how many valuable targets might have had outstanding validations from 10 years ago, or even 3 years ago.
But yeah, as Ryan Sleevi asks on the bugreport, I wonder how many other certificates may have been issued based on "historical" verification tokens...
It looks like an expected long tail, most of them were probably issued only days or weeks after the codes used should have expired (in 2017).
What made this one jump out was the use of a 1024-bit RSA key. When it was really ordered that would be completely normal, but even in 2017 it would have been obviously prohibited, I have looked at a few of the other 112 and they're unextraordinary, typically 2048-bit RSA keys, likely originally ordered earlier in 2017.
It would be useful (perhaps I will ask on m.d.s.policy) to know what the actual original order dates were for the 112 certificates in this list.
It'd be really fascinating to compare that with WHOIS data for the domains. That might show other misissuances - certificates issued for non-registered domains.
Or alternatively, when searching their inbox for "SSL certificate" they found the verification email and clicked the link as a quick way to get to the homepage of whichever company they used last time for their certificate.
We know that by the time the Ten Blessed Methods were coming into existence CAs were discovering that they had numerous cases where an order would be successfully "validated" by contact with a human, but without any human actually doing anything. Email links "clicked" by automated systems, a system that checks you've uploaded a file to your web server treats the 404 error it gets as an acceptable response showing the file was uploaded, several hilarious goofs.
[ The irony, for those playing along at home, is that we don't know of any incidents where this was used for an attack. It's like coming home to discover the reason you didn't have your front door key is because you left it on the kitchen counter, with the door unlocked, all day. But no harm done... ]
The current authorised method closest in spirit to what GlobalSign were doing back in 2010 is 3.2.2.4.4 (Constructed Email to Domain Contact). This method actually doesn't tell them to prevent this particular mishap (machines clicking links), so perhaps that should be corrected.
Idempotence is not enough, it only requires at executing a request a second time (with no other changes in between) has no significant effect, the first time is allowed to have a significant effect. PUT requests are the typical example.
No. An idempotent operation is an operation that can be applied any number of times without a change after the initial invocation. This means the first GET would fire and change state, but subsequent GETs would not.
GET should be nullipotent: it is an operation which can be applied any number of times without any change.
TLS 1.3's zero round trip resumption by its nature can't be bullet proof against repetition. In toy systems you can defend against this (and you might want to) but in big distributed systems it's going to be possible for an attack to play some HTTP request to your system (perhaps different nodes of it) more than once without you realising.
This is obviously unsafe if that HTTP request moves $1000 from your bank account to somebody else's bank account (now they have $1000 extra), and on the other hand obviously safe if the HTTP request is just for the Favicon of Hacker News (now they have another identical copy of the encrypted data) but there's also lots of grey area in between.
Translating this human understanding into a rule for machines to follow is the exercise where security holes can occur. The safe option is to just never allow 0RTT resumption (as a client or as a server, either can safely refuse to have any part in this) but in pursuit of performance you can expect other choices to be made.
The server side can easily keep track of which nonces have been used, and not allow any re-used nonces to have any side effects.
This can be implemented in something like spanner by keeping a database of valid nonces, and tying the marking a nonce as used to the other effects of the HTTP request in the same transaction. Spanners global-consistency will ensure only one transaction will commit. In most cases, the data can be sent back to the user without waiting for commit, preventing any cross-ocean latency penalty too.
It's a shame that I don't see a single HTTP/3 server which has any support for a scheme like this.
The right thing to do in my mind is to only send requests in 0-rtt if they're specifically flagged as eligible.
Yes, do this. Processing the request isn't bad as long as it doesn't commit.
Most evil attackers won't come back to you complaining that your server gave back an HTTP200 response when it should have returned an HTTP400 because they'd reused a nonce.
I know it was intended flippantly but the attackers do not get the HTTP response code - they're obliged to conduct this attack blind, if they can decrypt the messages then your problem isn't a replay attack but some previously unimagined TLS flaw.
So, suppose your system sees three versions of a transaction to transfer $1M to Dr Evil. You're not worried because two of them "won't commit" but do your end users understand that? (hint: they do not). Let's call these versions A, B and C. The real user's web browser believes transaction A succeeded, and is unaware of B and C.
Dr Evil complains that the money never arrived, and our user checks, alas they see that eventually transaction A failed (it was judged a duplicate of B, and so was C) and this explains the problem. They initiate a new transfer, let's call that Z. This succeeds and Dr Evil is satisfied.
Unfortunately now Dr Evil has $2M instead of $1M.
Sadly this doesn't scale, as you almost seemed to realise by saying you'd just respond without waiting for the commit step to complete. Next I'll respond to your dismissal of this huge problem in the other thread.
https://twitter.com/XorNinja/status/1357828529610608641 may be(-come) relevant.
(Sorry, your post is already a day old as I type this. You might still see this response?)
Of course it wasn't working properly, they were waiting for their x.509 certificate to issue. /s
> GlobalSign has concluded there are too many procedural and technical gaps in the current process to prevent incorrect CA configuration occurring in the future and has decided to halt the provisioning of new CA in the GCC platform until 31/03/2021, this in order to fully rebuild the CA request, creation and provisioning process. Whereas the previous process could be considered fragmented (with different tickets and information depending on the team that had to execute a specific sub-action) we will be building a new flow that consolidates all required information and actions into a single process, with additional steps in terms of compliance oversight and a hard compliance approval step prior to putting a new CA into production / issuing state. We will share the details of the new process 16/02/2021 and will complete implementation before 31/03/2021. CA provisioning on the GCC platform will only resume after the implementation of the new process has been successfully completed.
> Domain flagged as high-risk (keyword "pay")
Somehow, I found this line particularly funny. Simple systems and heuristics like keyword searching continue to power and protect systems around the world, triggering false positives (and negatives) that can often be opaque to outside observers.
Will ML and NLP eventually overtake these systems, bringing along their own pros/cons? Or will these systems continue to proliferate into the far future, such that in the distant year 2077, some poor traveler gets dragged out for special screening because they listed "burgundy" in their customs manifest?
The obvious contrast is Google closing legitimate paying accounts with no option for human review.