Curious to see how the CAB will handle this or if they're going to be "soft" as it's the first days of the CAA enforcement. Historically, they've been very accurate in enforcing their rules, which could mean a serious reprimand of Comodo.
If anyone is interested in testing their own CAA records, we built an online CAA validator specifically for this; https://dnsspy.io/labs/caa-validator
Between their decision to release a rebranded "more secure" version of Chrome where their changes had introduced security holes, and their decision to try and trademark "Let's Encrypt", I can't imagine they have many friends in the CAB.
On the other hand, they issue a lot of certs - if you've used Cloudflare's free SSL stuff, you've got a Comodo certificate - so they're unlikely to be shut down or anything that extreme.
> On the other hand, they issue a lot of certs - if you've used Cloudflare's free SSL stuff, you've got a Comodo certificate - so they're unlikely to be shut down or anything that extreme.
They're still smaller than Symantec, for whom the nuclear option was considered (and ultimately decided against, in favour of a phased distrusting).
To clarify, the CA/B Forum has no enforcement powers. This would only really be an issue if a root program takes issue with it or their auditor qualifies their audit.
Revocation for being three days late in implementing a new standard would be a massive overreaction. It would harm thousands of businesses in the process and would be extremely petty. Symantecs misdeeds are far worse than this.
Not at all. Revoke trust in the signing authority and all it's certificates turn invalid the second that change hits. Update your OS's trust store to not trust Comodo anymore and you'll see the effects immediately in IE/Edge on Windows or Safari and Chrome on macOS (Firefox still ships its own CA bundle).
It's just that when trust has been revoked in CAs recently, it's been done in a more gradual fashion to not hurt customers too much and give them time to migrate. After all, it's not the customer's fault the CA went rogue. This usually starts with not accepting certificates past a certain date but still validating previously issued ones, until sometime 6 months to a year later they're completely distrusted.
Exactly. For some contrast: I use tinydns, which doesn't natively support CAA records. But I wanted them so I read the CAA spec and wrote a little tool to add support for these records using the generic record format. And I'm not even a programmer! (I can write a little code, nothing serious, and nobody would hire me for it.)
And yet with all of Comodo's resources, they couldn't get this done? I can't take them seriously.
According to the mail, they have claimed to check CAA much earlier, so it's not just CAB guidelines they have broken, but their own word, which can be seen as worse.
Is it expected from all CAs that they obey CAA records, or is it something just made up by the community to crush the big CAs? I see an RFC from just a few years ago, and I'm not sure how these things are standardised.
Three days is quite a long time to be late, so I'd hope someone over there is getting a reprimand, but yeah, it's also not a disaster. They're response and time to remedy this will be more telling I think.
Three days over a weekend, though. Context matters. Even if it's the most critical incident, you can't force employees to work outside of business hours.
That makes no difference as to when the three days where. I never made any claims as to why it's late or them lying. I merely clarified that the three days were over a period where people don't usually work.
'Lie' is indeed a strong term, informally suggesting an intent to deceive. Personally, I would suspect negligence and incompetence rather than deceit, but negligence is a serious matter here.
Given the general crappiness of the CA industry, my first instinct is to say they willfully said "yeah we support it" without actually doing it first, knowing that it wasn't actually done.
On a requirement they voted for half a year ago for a standard specified 5 years ago. It's sloppy and sloppy is not a property you want in a certificate authority.
Highlights include allowing fraudulent certs to be issued for Google, Yahoo, and Mozilla; working with a malware company; and trademark shenanigans with Let's Encrypt.
Why is this a bad thing? A cert says "Yes, person x REALLY IS person x, and I can prove it mathematically." It doesn't say "Person x is trustworthy enough for me to vouch for them."
For malware concerns there's also Comodo's relationship with PrivDog. I'm not clear on whether PrivDog is deliberate malware or just incompetent insecure software.
That is ridiculous, to expect a CA to google every cert request domain and check what they are doing? That is a ridiculous requirement - cert issuance is automatic.
Do you think we should stop allowing Let's Encrypt to issue certs? You know they don't google each domain to see what they are doing.
In this particular case it's more likely negligence.
I can easily see how a missing integration test, and a few code changes later, the feature gets broken and nobody notices it. It doesn't prevent any normal codepath from executing properly.
Comodo is communicating actively in the bug, in the email discussion, and proactively CC'd the original reporter on the bug that was filed about this without being asked to do so.
The general rule in Operations work is tolerance. You don't fire someone for making a mistake, you fire them for lying about the mistake, for refusing to avoid mistake-prone behaviors, and other problem of that sort.
Applying that same principle to CA operations, why on earth should Comodo be "fired" as a CA for this? They're actively working to solve it, they're responsive, and the agreements they operate under will compel a post-mortem into existence if only to meet the terms of their next CA audit.
If, after investigation, they are found to be willfully violating policy, knowingly lying about CAA support, fail to produce an acceptable post-mortem, and/or fail their next CA audit, then of course they should be at risk of direct sanctions - not just a verbal reprimand.
So if your initial response to this issue is "Comodo should be fired", remember that in Operations culture that specific response "X should be fired" is a harmful response to a zero-day report of an incident, and many Operations teams would fire anyone who called for a firing in response to a fresh incident, in order to protect others from that harmful posture.
I would bet money (maybe not a lot of money, but money) that the only outcome of this is that Comodo ends up within a month being the CA that most reliably checks CAA records. It would be shocking if Google penalized them for this.
As you say, that's based in part on how they handle it.
The thing I'm sad to see is that there's no third-party information stated on when Comodo's CAA verification started and stopped. When they first announced CAA support, who tested it? When did it stop working? So I'm hoping this will come out of the post-mortem and be shared with us all.
Let's Encrypt is pretty good about checking CAA records. At work I built a system that allows our customers to order certs from LE and we see lots of failures for CAA query timeout (customer using a DNS provider that doesn't support CAA queries).
An employee at a big company makes a mistake that costs the company $500,000. The client is livid, and demands he be fired. The owner's response? "Why would I fire him? I just spent half a million dollars on training!"
This tends to be over-used though. Unfortunately some employees would continue making $500,000 mistakes and never learning. It really depends on the employee's attitude and response to the incident, as well as the circumstances of the incident.
It depends also on the employer's attitude and response to the incident. If the employee behaves perfectly and writes an impeccable post-mortem and is then fired "at the request of a client", that's on the employer, not the employee.
Kudos on this. Coming off of the Symantec thing, I think we're all viewing businesses in the CA space a lot more skeptically.
I worked in a hybrid Infrastructure / Development job for almost 17 years. One of the reasons I could never shake the infrastructure part of things[0] was that I knew the environment from more angles than most of the staff and I tended to be key in high severity issues. We were a highly geographically distributed team and coordinated almost all activity via conference calls.
We had two major rules when a critical failure was discovered. The first was that anyone asking "who caused this" was admonished and often ejected from the call. Often, the person responsible for the failure was on the calls and everyone had a pretty good understanding of who was at fault (if it was, indeed, a self-inflicted wound). The person at fault was also usually critical to the fix and putting them on the defensive caused evasiveness out of self-preservation. To help the first rule, the second rule was barring of anyone at the Director level or higher on these calls[1]. Generally, they were concerned with two things: (1) When's it going to work again and (2) why did it fail in the first place -- both result in delays. They were given updates, separately, by someone from the call at regular intervals. Keeping the politics low and focusing on the problem meant much shorter outages and calmer, more focused, brains of our technical staff.
And to your point, despite working at a company that laid off 10% or so of it's staff about every 9 months, I can't think of a case where a member of technical staff was specifically let go because of an (infrequent) failure -- even situations that would fall under gross negligence of the technician were forgiven provided they were one-off. People get distracted and fail to follow testing procedures/forget to read architecture documentation and make mistakes[2].
After the situation is resolved, a post-mortem/"root cause analysis" was always performed. It was often done via documents rather than meetings, to avoid the confrontation associated with a bunch of managers pointing fingers, and we always learned from our mistakes and got better as a result.
[0] I was a developer of infrastructure services, doing similar work that a DevOps might do, today but more focused on providing internal services and automating support. I was a full-time developer for 3/4 of that time with regular interruptions of sev0 incident response.
[1] This didn't always happen, but I can count on one hand the number of times when a director/VP/C-Level joined and actually made his/her presence known.
[2] I'll never forget the time early in my career when I was on-call and had to restore our most critical/largest file server that was destroyed as a result of a technician getting frustrated with a rack and dislodging a 60(?)-pin SCSI cable...partly...where it attempted to run for several hours in that condition. The best part was the maintenance he was doing was on the backup server that had required his attention for a few weeks and hadn't functioned since then. When I saw the condition of the cables, I was so pissed off that while restoring the weeks' old backups at 4:00 AM, I had the operators in that room show me how to review the (VHS) security tapes. Watching him kick the door several times to get it to latch, then peering in -- directly at the bent-up cable -- just before walking out the door was the best part. He should have been fired for that, and eventually a similar behavior had that result.
If you're not already familiar with his work, you should know that Hanno Böck is a machine, and someone worth following. If there's some mistake you can make with the web PKI that is so stupid nobody would ever bother to check for it, rest assured that Hanno will eventually check.
If this is the first time you've heard of CAA records, note that the "dig" command isn't yet aware of these records types, so you need to tell it to use "type257" as the record type:
This inspired me to add a CAA record for my domain - but https://www.namecheap.com doesn't support that record type. Has anyone had better luck other places?
I used your site - it was quick and easy. It soon went downhill from there. I don't run my own DNS or I'd be done pasting the line into my config. Instead I ran into a UI that doesn't support CAA as a type.
Unfortunately, if you want to use CAA your provider must support it. I recommend you use one of the DNS providers on the page I linked - there are quite a few to choose from.
CAA is fresh like IPv10. Done some tracking the last couple of months on the alexa top 1 million domains. In April 400(!) had CAA records. In August 800. 10% have errors (the issuer-critical flag's value is set to 128, 4 6 or 9) where it should be either 0 or 1. iodef set to totally unusable web addressed, you name it.
As a matter of fact there are MANY large DNS service providers that do not even bother providing CAA options. If you want to name and shame, there is ample opportunity for the years to come. Getting it up and running is harder but more rewarding in the end.
70 comments
[ 38.4 ms ] story [ 2266 ms ] threadIf anyone is interested in testing their own CAA records, we built an online CAA validator specifically for this; https://dnsspy.io/labs/caa-validator
On the other hand, they issue a lot of certs - if you've used Cloudflare's free SSL stuff, you've got a Comodo certificate - so they're unlikely to be shut down or anything that extreme.
They're still smaller than Symantec, for whom the nuclear option was considered (and ultimately decided against, in favour of a phased distrusting).
It's just that when trust has been revoked in CAs recently, it's been done in a more gradual fashion to not hurt customers too much and give them time to migrate. After all, it's not the customer's fault the CA went rogue. This usually starts with not accepting certificates past a certain date but still validating previously issued ones, until sometime 6 months to a year later they're completely distrusted.
> Revocation can be done only for new certs.
That's not true. The "newness" of a cert or a CA has absolutely nothing to do with how, if and when it can be revoked.
And yet with all of Comodo's resources, they couldn't get this done? I can't take them seriously.
Any CA that issues certificates publicly need to check CAA from the 8th of September onward.
[1] https://cabforum.org/2017/03/08/ballot-187-make-caa-checking...
CAA was made up by... drumroll.... Comodo.
Yes, check the authors on the RFC: https://tools.ietf.org/html/rfc6844
Highlights include allowing fraudulent certs to be issued for Google, Yahoo, and Mozilla; working with a malware company; and trademark shenanigans with Let's Encrypt.
Why is this a bad thing? A cert says "Yes, person x REALLY IS person x, and I can prove it mathematically." It doesn't say "Person x is trustworthy enough for me to vouch for them."
For malware concerns there's also Comodo's relationship with PrivDog. I'm not clear on whether PrivDog is deliberate malware or just incompetent insecure software.
Do you think we should stop allowing Let's Encrypt to issue certs? You know they don't google each domain to see what they are doing.
The legitimate owner of that domain or operator of the website. That's what certificates are supposed to attest to.
Malware authors might be bad people, but they're definitely people who exist in real life and have an identity that can be attested to.
I can easily see how a missing integration test, and a few code changes later, the feature gets broken and nobody notices it. It doesn't prevent any normal codepath from executing properly.
https://threatpost.com/chromodo-browser-disables-same-origin...
The general rule in Operations work is tolerance. You don't fire someone for making a mistake, you fire them for lying about the mistake, for refusing to avoid mistake-prone behaviors, and other problem of that sort.
Applying that same principle to CA operations, why on earth should Comodo be "fired" as a CA for this? They're actively working to solve it, they're responsive, and the agreements they operate under will compel a post-mortem into existence if only to meet the terms of their next CA audit.
If, after investigation, they are found to be willfully violating policy, knowingly lying about CAA support, fail to produce an acceptable post-mortem, and/or fail their next CA audit, then of course they should be at risk of direct sanctions - not just a verbal reprimand.
So if your initial response to this issue is "Comodo should be fired", remember that in Operations culture that specific response "X should be fired" is a harmful response to a zero-day report of an incident, and many Operations teams would fire anyone who called for a firing in response to a fresh incident, in order to protect others from that harmful posture.
As you say, that's based in part on how they handle it.
An employee at a big company makes a mistake that costs the company $500,000. The client is livid, and demands he be fired. The owner's response? "Why would I fire him? I just spent half a million dollars on training!"
I worked in a hybrid Infrastructure / Development job for almost 17 years. One of the reasons I could never shake the infrastructure part of things[0] was that I knew the environment from more angles than most of the staff and I tended to be key in high severity issues. We were a highly geographically distributed team and coordinated almost all activity via conference calls.
We had two major rules when a critical failure was discovered. The first was that anyone asking "who caused this" was admonished and often ejected from the call. Often, the person responsible for the failure was on the calls and everyone had a pretty good understanding of who was at fault (if it was, indeed, a self-inflicted wound). The person at fault was also usually critical to the fix and putting them on the defensive caused evasiveness out of self-preservation. To help the first rule, the second rule was barring of anyone at the Director level or higher on these calls[1]. Generally, they were concerned with two things: (1) When's it going to work again and (2) why did it fail in the first place -- both result in delays. They were given updates, separately, by someone from the call at regular intervals. Keeping the politics low and focusing on the problem meant much shorter outages and calmer, more focused, brains of our technical staff.
And to your point, despite working at a company that laid off 10% or so of it's staff about every 9 months, I can't think of a case where a member of technical staff was specifically let go because of an (infrequent) failure -- even situations that would fall under gross negligence of the technician were forgiven provided they were one-off. People get distracted and fail to follow testing procedures/forget to read architecture documentation and make mistakes[2].
After the situation is resolved, a post-mortem/"root cause analysis" was always performed. It was often done via documents rather than meetings, to avoid the confrontation associated with a bunch of managers pointing fingers, and we always learned from our mistakes and got better as a result.
[0] I was a developer of infrastructure services, doing similar work that a DevOps might do, today but more focused on providing internal services and automating support. I was a full-time developer for 3/4 of that time with regular interruptions of sev0 incident response.
[1] This didn't always happen, but I can count on one hand the number of times when a director/VP/C-Level joined and actually made his/her presence known.
[2] I'll never forget the time early in my career when I was on-call and had to restore our most critical/largest file server that was destroyed as a result of a technician getting frustrated with a rack and dislodging a 60(?)-pin SCSI cable...partly...where it attempted to run for several hours in that condition. The best part was the maintenance he was doing was on the backup server that had required his attention for a few weeks and hadn't functioned since then. When I saw the condition of the cables, I was so pissed off that while restoring the weeks' old backups at 4:00 AM, I had the operators in that room show me how to review the (VHS) security tapes. Watching him kick the door several times to get it to latch, then peering in -- directly at the bent-up cable -- just before walking out the door was the best part. He should have been fired for that, and eventually a similar behavior had that result.
https://support.dnsimple.com/articles/caa-record/
https://caatestsuite.com/
Are you certain?
https://datatracker.ietf.org/doc/rfc6844/?include_text=1
> The data fields are defined as follows:
>
> Flags: One octet containing the following fields:
>
> Bit 0, Issuer Critical Flag: If the value is set to '1', the
> critical flag is asserted and the property MUST be understood
> if the CAA record is to be correctly processed by a certificate
> issuer.
>
> A Certification Authority MUST NOT issue certificates for any
> Domain that contains a CAA critical property for an unknown or
> unsupported property tag that for which the issuer critical
> flag is set.
>
> Note that according to the conventions set out in [RFC1035], bit 0
> is the Most Significant Bit and bit 7 is the Least Significant
> Bit. Thus, the Flags value 1 means that bit 7 is set while a value
> of 128 means that bit 0 is set according to this convention.
EDIT: Fixed formatting? I hope.