I don't think Google was soliciting for a counter proposal from Symantec. Will be interesting to see their reply, and whether it's a literal reply or just a version push of chrome with their original plan.[1]
"This site does not support Chrome. Please use a browser that does not take unilateral CA authority action." might very well be the response of orgs married to Symantec.
As a user, you need your bank (or other large org) more than you need your preference of browser.
well, for any organization that is doing serious ecommerce these days, you would want an EV SSL cert. Which is about $95/year from namecheap or a competitor. The GUI advantages for ordinary totally ignorant end users of seeing the happy green bar at the top are enough to justify the less than $10 per month ongoing cost.
In this context, I assume we're talking about EV certs. Though LE might be an emergency temp option for the big customers that forgot to plan around this.
Ah, yes. If you didn't have a backup cert from another provider and missed your 60 day window. But what was your plan for any other incident that required revocation? I assume anyone turning on pinning hand wrings over it for a while first.
That's the problem, a lot of Big Corps thought Symantec is okay, because it's (was?) expensive as fuck and it has a nice long history! (So reputation. They're on NASDAQ, they must be okay!)
But that's just business as usual. You never know when your vendor goes rouge. And really Big Corps should have plans for these events.
Have local credit unions gotten better at this? I left a local credit union a decade ago because they didn't offer online banking at all and when they finally did, the security practices they used were horrible. Memory may not serve perfectly here but I recall they had no EV cert (while common with banks, not everyone had them back then), the login form was provided in-the-clear POSTing to HTTPS and many of the banking functions were issued via GET requests, causing certain "internet accelerator extensions" that worked by requesting data before you clicked them to actually make changes to your accounts on certain pages (this is where memory doesn't serve -- I think it was a "theoretical" problem because these extensions usually didn't pre-request HTTPS pages for this very reason).
It depends upon the CU in question. There are typically /many/ local (to a state/region) credit unions, so you get multiple shots at this. SECOND, the reason you're better off is that you are literally part owner of that 'union' as a member. They inherently (typically) give you better rates and treat you better.
Mozilla seems in total agreement with Chrome on this, and it also takes unilateral CA authority action. This is not a negotiation where the orgs have the right side of the power dynamic.
It's always a negotiation. He who has the least to lose wins.
"Why isn't my browser working??" "So sorry about that, those darn nerds made a change on us."
I have seen this exact scenario play out. Mozilla and Google might be respected in the tech community; to most people they're an extremely tiny part of their life. "Your the IT man, just make it work"
i would say the relevant "this is how it works" is having a globally trusted CA system that isn't abused by large CAs just because they think no one will dare punish them.
What would any of these large Symantec customers says if in the future their security was compromised due to mis-issued certificates? I doubt they'd be thrilled they were saved some dev time getting their certificates signed by another root in exchange.
it's very clear from Symantec's language here and elsewhere that they wouldn't be doing shit about this is chrome weren't taking it so seriously
And when Google and Moz both are "blocking a security threat" MS and Apple will have follow suit or they'll get buried by by PR and advertising attacks.
I disagree, a bank should especially be knowledgeable of security (and certificate) issues. I've switched banks because of bad security practices before, it's a simple matter of voting with your feet.
Given some of the internal CA systems I've dealt within the past, I'd almost prefer a public CA in some cases. Sometimes your internal CA is just the group with manual access to the certificate provisioning and signing systems with either no API or some awful re-implemented API.
What API do you need? The signing system should be airgapped or you end up with the same shit that is the public CA system such as roots sitting on public FTP servers.
It's a bunch of command line scripts because if you are using it any different way you are probably doing it wrong.
I'm sure people make plenty of money wrapping OpenSSL in ever more elaborate Enterprise schemes.
But if the net effect is that one bug in a frontend can cause the server to tell the cert server to tell the secure enclave to tell the wire mesh protected crypto coprocessor to sign a bogus certificate then what are we really doing here.
They sure do! We have an offline root on an HSM for various reasons.
We looked at various systems to manage intermediate ca issuance and reminders for renewal and the cost was incredible... most were >$200k!
Using scripts to automatically generate certificates and using the ugly but effective Windows CA web interface for "vanity" stuff does 90% of what those solutions do. I think we had an intern write a monitoring plugin to open tickets for soon to expire certificates and a few other things that covered the remaining 10%.
I know several companies are using Amazon's ACM and public domains that resolve to internal ip addresses. It's not 100% automated, as it requires a validation email, but better than most homegrown stuff.
There's a massive logical leap between having an API and having certificates on public FTP servers.
An API is required for you to have proper infrastructure as code and leaving any bit to human input other than another validation step in the automated process is waiting for errors to happen (invalid or incorrectly spelled cert info, missing cert chains, wrong cert chains, etc.).
Symantec even calls out how much work these companies will have to do because so few of them have a proper certificate management system in place that's come back to bite them.
There's more to it than you've seen so far. Think about environments with hosts starting on demand. You need to encrypt data in flight -> how do you automatically assign a new certificate to an instance on startup? You can't do it with an airgapped system.
The topic was internal CAs. Those have completely different requirements and possibilities than public ones. The authentication will look completely different as well. You just can't compare those issues.
And you're missing the point I raised as well - if you have hosts coming up automatically, you either need automatic certificates, or shared keys. As long as you rely on SSL, there's no other way to go.
(For example of authentication: public CA verifies your domain, internal CA can ask the hypervisor to verify a token on your instance - not remotely in the same class)
I'm curious if people other than LE have tried deploying Boulder in production, now that it exists. It seems like probably close to what a public CA wants.
Ideally you'd have some way of tying it into some internal authorization database instead of relying on HTTP challenges, but HTTP challenges would work too.
The are some very-not-enterprisey internal solutions. For example there's Anchor from the OpenStack project with a minimalistic approach (as long as you're cool with the idea of rotating certs daily)
I find it hard to believe you can't imagine any valid scenario where at least one of those scenarios shouldn't be using pubic CAs.
I've got personal experience with the first two (embedded devices and mobile apps) having pinned certificates (or limited sets of supported CAs)
There's millions of internet-connected set top boxes, tvs, and dvd/bluray players deployed around the world that run with a limited set of supported CAs. (e: Also, don't forget the tons of 'smart' phones out there running ancient versions of whatever)
If you're someone who supports applications that run on those, then your choices for what CAs you offer on your API/Media/etc endpoints is extremely limited. Some of these decisions were made more than a decade ago, and migrating off those is going to take another decade (i.e as old hardware dies).
> There's millions of internet-connected set top boxes, tvs, and dvd/bluray players deployed around the world that run with a limited set of supported CAs.
What exactly is the issue? If an embedded device trusts Symantec CA, then connections from the device to its central servers will continue working without interruption.
In this example, the device trusts the Symantec CA, and the server presents a certificate signed by the Symantec CA. Everything's hunky-dory. What's the problem?
Clients have no way to indicate what CAs they trust, so my infrastructure has to just present a certificate and say "Here I am"
So now if Google go ahead and distrust Symantec CAs, I need to duplicate that infrstructure (or at least the configuration at load balancers, etc) - this leads to all sorts of complexity.
You have api.mysite.com which serves both an embedded device as well as your site. To allow the site to be viewed in Chrome, you need a new cert for your API, and so you now have to either update your site to point to a different endpoint, or update your devices to accept the new cert. Sure it might be easier to do the former for some companies, but either way there's certainly a cost.
Sure. If api.mysite.com supports both web browsers and embedded devices, then you can issue a certificate for it your publicly-trusted CA of choice, and then can cross-sign that certificate with the Symantec CA.
If you control the client and you control the servers it needs to communicate with, then you should probably not be using the public CA system.
Well yes if your "embedded" device has a user-facing webbrowser you will need the public CA system, but then I'd wager if you shipped such a device and don't have ongoing updates you have much much bigger problems than an outdated set of root CAs.
But there is zero reason for your mobile app communicating exclusively with your API endpoint to use the public CA system. You're paying for a less secure system with all the same hassles.
>require these applications to be recoded, recompiled and redistributed.
Aka "updated".
The entire post is basically "ok how about we be really good from now on and suffer no consequences, cause it'd be really shitty for us if we had to be penalised".
They also posture a lot talking about how big their customers are, almost boasting about how inflexible and slow these big companies are, as if that's somehow Google's or general Internet users' problem.
Isn't that largely Google's stance with regard to Android? They blame the carriers and point to how slow they are, meanwhile tons of smart phones go without patching, which is no good for the general internet or users'
No, there's a huge difference. The difference here is that the Manufacturers and the Carriers are gatekeepers to the customer's phones.
The manufacturer made customizations to Android for their phone. Android isn't a cleanly-separated stack where the manufacturer modifies only a certain part of the stack for their phone but Google could go and update another part without risk of breaking anything (is any software project?)
Then, the manufacturers sell the phones to carriers, who usually request extra customizations to (at best) add their logo, or replace default apps with the carrier's own version. This means that any updates to Android likely require updates to this customization.
But even if that customization was free (from a development perspective), Carriers still have maintenance/service contracts with the manufacturers such that any manufacturer's updates must go through the carrier's "quality control" procedure. You wouldn't want an update to make the, say, LTE radio misbehave and break the carrier's network would you? Oh no... (and you certainly wouldn't want to "forget" to replace the good default app with the Carrier's crappy one)
So there you go, a quick glimpse into the complex economic/contractual obligations that delays or blocks Android updates. FWIW, Google has been working both technically and contractually to untangle this mess, and is one reason for the existence of their "own" line of Nexus and now Pixel phones.
Google holds all the cards. They have to sign off, or the phones will lose access to the Play Store and Google services like Maps. Google only needs to play the cards it already holds.
The networks have to sign off too, or the phones won't get the OTA updates or be allowed to connect to the network.
The SoC / OEMs also have to sign off because they are the ones with access to the driver source code which needs to be updated and recompiled against a new release.
If Google plays hardball, these parties can always threaten Google with "We're leaving the OHA; we will be going with Amazon's Android fork and bundle Kindle and Prime apps." I'm sure Bezos will throw in Amazon Maps for free.
I'm constantly amazed by how some people think Google has this magical ability to update an OS that was built by another OEM. That's like asking Debian to update Ubuntu.
So let's say they put in a rule that said all OEM's must update their devices every month or lose access to Google Play apps and services. What happens when the carriers start delaying or preventing the patches and updates from flowing to the devices? Does Google uphold the rule and revoke their access? Or do they start extending the time by 1 month? Or how about 2 months? Or what about 6 months? What's the point of adding a rule you can't enforce?
In the case of the carrier preventing the transmission of patches or OS upgrades you would be punishing the OEM for a problem created by the carrier and out of the control of the OEM. Tangentially, If I was a carrier I would use this to my advantage when negotiating. If you thought carriers were bad now, just wait until they have the power to withhold Google Play services and apps to an OEM's future phones.
Google could selectively disable play services with a very scary warning. Simply, if you are on a very old version of Android, display an alert that tells the end user to complain at the carrier.
There's an argument that it's good for end user security to force mobile apps to be updated, even if there weren't a specific reason to distrust Symantec's roots, because who knows when the next Heartbleed will come out.
I suppose that's true for mobile apps with embedded certs. Bet there's an app store / play store approval logjam for Symantec customers that miss this news, and all come crashing in after the expiry.
1) The point of the first part of their post is not "we're too big", it's "a move like this would disrupt the lives of an awful lot of people." Even more so, consider what would happen if Google were to update Chrome to not accept these certs. For internal applications, the IT departments of all these companies—which likely total a few hundreds of thousands of users in total—would simply mandate that Chrome is no longer an acceptable browser. For external applications, companies would have to engage in an enormous PR effort to communicate to customers that their product doesn't work with Chrome. These companies have done nothing wrong, but they're the one being punished, as customers don't know and don't care about cert issues.
2) "Updating" can be an enormous effort. As the blog post states explicitly, there are a lot of use cases where updating a deployed app is very difficult, almost impossible, or cost prohibitive. Again, remember that the person being punished here is not Symantec; you're publishing some poor development company who happened to use Symantec—one of the most popular cert providers on the planet—for an SSL cert. Not good.
I'm not taking sides on the issue, because I honestly don't know that much about it. But comments like the above really trivialize the huge impact that something like this will have.
>) The point of the first part of their post is not "we're too big", it's "a move like this would disrupt the lives of an awful lot of people."
not doing something is allowing the previous lack of security standards to put all of those people in danger. there is a tradeoff with every decision.
the entire PKI ecoysystem relies on self management and I think google is making a tough decision, but one that ultimately has already led to better decisions regarding security.
> not doing something is allowing the previous lack of security standards to put all of those people in danger.
So glad others are making this point. Not to mention that the CA system is one who's sole feature and only purpose is trust. There is zero security if you can't trust the root. We'd love to live in a world where providing security didn't require a trusted third-party, but that's not the world the internet operates on, today. Transparency is provided by establishing strict guidelines and expectations for CAs. Maintaining this trust is enforcing those guidelines equally among the CAs regardless of size/scope and punishing (preferably severely) any violations. Otherwise the old saying of "rules that are not enforced are just suggestions" comes into play.
Because it's a trust-based model, the consequences for violations of that trust have to be strong enough to make a compromised actor willing to refuse government intelligence divisions, corporate criminal actors or even organized crime that has the resources and could stoop to any level of blackmail/bribery to squeeze out a certificate that would allow one to spoof a connection to a high-value target like Amazon, Google, Paypal or Microsoft.
Part of the premise of Google's plan is that Symantec's customers will experience disruption. If you poll Symantec's customers, the overwhelming majority of them will day "no, don't punish our CA". We didn't need Symantec to tell us that.
On the other hand, Symantec's CA was caught mis-issuing certificates. That affects everyone, not just Symantec's own customers.
The "survey" that accompanied the original customer facing message was so biased as to be ridiculous. Like "Will Google's proposed change be disruptive, massively disruptive, or business ending for you?"
> The point of the first part of their post is not "we're too big", it's "a move like this would disrupt the lives of an awful lot of people."
The real question is, what effect would doing nothing have on all of those people? If Symantec continues to mis-issue certificates (as they have done repeatedly: https://wiki.mozilla.org/CA:Symantec_Issues), keeping their EV status intact could be much, much worse for them.
> As the blog post states explicitly, there are a lot of use cases where updating a deployed app is very difficult, almost impossible, or cost prohibitive.
As others have pointed out, this change would actually have no effect on the 'complex dependencies' Symantec identified:
* This has absolutely no effect on embedded devices, which are certainly not running Chrome.
* Mobile applications that pin certificates shouldn't be affected, since the very fact that they're pinning certificates means they're not using Chrome (there's no way for an app to tell Chrome to pin a specific certificate).
Not completely true. There's nothing to prevent customers to obtain a new certificate from a reputable CA while keeping the old one around for the embedded devices to use.
Symantec should have thought about this before using half ass processes and half ass people in a business that will cost other people money because of their screw ups.
Unless your software is running on a 30 year old embedded system, located in a submarine, 5KM below the ocean, guarded by tigers, and stuck there for the next decade, it's hard to muster sympathy about how hard updating your software is. If the cost to a company to deploy an update their software is really greater than the cost to just allowing the software fail, then maybe they should just let the software fail and prepare a great story for their users--I'd love to hear it.
would simply refusing to recognize NEW symantec certificates issued after a certain date in the near future be a sufficient compromise? it won't affect institutions that have pre-existing certificates, which will eventually expire by themselves.
Again, how is this anyone else's problem? It'd be rewarding bad behaviour and poor technical solutions. These companies will find a way to workaround it.
A suitable compromise could be an override switch that can only be activated via Group Policy. That'd at least be a reasonable suggestion from Symantec. Not their current offer of nothing.
As far as "updating" my point was that they described a routine mobile app update as "recoding" as if it'd be lots of actual code to rewrite. And again, the fact a large company has no way to easily update in face of a security problem really is their problem, not ours.
Sometimes doing nothing is the wrong thing. They did wrong by supporting Symantec (or whatever shitty CA they originally did business with that Symantec acquired) with their money.
Or if initially they did not know about how shitty Symantec is, now they ought to know, and yes, it's a business risk (counterparty/vendor risk basically).
> The point of the first part of their post is not "we're too big", it's "a move like this would disrupt the lives of an awful lot of people."
Those are precisely equivalent.
> Again, remember that the person being punished here is not Symantec;
Yes, it is, though you make exactly the standard "too big to fail" argument. yes, customers who relied on the misbehaving CA will also incur some costs, but if misbehaving CAs don't see reduced or eliminated trust at the level of browsers and similar trust stores, the entire ecosystem -- including the good actors and their customers -- pays the price as end-user trust in the ecosystem weakens because certificates do not guarantee what they are supposed to. And if any actor is recognized as too big to fail, that creates an unlimited license for bad behavior on the part of that actor at the price of the whole ecosystem.
For some reason, technical professionals associated with large organizations seem to actively enjoy pretending that basic upkeep is beyond the means of their organization (or has a poor value proposition). I suspect that it comes from a perverse enjoyment of being "pragmatic" in the face of "idealism" but really they're just saying "My org refuses to pay the costs associated with correctly operating the technologies it uses". If you're going to take that attitude, you need to expect that your bluff is going to be called from time to time and you'll have to do some large, unscheduled projects when it happens... that's what being pragmatic actually means: recognizing the positive and negative effects of your options and owning the whole package.
I recall a lot of horror stories from mobile telco providers when the US government proposed to users to take their phone number from one carrier to the next. It was just one terrible prediction after another that involved so many risks and complications.... until they had to do it and then it just worked.
I worked in telecoms at the time. It was a huge pain in the ass, but in the end it was fixable. The company I worked for actually made a nice chunk providing the HLR (home location register) updates needed to get it to work.
It's especially irksome to me in this case because Symantec (and maybe some of their actual customers) are acting like this is unheard of. Services did just recently have to update their certs to move away from SHA1 intermediate CA cert, didn't they?
The current Symantec SHA1 intermediate CA cert is valid between Feb 2010 and Feb 2020, so it seems like there was probably a cert change ~7 years ago too. The horror!
Symantec has got to be facing some very large and very serious lawsuits from those very same giant companies they're try to leverage.
This is happening because Symantec was grossly negligent. They're going to be held to account for that.
All those giant companies should be firing up their lawyers and starting emergency procedures to change to a different provider. Anyone still using Symantec at this point is absolutely playing with fire.
There is a significant amount of "we will" language in here, rather than "we have started" or "have already begun". The choice of language speaks volumes that they only appear to be willing to take necessary audit action if Google agrees to it, rather than something that needs to be done regardless.
> This cohort is an important constituency that we believe has been under-represented to date in the public commentary that has been posted to the Google and Mozilla boards since large organizations rarely authorize employees to engage in such public discussions, particularly in an area related to security.
Are these large organizations somehow incapable of putting out official statements regarding CAs? If they're being 'under-represented', it's their own fault for not speaking up.
To me it was really bold of them to take the tone and stance that they did with regard to these large organizations. They're making the case as to why Google needed to do what it did ... all of these large organizations that will be severely impacted by having to replace all of these certificates are relying on a CA that has shown itself to not be worthy of the trust that CAs are relied upon to provide.
I'm at a loss for a good analogy so I'll pick a lousy one: it'd be like if Samsung's response to the exploding Note 7 recall mandate were "Don't do this -- you're going to keep some of the biggest customers -- users of our phones -- from being able to make emergency calls/check their e-mail/respond to texts" ...yeah, but I'm still not OK with the small possibility that carrying my phone in my front pocket might result in third-degree genital burns.
> These customers include many of the largest financial services, critical infrastructure, retail and healthcare organizations in the world, as well as many government agencies. This cohort is an important constituency that we believe has been under-represented to date in the public commentary that has been posted to the Google and Mozilla boards since large organizations rarely authorize employees to engage in such public discussions, particularly in an area related to security.
... well that's their problem, right?
You can't simultaneously say "These are some of the most important organizations in the world and you'll cause worldwide chaos" and "Won't someone listen to these poor companies, I am the Symantorax, I speak for the cohort, for the cohort has no tongues."
> These are some of the most important organizations in the world and you'll cause worldwide chaos
Thanks for that -- reading this comment connected a few thoughts I was struggling with.
Their response felt like "if you don't follow through with your plan to punish us we'll do all of these nice things for you." It reminds me of when my kids get caught doing something wrong and I get "but I'll do all of my chores and treat my siblings nice" along with a handful of other promises of future deeds which I know will be partially or completely not-done. How different the result would have been if they came to me with the chores done and some/all of the promises fulfilled already. It would have been much better for Symantec to have responded with these items in the "completed" category with a smaller number of "we will's".
But to the quote I highlighted: the issue at hand is the past failures indicate that "some of the most important organizations in the world" need to not be trusting Symantec in the first place. They've had a big enough history of failures as a CA to get the attention of Google, who I'm sure isn't terribly interested in degrading the browsing experience of their customers. The choice becomes: (1) Banhammer Symantec, take the resulting pain all at once, visibly and obviously, and get the problem fixed painfully (but it's fixed) or (2) Trust that things will be different this time and if they're not, have another security failure of some kind that results in these critically important company's customers' data being exposed, which is a circumstance that will almost certainly go unnoticed for a period of time, will be covered up if the exposure is minimal and covering up the exposure is possible or may not be discovered at all. The CA system is centered around trusting a handful of rather powerful bodies with an enormous responsibility around the overall security of the Internet. It's not a great model to begin with, but it's what we have and I haven't read about any alternatives that are viable. The only way to keep a model like this secure is to ensure that the CAs themselves are actually trustworthy and one of the more likely ways that will happen is to enforce violations of that trust in the manner Google is suggesting.
Yup. It's very interesting that the argument from Symantec is that their customers have to buy Symantec certs indefinitely - not that they can't replace existing certs in servers, but they've hardcoded the Symantec root in enough clients, such that buying certs from a Symantec competitor will break their clients.
Among other things, if what Symantec is saying is true, this means there is zero financial incentive for Symantec to behave. The largest certificate customers in the world have all voluntarily made it infeasible to choose a Symantec competitor, so as long as Symantec has any product at all of any quality, the market will keep buying Symantec.
If what Symantec is saying is true, the mere fact that they went to their largest customers and said "Suppose you had to buy certs from one of our competitors, what would you do" and they replied "We've never thought about that before" is itself a major victory.
I hadn't read it that way and that's a really good point.
If you have too many trusted-third parties in a trust model like this, you run into difficulties with enforcement due to scaling problems, but what we're seeing is the converse of that. Symantec went out and bought up other CAs (Verisign is the only one I can remember for some reason) and they've become so large that them losing CA status would be prohibitively damaging to the internet-at-large. This affects not only enforcement (we don't want to move on a bad actor because the consequences of enforcement would be more severe than the consequences of a rule violation), it also negatively affects the trust model -- now if I'm a bad actor, Symantec is my only target -- even though I could attack the others and yield some of the same benefits of attacking Symantec, if I breach Symantec, the rewards are larger by several orders of magnitude depending on what they store and how they store it.
To digress a little ... today ... I'm not all that surprised that Symantec has reached sort-of monopoly status in this area. Not because Symantec is a well managed company (I'm making no statement for or against that despite my personal opinions), but just because so much of The Internet has turned out this way. If you would have told me in 1998 that for the very vast majority of US Internet users, there'd be one search engine, one e-commerce company, one (expensive) ISP, one e-mail provider, all of it would be secured by services provided by one company's CA and all of that wouldn't matter much because the typical Internet experience would basically be a web-enabled, bluer version of AOL[0], I'd have called you some unfriendly things, but that's where we are. It's a centralized system operating on decentralized technology.
[0] Oh, and that the "one e-commerce company" was that bookstore web-site and would be responsible for hosting such a large part of the rest of the Internet that when they had a major failure and people had that handful of rare clicks off of FAOLBook, they'd have about a 50% chance that whatever their target was wouldn't load.
Symantec is just going through the 5 stages of CA grief. First they were oblivious to it, then they were angry about it and called Google irresponsible. They're now at the proposal stage. Next will be depression and finally acceptance for their incompetence.
> Embedded devices that are pinned to certificates issued by a Symantec public root to communicate to resources over the Internet or Intranet. Replacing these certificates would result in immediate failures and the need to recode and reimage the firmware for these devices.
> Mobile applications that have pinned certificates. Replacing server certificates would require these applications to be recoded, recompiled and redistributed.
Are either of these relevant? Isn't google's proposal to eventually have Chrome stop trusting the Symantec CA system? That change seems like it would have no effect on 1) embedded devices that aren't running Chrome 2) things that already are in place that trust the CA.
It was either a bizarre misunderstanding on what will happen on Symantec's side, or "pinned" was used not in the technical meaning.
I could see someone translating "these devices use Symantec's server certs and serve web traffic" to "these devices are pinned to certs issued by Symantec". But if that's the case, they should pay more attention in announcements like this one...
I think the point there is that if Chrome doesn't trust that CA, then the servers will need to be updated to use a different CA, which will break clients that pinned the old CA.
Iirc you can have certs signed by multiple roots though, right? In which case these sites can just get their cert signed by the pinned symantec root and a real still-valid root. Nobody's stopping symantec from signing new certs, they just won't work in Chrome (and likely other browsers)
Unfortunately, you can't keep multiple chains in a single X.509 certificate. While you can have a given key signed by multiple CAs, AIUI, this just lets you switch easily, not tell a client about both, and just work if they trust either CA.
Huh, okay. Surprised that's not possible. IIRC letsencrypt did something like this to bootstrap but I guess you can have a root sign a root, but not have two roots sign something and have both in the same cert.
I've always been of the opinion that the rogue/careless CA problem can be solved by mandating N root sigs and recommending M>N. In a system with multiple independent LetsEncrypt like entities this is doable. If a CA needs to be revoked you can do it in a much easier way here, since it's still ok to transitively allow sites with N-1 certs and one now-disallowed cert signed by that CA. Doesn't fix all the problems but it certainly removes the inertia behind kicking out bad CAs.
Looks like we'd need a x509 extension to do this then. :/
If the embedded devices are accessed over the net by Chrome, or access services that also need to be accessed by Chrome, then there is an issue.
But I don't think Google is going to be swayed by "you can't distrust us if we do wrong because too much depends on us" because if they do, it encourages laxness at big CAs.
Imagine the embedded device or mobile app connects to a https://api.example.com/... and so does their web app. They need to use a new certificate for api.example.com, signed by a different CA root, because when Chrome tries to run the web app it won't trust the old certificate, nor a new cert signed by the old root. So they get a new cert signed by a different root so Chrome works, but then the embedded or mobile apps stop working.
In this case the story is more complicated - I think new certificates signed by existing Symantec CA roots would be trusted if their duration is 90 days or less. Historically, most CAs offered one, two, or three year duration. Presumably, the "big customers" would not accept the need to rotate their certificates for this purpose in less than 90 days, they need their 3 years because it takes over 90 days to get something through their bureaucracy. And presumably there are other things they can't do in 90 days, like update an embedded device configuration or firmware, or a mobile app.
Of course it depends on the particulars, but of course these "big customers" can't figure any particulars out in 90 days. But that's why Google proposed a slow sliding scale that would gradually reduce the max duration allowed over the course of a year, so random unknown stuff breaks more gradually throughout the year, not in one huge event like Symantec's blog post suggests.
An easier way around this sort of issue is to change their web app to use a different domain, and only put the new certificate from a different CA on that domain. It's not necessary to update every device.
Another approach is that if the devices are old enough they're probably using a deprecated version of TLS, so the existing https://api.example.com server could choose which certificate to issue based on the version of TLS offered by the client.
You have to have planned for this (or be lucky) though. Some people just put their api on www.example.com; assuming you have clients in the field which have pinned BIG CA, and you've got HSTS preloaded (because you're forward thinking). Unless you can detect Chrome N+1 from your existing clients during the TLS handshake, you have to pick between browsers or existing clients.
If Chrome ships this at the same time as re-enabling TLS 1.3 support, that would be a decent way to target, since presumably very few of your distributed clients are using TLS 1.3.
True, but you know how it is ... the management of these companies would prefer to do nothing if it is possible. They'll put most of their efforts into that strategy.
Does anyone else get a blocking "Symantec Connect" and "Loading your community experience" for a good 10 seconds before it loads? Could just be my mobile connection but man is that lame.
This is how we make sure issuing certificates for gmail.com, etc will never happen again. An external audit every 3 months is not going to fix anything.
CA infrastructure business is built on trust. Symantec lost the trust and got thrown out by Chrome. I hope that as the result Symantec will lose significant amount of CA business thus making a show case for other CAs to demonstrate that good processes and trust are key for staying in this business.
I'm still trying to figure out if my 36 month wildcard carts from RapidSSL are going to be distrusted. Their intermediate is signed by GeoTrust which is owned by Symantec, and a blog post says with Chrome 59, certs with a period over 33 months will be distrusted. Chrome Canary on v60 shows them still working.
Read the statement carefully. Symantec still does not offer a verifiable audit trail like Google's Certificate Transparency requires. They just offer to have 3rd parties review the certificates sold to customers. Without a COMPLETE audit trail Symantec can still continue to secretly issue additional certificates not part of any audit trail. The whole announcement rather decreases my trust in Symantec even more.
128 comments
[ 2.8 ms ] story [ 214 ms ] thread[1] https://groups.google.com/a/chromium.org/forum/m/#!topic/bli...
Edit: They did ask for community feedback, comments on risk, etc. But they do already have a timeline. See link above.
As a user, you need your bank (or other large org) more than you need your preference of browser.
Technically-correct well-actuallys lose to real-world perception ninety-nine percent of the time.
I think the world is changing in ways which may cause the affected organizations to reconsider what they consider an acceptable CA solution.
But that's just business as usual. You never know when your vendor goes rouge. And really Big Corps should have plans for these events.
For a bank maybe a user would temporarily accommodate but it would affect online stores and SaaS business a lot.
Works for me without Javascript.
"Why isn't my browser working??" "So sorry about that, those darn nerds made a change on us."
I have seen this exact scenario play out. Mozilla and Google might be respected in the tech community; to most people they're an extremely tiny part of their life. "Your the IT man, just make it work"
As you get older, you transition from "this is how it should be!" to "this is how it works, what is a realistic amount of change that can occur?"
It's a bunch of command line scripts because if you are using it any different way you are probably doing it wrong.
But if the net effect is that one bug in a frontend can cause the server to tell the cert server to tell the secure enclave to tell the wire mesh protected crypto coprocessor to sign a bogus certificate then what are we really doing here.
We looked at various systems to manage intermediate ca issuance and reminders for renewal and the cost was incredible... most were >$200k!
Using scripts to automatically generate certificates and using the ugly but effective Windows CA web interface for "vanity" stuff does 90% of what those solutions do. I think we had an intern write a monitoring plugin to open tickets for soon to expire certificates and a few other things that covered the remaining 10%.
An API is required for you to have proper infrastructure as code and leaving any bit to human input other than another validation step in the automated process is waiting for errors to happen (invalid or incorrectly spelled cert info, missing cert chains, wrong cert chains, etc.).
Symantec even calls out how much work these companies will have to do because so few of them have a proper certificate management system in place that's come back to bite them.
This is the exact issue with the public CA system, their automated processes have been found lacking and now what they certify can not be trusted.
And you're missing the point I raised as well - if you have hosts coming up automatically, you either need automatic certificates, or shared keys. As long as you rely on SSL, there's no other way to go.
(For example of authentication: public CA verifies your domain, internal CA can ask the hypervisor to verify a token on your instance - not remotely in the same class)
Ideally you'd have some way of tying it into some internal authorization database instead of relying on HTTP challenges, but HTTP challenges would work too.
https://github.com/letsencrypt/boulder
I've got personal experience with the first two (embedded devices and mobile apps) having pinned certificates (or limited sets of supported CAs)
There's millions of internet-connected set top boxes, tvs, and dvd/bluray players deployed around the world that run with a limited set of supported CAs. (e: Also, don't forget the tons of 'smart' phones out there running ancient versions of whatever)
If you're someone who supports applications that run on those, then your choices for what CAs you offer on your API/Media/etc endpoints is extremely limited. Some of these decisions were made more than a decade ago, and migrating off those is going to take another decade (i.e as old hardware dies).
What exactly is the issue? If an embedded device trusts Symantec CA, then connections from the device to its central servers will continue working without interruption.
In this example, the device trusts the Symantec CA, and the server presents a certificate signed by the Symantec CA. Everything's hunky-dory. What's the problem?
So now if Google go ahead and distrust Symantec CAs, I need to duplicate that infrstructure (or at least the configuration at load balancers, etc) - this leads to all sorts of complexity.
Well yes if your "embedded" device has a user-facing webbrowser you will need the public CA system, but then I'd wager if you shipped such a device and don't have ongoing updates you have much much bigger problems than an outdated set of root CAs.
But there is zero reason for your mobile app communicating exclusively with your API endpoint to use the public CA system. You're paying for a less secure system with all the same hassles.
Aka "updated".
The entire post is basically "ok how about we be really good from now on and suffer no consequences, cause it'd be really shitty for us if we had to be penalised".
They also posture a lot talking about how big their customers are, almost boasting about how inflexible and slow these big companies are, as if that's somehow Google's or general Internet users' problem.
The manufacturer made customizations to Android for their phone. Android isn't a cleanly-separated stack where the manufacturer modifies only a certain part of the stack for their phone but Google could go and update another part without risk of breaking anything (is any software project?)
Then, the manufacturers sell the phones to carriers, who usually request extra customizations to (at best) add their logo, or replace default apps with the carrier's own version. This means that any updates to Android likely require updates to this customization.
But even if that customization was free (from a development perspective), Carriers still have maintenance/service contracts with the manufacturers such that any manufacturer's updates must go through the carrier's "quality control" procedure. You wouldn't want an update to make the, say, LTE radio misbehave and break the carrier's network would you? Oh no... (and you certainly wouldn't want to "forget" to replace the good default app with the Carrier's crappy one)
So there you go, a quick glimpse into the complex economic/contractual obligations that delays or blocks Android updates. FWIW, Google has been working both technically and contractually to untangle this mess, and is one reason for the existence of their "own" line of Nexus and now Pixel phones.
The SoC / OEMs also have to sign off because they are the ones with access to the driver source code which needs to be updated and recompiled against a new release.
If Google plays hardball, these parties can always threaten Google with "We're leaving the OHA; we will be going with Amazon's Android fork and bundle Kindle and Prime apps." I'm sure Bezos will throw in Amazon Maps for free.
Edit: Yes, really. https://arstechnica.com/gadgets/2014/02/new-android-oem-lice...
The CA system is composed of alow-moving companies like Symantec dependent on, or more specifically, beholden to fast-moving companies like Google.
In the mobile space, the situation is inverted. Android can move quickly, but they have little to no leverage over the slow-moving carriers.
I suppose that's true for mobile apps with embedded certs. Bet there's an app store / play store approval logjam for Symantec customers that miss this news, and all come crashing in after the expiry.
1) The point of the first part of their post is not "we're too big", it's "a move like this would disrupt the lives of an awful lot of people." Even more so, consider what would happen if Google were to update Chrome to not accept these certs. For internal applications, the IT departments of all these companies—which likely total a few hundreds of thousands of users in total—would simply mandate that Chrome is no longer an acceptable browser. For external applications, companies would have to engage in an enormous PR effort to communicate to customers that their product doesn't work with Chrome. These companies have done nothing wrong, but they're the one being punished, as customers don't know and don't care about cert issues.
2) "Updating" can be an enormous effort. As the blog post states explicitly, there are a lot of use cases where updating a deployed app is very difficult, almost impossible, or cost prohibitive. Again, remember that the person being punished here is not Symantec; you're publishing some poor development company who happened to use Symantec—one of the most popular cert providers on the planet—for an SSL cert. Not good.
I'm not taking sides on the issue, because I honestly don't know that much about it. But comments like the above really trivialize the huge impact that something like this will have.
not doing something is allowing the previous lack of security standards to put all of those people in danger. there is a tradeoff with every decision.
the entire PKI ecoysystem relies on self management and I think google is making a tough decision, but one that ultimately has already led to better decisions regarding security.
So glad others are making this point. Not to mention that the CA system is one who's sole feature and only purpose is trust. There is zero security if you can't trust the root. We'd love to live in a world where providing security didn't require a trusted third-party, but that's not the world the internet operates on, today. Transparency is provided by establishing strict guidelines and expectations for CAs. Maintaining this trust is enforcing those guidelines equally among the CAs regardless of size/scope and punishing (preferably severely) any violations. Otherwise the old saying of "rules that are not enforced are just suggestions" comes into play.
Because it's a trust-based model, the consequences for violations of that trust have to be strong enough to make a compromised actor willing to refuse government intelligence divisions, corporate criminal actors or even organized crime that has the resources and could stoop to any level of blackmail/bribery to squeeze out a certificate that would allow one to spoof a connection to a high-value target like Amazon, Google, Paypal or Microsoft.
On the other hand, Symantec's CA was caught mis-issuing certificates. That affects everyone, not just Symantec's own customers.
The real question is, what effect would doing nothing have on all of those people? If Symantec continues to mis-issue certificates (as they have done repeatedly: https://wiki.mozilla.org/CA:Symantec_Issues), keeping their EV status intact could be much, much worse for them.
> As the blog post states explicitly, there are a lot of use cases where updating a deployed app is very difficult, almost impossible, or cost prohibitive.
As others have pointed out, this change would actually have no effect on the 'complex dependencies' Symantec identified:
* This has absolutely no effect on embedded devices, which are certainly not running Chrome. * Mobile applications that pin certificates shouldn't be affected, since the very fact that they're pinning certificates means they're not using Chrome (there's no way for an app to tell Chrome to pin a specific certificate).
https://medium.com/@sleevi_/a-history-of-hard-choices-c1e1cc...
> It turns out WoSign has repeatedly been lying to Mozilla about backdating SHA-1 SSL Certificates
Source: https://www.riskiq.com/blog/labs/wosign-and-startcom-caught-...
Certs could be checked via Certificate Transparency but currently this is not required for DV certs.
A suitable compromise could be an override switch that can only be activated via Group Policy. That'd at least be a reasonable suggestion from Symantec. Not their current offer of nothing.
As far as "updating" my point was that they described a routine mobile app update as "recoding" as if it'd be lots of actual code to rewrite. And again, the fact a large company has no way to easily update in face of a security problem really is their problem, not ours.
Sometimes doing nothing is the wrong thing. They did wrong by supporting Symantec (or whatever shitty CA they originally did business with that Symantec acquired) with their money.
Or if initially they did not know about how shitty Symantec is, now they ought to know, and yes, it's a business risk (counterparty/vendor risk basically).
Those are precisely equivalent.
> Again, remember that the person being punished here is not Symantec;
Yes, it is, though you make exactly the standard "too big to fail" argument. yes, customers who relied on the misbehaving CA will also incur some costs, but if misbehaving CAs don't see reduced or eliminated trust at the level of browsers and similar trust stores, the entire ecosystem -- including the good actors and their customers -- pays the price as end-user trust in the ecosystem weakens because certificates do not guarantee what they are supposed to. And if any actor is recognized as too big to fail, that creates an unlimited license for bad behavior on the part of that actor at the price of the whole ecosystem.
The current Symantec SHA1 intermediate CA cert is valid between Feb 2010 and Feb 2020, so it seems like there was probably a cert change ~7 years ago too. The horror!
How hard is it to run some version of Chrome or Firefox that is within the last 5 major releases?
I don't care how big your organisation is, it's not that hard.
Yet they want to off-load the costs to the rest of the world to keep sites compatible with their IE8.
This is happening because Symantec was grossly negligent. They're going to be held to account for that.
All those giant companies should be firing up their lawyers and starting emergency procedures to change to a different provider. Anyone still using Symantec at this point is absolutely playing with fire.
Are these large organizations somehow incapable of putting out official statements regarding CAs? If they're being 'under-represented', it's their own fault for not speaking up.
I'm at a loss for a good analogy so I'll pick a lousy one: it'd be like if Samsung's response to the exploding Note 7 recall mandate were "Don't do this -- you're going to keep some of the biggest customers -- users of our phones -- from being able to make emergency calls/check their e-mail/respond to texts" ...yeah, but I'm still not OK with the small possibility that carrying my phone in my front pocket might result in third-degree genital burns.
... well that's their problem, right?
You can't simultaneously say "These are some of the most important organizations in the world and you'll cause worldwide chaos" and "Won't someone listen to these poor companies, I am the Symantorax, I speak for the cohort, for the cohort has no tongues."
Thanks for that -- reading this comment connected a few thoughts I was struggling with.
Their response felt like "if you don't follow through with your plan to punish us we'll do all of these nice things for you." It reminds me of when my kids get caught doing something wrong and I get "but I'll do all of my chores and treat my siblings nice" along with a handful of other promises of future deeds which I know will be partially or completely not-done. How different the result would have been if they came to me with the chores done and some/all of the promises fulfilled already. It would have been much better for Symantec to have responded with these items in the "completed" category with a smaller number of "we will's".
But to the quote I highlighted: the issue at hand is the past failures indicate that "some of the most important organizations in the world" need to not be trusting Symantec in the first place. They've had a big enough history of failures as a CA to get the attention of Google, who I'm sure isn't terribly interested in degrading the browsing experience of their customers. The choice becomes: (1) Banhammer Symantec, take the resulting pain all at once, visibly and obviously, and get the problem fixed painfully (but it's fixed) or (2) Trust that things will be different this time and if they're not, have another security failure of some kind that results in these critically important company's customers' data being exposed, which is a circumstance that will almost certainly go unnoticed for a period of time, will be covered up if the exposure is minimal and covering up the exposure is possible or may not be discovered at all. The CA system is centered around trusting a handful of rather powerful bodies with an enormous responsibility around the overall security of the Internet. It's not a great model to begin with, but it's what we have and I haven't read about any alternatives that are viable. The only way to keep a model like this secure is to ensure that the CAs themselves are actually trustworthy and one of the more likely ways that will happen is to enforce violations of that trust in the manner Google is suggesting.
Among other things, if what Symantec is saying is true, this means there is zero financial incentive for Symantec to behave. The largest certificate customers in the world have all voluntarily made it infeasible to choose a Symantec competitor, so as long as Symantec has any product at all of any quality, the market will keep buying Symantec.
If what Symantec is saying is true, the mere fact that they went to their largest customers and said "Suppose you had to buy certs from one of our competitors, what would you do" and they replied "We've never thought about that before" is itself a major victory.
If you have too many trusted-third parties in a trust model like this, you run into difficulties with enforcement due to scaling problems, but what we're seeing is the converse of that. Symantec went out and bought up other CAs (Verisign is the only one I can remember for some reason) and they've become so large that them losing CA status would be prohibitively damaging to the internet-at-large. This affects not only enforcement (we don't want to move on a bad actor because the consequences of enforcement would be more severe than the consequences of a rule violation), it also negatively affects the trust model -- now if I'm a bad actor, Symantec is my only target -- even though I could attack the others and yield some of the same benefits of attacking Symantec, if I breach Symantec, the rewards are larger by several orders of magnitude depending on what they store and how they store it.
To digress a little ... today ... I'm not all that surprised that Symantec has reached sort-of monopoly status in this area. Not because Symantec is a well managed company (I'm making no statement for or against that despite my personal opinions), but just because so much of The Internet has turned out this way. If you would have told me in 1998 that for the very vast majority of US Internet users, there'd be one search engine, one e-commerce company, one (expensive) ISP, one e-mail provider, all of it would be secured by services provided by one company's CA and all of that wouldn't matter much because the typical Internet experience would basically be a web-enabled, bluer version of AOL[0], I'd have called you some unfriendly things, but that's where we are. It's a centralized system operating on decentralized technology.
[0] Oh, and that the "one e-commerce company" was that bookstore web-site and would be responsible for hosting such a large part of the rest of the Internet that when they had a major failure and people had that handful of rare clicks off of FAOLBook, they'd have about a 50% chance that whatever their target was wouldn't load.
> Mobile applications that have pinned certificates. Replacing server certificates would require these applications to be recoded, recompiled and redistributed.
Are either of these relevant? Isn't google's proposal to eventually have Chrome stop trusting the Symantec CA system? That change seems like it would have no effect on 1) embedded devices that aren't running Chrome 2) things that already are in place that trust the CA.
I could see someone translating "these devices use Symantec's server certs and serve web traffic" to "these devices are pinned to certs issued by Symantec". But if that's the case, they should pay more attention in announcements like this one...
Or maybe I've misunderstood the situation.
I've always been of the opinion that the rogue/careless CA problem can be solved by mandating N root sigs and recommending M>N. In a system with multiple independent LetsEncrypt like entities this is doable. If a CA needs to be revoked you can do it in a much easier way here, since it's still ok to transitively allow sites with N-1 certs and one now-disallowed cert signed by that CA. Doesn't fix all the problems but it certainly removes the inertia behind kicking out bad CAs.
Looks like we'd need a x509 extension to do this then. :/
But I don't think Google is going to be swayed by "you can't distrust us if we do wrong because too much depends on us" because if they do, it encourages laxness at big CAs.
In this case the story is more complicated - I think new certificates signed by existing Symantec CA roots would be trusted if their duration is 90 days or less. Historically, most CAs offered one, two, or three year duration. Presumably, the "big customers" would not accept the need to rotate their certificates for this purpose in less than 90 days, they need their 3 years because it takes over 90 days to get something through their bureaucracy. And presumably there are other things they can't do in 90 days, like update an embedded device configuration or firmware, or a mobile app.
Of course it depends on the particulars, but of course these "big customers" can't figure any particulars out in 90 days. But that's why Google proposed a slow sliding scale that would gradually reduce the max duration allowed over the course of a year, so random unknown stuff breaks more gradually throughout the year, not in one huge event like Symantec's blog post suggests.
Embedded devices => https://api.example.com (Symantec cert)
Browsers => https://new-api.example.com (New trusted cert)
Another approach is that if the devices are old enough they're probably using a deprecated version of TLS, so the existing https://api.example.com server could choose which certificate to issue based on the version of TLS offered by the client.
If Chrome ships this at the same time as re-enabling TLS 1.3 support, that would be a decent way to target, since presumably very few of your distributed clients are using TLS 1.3.
This is how we make sure issuing certificates for gmail.com, etc will never happen again. An external audit every 3 months is not going to fix anything.
I see it as a polite way of saying "You kids".
https://archive.fo/FMemh