Looks like all extensions have been disabled for all Firefox users.
I think this fail-closed behavior is more of a security issue than the one it is trying to solve. All of my security add-ons - Privacy Badger, NoScript, Decentraleyes, and many more were disabled. Even worse, it happened without notice to the user.
One moment I was browsing the internet (just barely) secured by these add-ons, and the next moment, all of them disappeared (without warning) and I only noticed when I saw my password manager was missing.
Mine still work. I tried to set my clock to two days ago to avoid it and promptly got errors on every HTTPS site I visited. Damned if you do, damned if you don't :/
If it failed open, anyone unlucky enough to update their extensions could end up having a malicious version installed. It also would have taken longer to notice.
Yes, but what's the point of cert expiration? Is it safe to have certs that never expire? I believe there is a security benefit to expiration. Expiration is useless if it's never enforced.
Probably the correct behavior is to have some sort of semi-annoying popup when it expires, and then only a week later do the full blocking. You need to strike the right balance of making it annoying enough that it can't be ignored by everyone (otherwise you just have the exact same problem, just delayed a week) and that fear of it happening is a sufficient motivator to stop people lazily relying on the grace period, but also not too annoying that it makes a lot of people quit. You also want to avoid permission fatigue.
Presumably because how would it differentiate between a legit "already installed" extension with a signature that cannot be verified, and an extension installed by malware that also cannot be verified?
Personally I despise the idea of the software already on my pc being dependent on signatures stored on a remote server. I installed it and Mozilla can fuck right off. It's my responsibility to police what software is on my computer, not theirs.
Browsers can only protect against malicious websites and malicious extensions. They can't protect against malware. Even without any cert problems, malware on your machine can modify the browser executable/process to insert whatever code it wants.
With this reduced threat model, it's easy to simply keep existing pre-installed extensions available, and disable updates. Your only problem is if a pre-installed extension is malicious or has a vulnerability, it will remain.
> Presumably because how would it differentiate between a legit "already installed" extension with a signature that cannot be verified, and an extension installed by malware that also cannot be verified?
This is why a signature can also be accompanied by a trusted time stamp which can confirm that the signature was made while the certificate was valid.
This is the common way to sign all Windows software to avoid this exact kind of problem.
Yes, that implies this is a known and solved problem. It’s embarrassing for Mozilla to not have prepared for this.
If an extension was already installed, it passed the signature check at the time of installation. I'm not sure what benefits we get from periodically re-running the exact same check -- particularly when balanced against the risks of the re-checks, which are now obvious.
It should fail 'locked'. continue to allow installed addons to work, notify the user of issue, disable any updates without explicit request by the user.
It's a great argument against centrally controlled walled gardens. Basically breaking or compromising a single certificate has a widespread impact. Even Tor browser is impacted. For most of us this is a temporary inconvenience but there are people whose personal security depends on some of the extensions that just stopped working.
On a positive note, it's been a while since I browsed without a lot of extensions. Ads are still annoying and I noticed some extensions apparently had more of a performance impact than you'd hope.
As a temporary fix, go to about:debugging, and click "load temporary addon", then paste in the download link of the missing add-on. Then just try and not restart Firefox until they fix the broken cert.
This is a goddamned disaster. I'm just thankful that I use an offline password manager, but even still ...
I like FF, don't get me wrong, but this is going to absolutely fucking destroy user trust in Mozilla. This kind of incompetence, on a browser scale, is breathtaking.
I think MrEldritch is referring not to Firefox's built-in password manager, but third-party password managers such as Bitwarden, KeePass, and LastPass. Those rely on add-ons for browser integration.
One you have targets mapped correctly keepass2android is great and I now that I've gotten used to it, I prefer the system ui password filling for everything including the browser. Also installing the keyboard extension is great and makes for an easy way to quickly access logins.
I dunno. I’m a typical Firefox user, and I’d rather jump off a bridge than switch to a different browser because of a fuckup like this. People make mistakes, but Mozilla still stands for things that certain other browser vendors don’t, last time I checked.
That's my thinking too. I went back to Firefox a few months ago and it is back to being a fantastic browser now, and it feels good to use something that is also a force for good. I'm hoping they resolve this quickly and that it all turns out ok.
While I agree with you two assuming the bridge is over water and not too high, there are real consequences that cannot be reversed. I cannot unsee the ads I saw in the past few minutes before switching to nightly.
Sadly, what they "stand for" and what they actually do are two different things. This is exactly the kind of centralization that a company supporting a "free and open internet" (to use their words) should be against on principle, let alone pushing in their only product of note.
This should not be possible.
Worse, had they not taken the paternalistic, nanny-like stance that you can't even disable the signing checks, I could roll out a script that would make this a non-issue for my users. But no, thanks Mozilla for ruining my Monday.
Might not be the most substantive comment I could possibly make in the circumstances, but I'm pissed. The only appropriate response feels like a string of infuriated profanity directed at their incompetence and decision-making.
True. But they also increasingly stand for things I completely disagree with. Namely, deciding which software is approved for me to run on my computer. The way I see it, extensions shouldn't need to be "approved" anyway.
Luckily, I can still type "make install" without debian informing me that "random_dangerous_untrusted_code_from_interwebs" is not approved.
I use firefox and am probably affected by this but don't even really notice atm. This doesn't even register on my user trust spectrum when the only other option is the browser that defines surveillance capitalism.
I think we'll all live. No need for the chicken little act.
I'm not sure the GP is overstating things. For technical folks with technical reasons to be using Firefox: yeah, a mass exodus is unlikely purely because there aren't any good alternatives. What are you going to jump to? Chrome, and knuckle under to the Goog? Unbranded FF forks and be weeks behind on patches? Doubtful.
My concern is around non-technical users (the group, mind you, that Firefox has been spending marketing dosh on courting recently with Quantum and all) who don't have as compelling reasons for not just switching back to Chrome. In the last hour, I've gotten several phone calls from family members asking me why the browser I convinced them to use is broken. I don't have a good answer, because platitudes about surveillance and muh freedoms don't count for shit when your grandma just wants to get rid of the ads on the local newspaper site.
I'm personally going nowhere and deeply appreciate Mozilla for all the work on FF and friends, occasional fuckups aside, but I don't think this is going to be a non-event for a browser that's been desperately fighting to regain market/mind-share.
I'm using the Chromium-based Edge at work, and it's going to be a great browser. It's the one that most people should switch to if they want to leave Firefox. I've been using Firefox since 2002 and never left, but native browsers like Safari/Edge are appealing for many reasons, including deep integration for the best battery use. I tend to recommend people use native browsers or Firefox but never anything else.
Yes, this is truly a gut-punch for everyone who has spent a bunch of time and effort getting their family and friends on a decent, cross-platform password manager (like LastPass, 1Password, or Dashlane).
Users will drop a product for the slightest reason. For instance, one of our users recently left a negative review. Paraphrasing, "Logging in is difficult".
We check our warning system (set up to detect suspicious logins, incidentally also catches any users who've been locked out because they forgot their password), and his last login attempt took a total of two tries.
(Big fat disclaimer: I work for Google. These are my opinions and not my employers. I don't work on browsers. I test my code in Firefox. Etc etc.)
Sadly, I have to agree that this feels like a big blow to user trust.
User trust is not really just about respect or values; it definitely also includes things like performance and reliability. The average user, right now feeling powerless, might even feel anger towards Mozilla for this - after all, they already downloaded the extension, why would they all just stop working behind their backs? They don't understand what CAs are or why certificates expire. People don't frankly care what place your heart is in when they are angry about something. Perhaps people are being dramatic, but that's normal. People are pretty darn dramatic about Chrome, too.
Meanwhile... I use Firefox everywhere, and I've lost my password manager, adblocking, security-related extensions, etc. all in one go, and the only solutions I'm aware of involve disabling extension signing. Gotta admit, even though I will probably continue using Firefox after this, that it certainly is a bummer.
And yet every other major browser vendor has punched their users with far worse catastrophes of privacy, security, ripping away features, breaking features, and general shitheaddedness.
Switching browsers because of this incident is like ordering a burger at your favourite restaurant and one time it comes out without the meat patty, so in protest you switch to a crappy alternative restaurant that has had a long history of health code violations.
I'm going to skip the analogies and just say this: If tomorrow this is still broken and I have a choice between installing Chromium, and installing Nightly + disabling security features, It's going to be a tough dilemma for me personally.
I'm glad you have software/vendors you feel you can trust. I definitely don't feel that way about most software anymore. I do think you are being a bit hyperbolic regarding other browser vendors, but to each their own, I don't know what trying to argue about that would solve for anyone.
>Well when you are google employee and you are testing code in firefox... you already have chrome and chromium installed.
I have computers other than my work computer(s.) I, indeed, do not have Chrome or Chromium installed on my home boxes running NixOS. I do not use my work devices for personal web browsing. I'm currently posting this message with Firefox 66.0.3 on NixOS 19.03.
>I think what is a real tough dillema is being sad about nonfunctioning adblocker while working for the biggest internet ads company.
>
>So are you working in chrome marketing department?
I work at Google because it's an excellent place to work. I'm far from elite; I didn't finish college (couldn't afford) and I grew up in the suburbs of Detroit, so being able to work at any large SV company is something I don't take for granted. I don't think any single employee can claim to love 100% of the things Google does, and that's fine. Nobody is required to.
As for why I would use an adblocker, practically speaking it's both for reducing annoyances and increasing security. Malware (and 0days!) delivered via ads is not unheard of, sadly.
I know the typical user my have struggles but FWIW, I installed nightly, toggled xpinstall.signatures.required to False installed ublock umatrix and will live with my pw manager's native application for a day or two and it took about 5 minutes.
In fairness: I don't really want to disable signature checking. I value these security features and I'm hoping that by tomorrow morning Mozilla has a better solution.
Looks like you're right. I saw the other discussion first and it still has more comments for now, but this one is ranked higher. Perhaps the threads could be merged or something.
I’ll still keep using Firefox since I recognize the importance of browser diversity and the hazards of a Chrome monoculture (that and vertical tabs), but, yikes.
Still, this type of oversight seems all too common even in large companies. I remember several cases from Fortune 500 companies in the past few years alone. What would be a good way to automate checking for them? Has anyone developed a tool designed specifically to avoid certificate expiry disasters?
We scan our codebase for anything that looks like a cert and send emails when it gets close. Might not have helped here if it was an intermediate owned by a CA. There but for the grace of God go I.
Monitoring CT lets you verify that somebody renewed the certificate, but it doesn't verify they actually installed the replacement correctly.
My employer (Kynd.io) currently monitors public web sites for customers so we can flag e.g. "Hey this site cert expires in a week! If it's dead probably just switch it off, otherwise renew the certificate" and we're in the process of integrating CT but mostly so we can say "You already have a newer cert but need to go install it" in our How To Fix instructions.
I've never seen a bulletproof solution for organizational tasks that need to be done yearly.
If someone's in charge... and both they and their manager happen to leave in the same year... and whatever system they had in place to remember (probably their personal calendars) is gone... and the manager's manager has 1,000 other things to remember...
...how does an organization ensure the task still gets done?
We resolved this issue at my last company with sufficiently large mailing groups for cert renewal reminders. Once you get to 12 people on a mailing list, with new employees being added all the time, it's hard to miss. Usually a manager on that list is pinging people about it. There is the chance of the tragedy of the commons occurring, but I never saw it.
Once you do this, the only checklist that matters are procedural checklists to add a new client or new cert to the renewal notification list. When you use a standard group email for all cert purchases, that one becomes tough to miss.
In my 7 years of being involved, we never missed a cert renewal with this process for ~300 client sites with multiple or wildcard certs.
There should be a separation between the things that need to get done and the people that do them. As in, tasks should be created first and then assigned.
Surely tasks are performed by and assigned to roles, not individuals (who just happen to be in those roles at some moment in time). If a role disappears, e.g. in redundancy, then the role's tasks are evaluated for either transfer to a role that remains, or being discarded.
It's not that complicated, just add scheduled health checks to the same system you use for checking if the website and such is up. If the expiry date isn't updated within a week of expiry start paging engineers.
I'm willing to bet Mozilla already does something like this but an engineer didn't set it up correctly for this certificate.
Let's not forget multiple mobile networks across Europe went down on the same day last year because Ericsson(?) let a cert expire on some internal management system that had not been updated. SSL cert renewal is one of the great unsolved problems in computer science
There was also an issue last year where every single Oculus Rift was essentially bricked because they forgot to renew a cert (apparently, what with the chaos of the Rift launch and the Facebook acquisition between the cert issuance and expiration, they just kind of ... lost track).
It took like two days before there was any kind of fix available, and they couldn't even roll it out automatically because the expiration had also disabled the auto-updating.
>SSL cert renewal is one of the great unsolved problems in computer science
Certificate expiry really only exists to make money for CAs. It doesn’t solve any security problem that CRLs don’t already solve (and solve better). There’s lots of unsolved problems relating to ‘how do you make a reliable PKI’, but cert expiry is really just an unrelated business requirement for CAs.
I'd argue it's a blunt hammer extra layer of defense, where if a certificate gets compromised and the owner never finds out at least it eventually stops working. This kind of compromise is pretty common.
If it really was only to make money for CAs we'd see LetsEncrypt offering very long lifetime certs. But:
* Very short lifetimes get people to automate, preventing problems where one cert lasts long enough to lose the institutional knowledge around it.
* CRLs don't work. For performance you don't want to check for a revocation in serial with the request, and you don't want to block all browsing if the revocation list server is down. Revoking a cert will cover some users, but lots will still get "https://" and no warnings.
CRLs are not equivalent at all. They are a last-ditch effort to fix a problem when all else (expiry) has failed.
CRLs require maintenance and distribution of a list by a 3rd party. Creating an accurate, all-inclusive CRL of all website keys that your browser should reject is far, far from easy.
(Case in point: "how many web sites are there?" Is not an easy question. )
Properly propagating such a list to any browser that might need it is another daunting task - less than 100% propagation means end users are exposed to security risks.
Certificate expiry is much more elegant: the client can check the certificate's validity himself, without relying on input from 3rd parties.
If certificates didn't expire, CRLs would (by now) be huge and growing enormously every day. They'd be so big that by the time you'd have downloaded one, it'd be outdated.
CRLs can be sharded, the cert carries the URL for the relevent CRL inside it. So they wouldn't need to have grown as huge as you suggest.
But, this sharing carries a cost for user privacy, if I shard certs 16 ways then each CRL download gives me 4 bits of info about which sites you were visiting.
OCSP effectively takes this to the extreme, each lookup is tiny because it's just for one cert, but it gives away exactly which cert you cared about each time.
Besides leaking data by on demand CLR checking, you also have a difficult fail open v fail closed decision.
Failing closed means failure of a third party immediately breaks your site. Failing open means a MitM can simply block the CRL check.
OCSP stapling and the 'must staple' header are a lot better for privacy, and OCSP responses have some validity so at least a 5 hour outage of your CA doesn't bring your site down immediately.
It is still vulnerable to a DOS and trust on first use though.
I would like to live in a world where OCSP stapling is widely deployed and we can require OCSP and advise people to set must-staple if possible while everybody who doesn't staple will just have to eat the privacy implications. But this is not (yet and for the foreseeable future) that world.
Apache and nginx both shipped OCSP stapling implementations that are very bad, awful enough that for almost anyone I'd say "No, don't enable that" rather than try to explain how they need to use it and get them to a place where it's useful and safe. Adam Langley wrote years ago about how to do this correctly, and there does seem to be a little bit of movement in the correct direction at Apache, but the situation remains pretty poor.
Cert revocation suffers from a very simple issue. If your check for revocation fails, do you fail open (ie accept the cert) or fail closed (ie reject the cert).
For any method, fail closed is user hostile and often a DOS vulnerability whilst fail open is another way for an attacker to use a revoked cert.
This is a big issue with on-line methods like OCSP as a MitM using a bad cert can probably block OCSP traffic as well.
CSLRs grow out of proportion, and leak information to the outside world.
Cert expiry serves as a backstop to these other revocation methods, and as a bonus ensures that simply forgetting about a cert cannot bite you 10 years later.
All TLS failures fail closed. The idea that if a cert is compromised it will eventually expire sometime within the next five years is a completely laughable security control. Leaking information is a complete non-concern too. Have you heard of certificate transparency logs?
Short lived certs are quite obviously better from a security perspective, but the security difference between a certificate that expires in five years, and one that expires never is irrelevant.
A missing OCSP response does not fail closed, nor does a CLR url 404-ing fail closed.
The information leakage of CRLs is stating to the public that a cert needed to be revoked.
Obviously, a compromised cert that will expire in 5 years is horrible. However, a non compromised cert you are no longer using that will never expire is more off a risk than a disused cert that will expire in a year. Not to say you should leave the one year cert lying around. However, there is no desire to put the one year cert on a pre-shipped CLR.
ACME / Let's Encrypt go in the direction of making expiry happen so often that renewal gets automated, rather than a being a rare manual process that can be forgotten about.
Not sure that's viable for a signing certificate like this, but that's the way to solve it for the web PKI.
This is just abusive to the vast majority of users who do not care but still want to use SSL for their servers, frankly. I should be allowed to choose a near unlimited lifetime for my server's certificate if I don't care about the risks that may present.
Yeah "just" a cron job except the implementation changes several times a year. Somehow this automated process was more time-consuming than the previous, manual one.
Many cloud providers will make this process pretty much entirely automated. But let's say you don't want to do that: when is the last time the way you run caddy changed? Or the last time python-certbot-nginx changed?
This was a few years ago, so things may have changed by now. But as they say, once bitten twice shy, and the wisdom of "just cron it" doesn't work with highly experimental tools like LE was for what I estimate to be the majority of its lifetime.
I'm sure there's a way to make your LE experience consistently suck but the way to run caddy for a static website has been the same for about as long as caddy has had support for automatic HTTPS, and that's also true for python-nginx-certbot. But more importantly: we can argue about what it was 4 years ago, or we can just observe that it's really easy now.
A tool not working well or being "experimental" does not dismiss the premise that frequently run automated tools are a better than infrequently run manual tasks when those manual tasks can take down your infrastructure if done improperly, missed or forgotten.
All it being new means is that depending on your risk ratio you need to decide whether updates to the software need testing or whether you need to invest in your own solution - or, how about just wait until it matures and keep the old process until then.
Waiting doesn't invalidate the premise either. It just means you lack the resources to implement it safely and that's ok.
Actually it could be not negligence but a way to perform an attack.
Register a domain, get a certificate lasting forever, let the domain expire and somebody buy it. Then somehow redirect all or part of the traffic to that domain to your own server with a valid certificate. Chances are that few people will notice something has changed in the details of the certificate.
However you'll have left traces all over the place: credit cards, phone numbers, etc.
Its also more secure. Long lived certs risk the possibility that someone who used to own the domain got a certificate on it and it still works after the domain is resold. Once you automate it there is no downside to short lived certs.
Revocation lists get huge, ultimately becoming another reason to limit cert lifetime (you don't have to tell people you revoked a certificate which is expired naturally).
Very few things check revocation, unfortunately - it puts an extra hop on the fast path of connecting to a server. OCSP stapling is pretty much the only thing a browser would care about - having the server fetch a signed OCSP response that is good for a limited period of time (say, hours), and send that along with the certificate during negotiation.
Or, you could just have the server fetch a certificate thats good for a limited period of time.
This is not true. In Let's Encrypt/ACME for example, you can simply obtain authorizations for all the domains a certificate is valid for and request revocation [1]. The only thing you still need to revoke the certificate, is the certificate itself. The certificate can be obtained from CT logs.
OCSP stapling together with OCSP Must Staple is the way to go here. All major browsers support these.
Firefox still does normal OCSP requests, Chromes does not. So if you are a Chrome user, to my understanding, there is now way to know if the server certificate was revoked or not, other than OCSP stapling together with OCSP Must Staple. Additionally, both Chrome and Firefox ship a list of revoked certificates, but it may not be updated quickly enough and as far as i can tell it mostly contains roots and intermediates.
Short-term certs _are_ a virtue. Not only do you not have a manual event rare enough for people to forget how to do it, you also don't have to worry about which 15 services someone granted a 10 year wildcard cert to early in the company's history.
Having once had to regenerate 600+ self-signed certs, test that everything still worked, and then insert them into the 600+ live app servers without breaking anything, all within a two week window because no-one had realised the 10 year expiry was just about to bring everything down, I concur.
See also: GPS vs GLONASS time encoding. GPS rolls over every 19 years, so devices, cars and even Boeing aircraft saw their GPS-based clocks turn back to 1999 last month. Meanwhile, GLONASS epochs are only four years long, so every device that uses it as a time reference is built to handle rollover.
Talking about vertical tabs, I was in the middle of studying for an upcoming exam, then when I alt-tabbed back into Firefox, all of my tabs are missing with that unsupported addon error. Fortunately refreshing Firefox gave me back normal tabs, at a cost of uninstalling all of my addons.
The problem is that Tree Style Tabs relies on userchrome.css edit to hide the tab bar, and when TST is forcibly removed there is no way to access the tabs, because that edited userchrome.css is still there. This is very disruptive. At least with the pre WebExtension addon TST itself hides the tab bar, so if TST is removed then the original tab bar comes back on automatically
> Still, this type of oversight seems all too common even in large companies. (...) Has anyone developed a tool designed specifically to avoid certificate expiry disasters?
LetsEncrypt renewal is supposed to be automated. [1]
I know of a company that hosted blogs for thousands of customers. They used LetsEncrypt, but the CTO considered automatic renewals a possible security risk, so they did it manually. Problem is, the expiration happened in a weekend and they "forgot" to update the certificates before that. Suffice to say that the next Monday wasn't pleasant. They automated after that.
I'm curious as well. My intuition would be that it's not a concern, since servers already keep their private keys stored locally in order to be able to communicate with clients anyway? Being able to update them doesn't really seem to make things any different. But I feel like I could be missing something/not have thought through it properly. (I imagine security implications can get more complicated if a different server decrypts traffic vs. processes it, etc.)
The "manual" process used previously by the company already involved some form of automation, so it was more about trusting CertBot not to do anything horrendous.
But now that you mention it, I wonder what's the opinion of security experts like tptacek on cert renewal automation.
Having a root process with write-privileges to /etc on production machines and also able to communicate over the Internet definitely is a security risk.
To mitigate that you end-up building a series of privilege-restricted jobs flowing from the DMZ back into the internal network. And maintaining that might be more complicated than just manually renewing, depending upon the processes and architecture of the company.
Most popular ACME (Let's Encrypt) clients allow you to provide a CSR instead of generating the keys themselves. That means a bunch more work for you, but if you're worried about this, that's what you should do. Have your safe (even manual if you insist) process make keys, make CSRs for the keys, and put those somewhere readable. The ACME client will hand them over to the CA saying "I want certs corresponding to these CSRs" without needing access to your TLS private keys at all.
This is technically true, but contextually lacking.
acme-go/lego doesn't use HTTP validation unless you disable just about every other form of validation first. TLS-ALPN validation is much more likely, so port 443.
That said, it is very easy to allow software to bind to privileged ports without providing it root access; this has been solved for a very, very long time.
Some shared hosting like Bluehost now provide LetsEncrypt by default for all their sites with auto-renewal (But I don't recommend Bluehost shared plans for anything even closer to serious hobby due to absurd downtimes like most other shared hosting).
I used manual renewal for LetsEncrypt for about 4 websites on other shared hosts & renewing them every 3 months was a pain; had to keep reminders and schedules just not to miss renewals until I synchronised their renewal schedules to batch (manual) renewing them.
I had automated renewal for 1 website on a cloud server, it was a one time effort, I never had to bother about SSL cert for that site and the most favourable of them all.
Another option is using a Web Server/Reverse Proxy that supports Let's Encrypt automatically, like Caddy [1]. I believe Apache HTTPD has partial support [2], too.
I own a webhosting provider. We offer Let's Encrypt with automatic issuing and renewal, securing 184,961 hostnames (SANs) at this moment.
We issue certificates automatically if none is existing when connecting to a website and renew the certificates in batches 30 days before they expire. When renewing, we merge certificates/hostnames into bigger certificates with 90 hostnames so we don't have so many moving parts.
If renewal would break, however (as it did once or twice before), nothing bad would happen because on page load there would be a new certificate issued.
> had to keep reminders and schedules just not to miss renewals until I synchronised their renewal schedules to batch (manual) renewing them.
Another use case for the app I am developing! The basic idea: You can enter an item (i.e. "MyOwnShop Cert") into the list. From that time on, it will be tracked how much time passed since the item was entered or renewed (by clicking the renew button). The item with the longest time since entering/renewing is at the top of the list.
Compared to schedules and reminders it has the advantage that the item is not out of our mind once the reminder or schedule pasts. It just sits there dutifully and its timer keeps increasing.
I use it for keeping up with middle-term contacts ("Wow, I have not written Carl for 3 weeks?") and health-related issues. Logging in stuff that easily spoils would be another use case. And, apparently, cert renewals :)
I have no idea why you'd deliberately wait the full 90 days to do a manual renew. For reasons, I renew manually, but every 60 days or so. Nowhere close to the deadline.
Just curious, are you talking about Webflow? Because I had to hunt down and make sure our Let's Encrypt auto renewal was working until I realized the certificate was served by them. They wait until the last 12 hours to renew the certificate. I have no idea what type of rationalization would lead to that decision.
I had the same thought, but I still find that absurd. Say they host 500,000 websites with HTTPS. 1,000,000 renewals they save spread across the year, roughly 2 renewals a minute. That is pennies. A t2.medium could handle that type of load increase
They didn't have renew automatically but they could automate notifications, alerts or even banners in their internal apps when 60-70% of the time was exhausted. If I was given such a restriction, I'd still automate it 100% but require a human to authorize it every time by clicking a magic link in their email, slack or some dashboard, and nag them with notifications until someone authorized it.
It's automated but things can go wrong even when correctly configured and tested. Real world example: certbot version got old, the renewal server didn't support it anymore, the certificate didn't renew, the web site got the dreaded https warning page.
Of course that is also a kind of misconfiguration. The site has Debian security auto updates on but certbot is not among them. It should be forced to be updated. Furthermore there was no monitoring of errors in its log file.
Still it's not as simple as one believes Letsencrypt to be.
Then the automatic update process stops for some reason and your certificate expires...
At the end of the day, someone needs to verify that new certificates gets acquired and installed before the old ones expire. Automation makes acquiring them less tedious, but not much for making sure someone pays attention.
I run one daily from cron and have it email me a report with the days to expiration for the certs I’m responsible for, even for certs that auto renew. I don’t filter the email. Daily is not too frequent for it to go to my inbox, but frequent enough that I’ll notice if it doesn’t mail me. YMMV.
I agree. What can be done to prevent developers from adding a certificate dependency without monitoring during the move-fast-and-break-things days of early development, which then sits for X years as developers come and go, and nobody notices until it fails?
Certificate Transparency works pretty darn well for most usecases, we (Latacora) have found while trying to solve exactly this problem (or at least the figure out which certs exist that aren't being regularly re-issued part) :-)
Certificates that aren't from the Web PKI almost invariably won't be logged. Most logs explicitly refuse everything except certs from the Web PKI so as not to be burdened storing garbage. So this won't find certs issued by the custom OpenSSL CA on that one guys Linux laptop.
Not all Web PKI certs are logged. There is no BR obligation and no root store programme rule that requires logging. The only things in place that strongly encourage logging are the Chrome and Safari policies. For systems that aren't designed to be accessed with a web browser or, much more rarely, enterprises that have persuaded themselves only IE is authorised anyway, the certs might deliberately not be logged. Yes there are (small) CAs doing this in the Web PKI, on purpose, in 2019.
Whilst I'll say "disclaimer, this is my project", monitoring Certificate Transparency with CT Advisor has helped me find out about certificates marketing people deployed and expected me to maintain without my knowledge.
We have an agent that pulls certs from an internal service and stores them on disk where apps can use them. We no longer manually install certificates. This solves discovery, and gives us alerts on services that have stopped refreshing their certs for any reason. The internal service is wired into lets encrypt and a commercial certificate provider. Setup is minimal, and after that completely automated.
>Has anyone developed a tool designed specifically to avoid certificate expiry disasters?
Is anything more than a calendar reminder on the phone of someone important enough to shake the Earth and get it fixed For. Certain. needed? Like, say, the CEO, CTO, and CFO should at a minimum get a notification so they can ask if the refresh was done when necessary?
Admin people. Often the most senior ones get the title "Personal Assistant (to senior person job title)" but not always. They're lead bureaucrats, and tracking things that need to be done and ensuring they get done, either by doing them themselves or assigning them to reliable underlings is the purpose of their role.
Corporations are often not very good at putting the right people in these roles but good ones are invaluable. Since the Marvel Universe is everywhere, Pepper Potts is the archetype in that setting to give you an idea of why you'd need people like this. Tony Stark would be "too busy" to renew the certificates, but Pepper would make sure it gets done.
> I’ll still keep using Firefox since I recognize the importance of browser diversity
Also, Chrome is not immune to "crashes for everyone at the same time" bugs. Like that time when the start of daylight saving time made it crash for a full day (a quick search tells me it probably was https://bugs.chromium.org/p/chromium/issues/detail?id=287821).
I love "firehose of disease", and will steal it. And I agree that bugs are bugs; every time you add a new capability, you add all the possible bugs that can occur with that capability.
Maybe we need more browser diversity than just two different teams with two different systems. Both are sitting very close to each other geographically, and both are produced in the same culture (as in silicon valley), so it would seem likely that, while they compete with each other, they will apply very similar answers to problems they face.
There’s lots of monitoring services out there that do it. A long time ago I worked at place that used a service called site24x7 for cert and API monitoring. That was before Pingdom kinda got better than most API monitoring services, but I don’t know if they monitor cert expiry.
Taking a look around, you’ll find lots of service providers, or tools you could use. But the main issue is all they do is tell a human being to do something, which they can still fail to do. Which is why automating cert rotation (with things like let’s encrypt or ACM) is arguably a better solution than monitoring it.
Systems designed around long TTLs make this problem worse. I love the default of 90 days for Let’s Encrypt. It forces some good discipline and hygiene. Wish there was a better solution for short lived CAs
> Has anyone developed a tool designed specifically to avoid certificate expiry disasters?
Not perfect, but I've added a TLS certificate extraction tool into a DPI that displays all visible certificates ordered by expiry date.
One could then mirror all one's site traffic to it and let it run in the background. Coupled with some alerting tool it would catch most of those cases I guess.
I could polish the tool a bit more if there is some interest, but anyone could do it as well.
Realistically: reduce your own cert renewal window to weekly, if not daily. This forces you to have a good renewal system in place and alerts you to failures long before actual expiration.
Quixotically: make cert failure a randomised number, linearly related to how long ago the cert expired. This slowly introduces more and more failures, over a certain “grace period”, which makes the problem less of an extinction level event. It’s not a solution but it definitely would help.
I'm not familiar with Firefox extensions (and have pretty much stayed away from the stuff ever since they started making it "mandatory"...) but shouldn't the expiration only mean new signatures won't be valid, yet signatures made before expiration should remain so? At least that's how I understand things like Windows' driver signing works (when that was first introduced, I was quite scared that it would mean perfectly working drivers could just stop working due to the expiration, and asked... but apparently no one at Mozilla asked this question.)
Edit: wow, downvotes? Care to explain what I'm missing?
This same behavior is how certs usually work. Stuff with expired certs just does not run after the expiration date; that's because the cert tells you what server to ask for authentication, and if you have an old cert, there's no way to be sure that the original issuer is still the one in control of that domain.
You're referring to things like HTTPS and contacting a remote server, and thus your reasoning there makes sense.
I'm referring to traditional code signing, which I assume Firefox extensions are more similar to --- the goal being to ensure that some data has not changed since it was signed, and only the validity of the certificate at the time the data was signed is meaningful; even after the certificate expires, a signature created when it was valid still asserts that the data it signed has not changed.
In short, yes, they should have implemented timestamping for their code signatures like most other code signing systems do.
Without timestamping the expired cert always would have caused problems, even if it was replaced early and correctly: Every add-on would still need to be signed again with the new replacement certificate and shipped to all users. It's not as easy as just replacing the certificate on some server.
Well, this is still what has to happen: replace the certificate, ship that new certificate[1], re-sign every add-on, ship every add-on to every user.
Now, in order to ship new versions of the add-ons, you probably will have to bump the add-on version numbers as well. Which can have further unintended consequences.
[1] Incorrect, see blow; it is my understanding that the certificate in question is baked into the browser itself, with no way to push updates just for the certificate remotely other than shipping an entire new Firefox build. Well 6 new builds: esr, stable, dev, beta, nightly, unbranded. Gonna be a fun night for a lot of mozilla folks... Well, a night is not gonna be enough...
I might be wrong tho, and misunderstood something.
EDIT I was wrong (https://news.ycombinator.com/item?id=19824520), the expired cert is not baked into the browser, just into the add-on package files. No need for new Firefox binaries, after all. Still, they have to resign all add-ons and ship new versions.
First they force code signing on everyone without a way to disable it then they break it. This is an extreme level of incompetence I didn't expect from Mozilla.
They'd better have the best post mortum ever, possibly with someone being fired.
People are generally not inclined to be truthful about their mistakes if they expect to be punished for them. Its how problems keep getting covered up until they become catastrophes.
I guess it depends on if it was an honest mistake or gross negligence. I don’t think people should be fired for mistakes. I also don’t think everyone should be trusted with important tasks.
You didn't answer my question. What does firing achieve? You fire a person who learnt their lesson and will never make the mistake again? And then hire someone new?
Or you fire the scapegoat because of a broken system that allowed one person to make a mistake?
If this mistake was due to incompetence then the person should be fired. Incompetence shouldn't be tolerated.
But we're outsiders looking in and don't know what's going on at this point. That's why I used the qualifier "possibly." It's quite possibly it wasn't incompetence.
> You fire a person who learnt their lesson and will never make the mistake again?
That's true, sort of. How often do you let people make huge mistakes before you decide that maybe they are just not apt for the position that they've been promoted to and Peter was right? Once? Twice? And unlimited amount, as long as it's never the exact same mistake?
Can you link to your linkedin profile so we all know who to dox next time you make a mistake? If you have a twitter, please add that as twitter works even better for mobs.
That might seem rather extreme, but the fact that this situation was even possible was a consequence of a series of bad decisions over an extended period of time about the required behaviour of new versions of Firefox, combined with technical failures that betray fundamental weaknesses in the whole system design. Whoever was ultimately responsible for those failings demonstrably isn't competent to run something of this importance and should probably either implement immediate and dramatic changes to the relevant policies and technical details or consider their position. Anything less is surely going to damage trust, which is something Firefox can ill afford when it's already in danger of being reduced to a niche product rather than a mainstream browser.
I was with you up until you said someone should be fired.
The fundamental problem here is the system (code signing.) It's a political thing with security being the excuse. They want control of a platform for business reasons.
Oh relax. A cert expired. An intermediate cert at that...
This has probably happened to every major cloud provider and countless companies at least once. Certs are hard.
Should Mozilla have had monitoring on their cert expiration? Yes. Will they after this? Probably. Is any one person ever at fault for something like this? No.
Firefox is an open source project. You're welcome to contribute and make things better.
> Oh relax. A cert expired. An intermediate cert at that...
Everyone's extensions broke. Including security ones. Including the ones bundled into the TOR browser. And end-users can't fix it. Because Mozilla decided that it was too dangerous to let users choose what extensions to run for themselves. This is an excellent moment to be upset.
Being upset is ok! I'm not particularly happy that I can't just override the certificate check on stable. But demanding someone get fired is just pointlessly punitive.
Alternatively, that person (if they exist) has gotten the best lesson in institutional certificate hygiene rules money can buy. They got their mistake potentially added to hundreds of companies playbooks so it can be caught.
Honestly that's one of the most successful things you can expect out of a failure of this magnitude.
Hopefully management being fired. This reeks of management not letting the technical team automate something or other bad decision making that lead to this. If one person was in charge of it and they messed it up, that is as much the fault of whomever gave that important task to only one person as the person making the mistake. I don't want the low-level person punished, I want the one who put them in the place to be able to make such a bad mistake without any sort of redundancy or contingency plan.
My extensions are also still running. Comments on the bugzilla bug restricted so adding details here and figuring that if this is useful someone can forward it to the right people
$ date
Fri May 3 22:45:22 EDT 2019
$ date --utc
Sat May 4 02:41:47 UTC 2019
$ firefox --version # Installed from arch repositories
Mozilla Firefox 66.0.3
1556920447 is unix timestamp Fri May 3 21:54:07 UTC 2019.
Edit: I think I know why. It checks the signatures daily, and the timing works out so it hasn't checked since the cert expired for me. Just luck, it will break within the next 21 hours for everyone. From the source code:
Copying a potential workaround here from my lobste.rs comment, not really tested obviously
If it hasn’t broken yet for you, I think (but I’m not very much not sure) setting that preference to 1556940100 should keep it working until 24 hours from now. And if you keep updating that value every 23 hours to the output of date '+%s' until it is fixed via a firefox update it should keep working forever.
I think you need to restart the browser as well after updating the preference for the above idea to work.
So that's why I've been struggling for the last hour. It's crazy that it won't even let me I stall extensions from xpi file, even with extension signature checking disabled.
Sure, anyone one can make mistake. We have seen big companies make stupid mistakes too. But this is Mozilla we are talking about. How this slipped by is beyond me.
This is why users need to be in control of their own computers. Why can't I tell my copy of Firefox to ignore the certificate? Why can't I sign my own extensions?
Mistakes happen, it's okay. But users should be empowered to work around them.
It's always felt somewhat unsettling that, although the Internet is decentralised, a small number of large organisations have a surprisingly large amount of control over the software which the majority of users view sites with. I don't like this concentration of power --- even if you think Mozilla is benevolent, it's not immune to making mistakes; and the more power it has, the bigger the consequences.
Before the forced code signing, before the automatic updates, Mozilla or any other organisation's mistakes would not have such dramatic effects; now, they have the power to basically break almost all their userbase nearly instantly, and that is what worries me the most.
> Why can't I tell my copy of Firefox to ignore the certificate? Why can't I sign my own extensions?
The issue is that if you leave any sort of lever that reduces security, it will be abused by bad actors. This is why browsers are having ever decreasing ways to bypass security and have full access. It is annoying, but at the end of the day, protecting 99.999% of the users trumps what us power users want.
It is horribly paternalistic to advocate for keeping users ignorant, unlearning, and --- dare I say it --- easily manipulated.
I will refrain from mentioning again that infamous Franklin quote. I am frankly very fucking pissed off by this authoritarian walled-garden trend, and vehemently oppose anyone who helps this industry put the nooses around the necks of others as well as their own.
I'm not understanding the relationship. Of course users aren't going to understand all the underpinnings of how software works.
I do think that in the future, it will be imperative for everyone to have some level of technological literacy above what is currently the average. And I'd like to work to get to that point, instead of taking all the tools away because they're too dangerous.
Also, sensible defaults are good! Hiding dangerous settings is also good! What's not okay is making those settings completely unavailable. At least in Firefox's case you have the option to recompile the source code, but that should not be the only recourse...
"don't run privileged code from people you don't trust." Is both critically important to understand for anyone using a network connected computer and not at all complicated.
If we're going to be authoritarian I would rather ban anyone who doesn't understand that from connecting to the internet then have a broken walled garden.
> "don't run privileged code from people you don't trust." Is both critically important to understand for anyone using a network connected computer and not at all complicated.
That is absolutely complicated for the vast majority of the world's internet users. No one else is my family would understand what the hell "privileged code" means and shouldn't have to.
The statement can be simplified down to "don't run programs downloaded from random websites which ask for your admin password."
Adjust the qualifier at the end depending on your platform. On Windows, it might be apps that present a UAC dialogue—or maybe just remove the qualifier, since Windows doesn't do much sandboxing by default.
>I still don’t want to have to understand everything I ever touch
If you don't understand it, don't touch it. The default settings should work for most users. There can even be a warning against touching without understanding, like with Firefox's about:config. The offensive thing is preventing users from touching even if they do understand.
The difficulty is in how to keep them available to end users while keeping them unavailable to malware and bad actors who post "helpful" advice or publish temporarily useful addons that get updated to malware.
I'm not disagreeing with you, but the right mechanism is not straightforward to figure out, and you'll always be in a game of cat and mouse. One that sucks resources from whatever other useful stuff you might be spending your (or Mozilla's) time on.
The issue here is that this wasn't done in a vacuum. Other software vendors were secretly and deceptively installing extensions that were tracking everything users were doing online.
Okay, figured it out. Thank you for giving me the search term "Unbranded Firefox"—I didn't realize this referred to something specific.
Unbranded Firefox is actually a specific version of Firefox distributed by Mozilla, which allows you to disable extension signing requirements. I am very glad to see that they offer this, and I will be using it from now on.
Consider the recent news stories about the Boeing 737 Max. Boeing added an automatic system to an airplane, and then didn't give users (the pilots) a way to disable that system. This worked out great while the automatic system is working properly. When the system broke, well, we all know what happened.
If we're going to assume that software is right and the user is wrong 100% of the time, then the software needs to actually be right 100% of the time. Unfortunately, our software isn't that robust, and it never will be.
It doesn't have to actually be right 100% of the time. The balance of downsides and upsides of any chosen solution just have to be more palatable than those of whatever alternate implementation you're considering, with a tradeoff between 100% correctness and ability to be implemented before the heat death of the universe being one of the axes to be considered, as well as the level of benefit provided over your whole user base.
In this case, dropping the extra control/ignoring power users is probably saving a lot of non-power users from shooting themselves in the foot in the vast majority of cases. Pilots (should be) 100% power users. The average operator of a browser is somewhere on the opposite end of the spectrum.
Any real system will have things go horribly wrong for some subset of users on a regular basis. It's impossible to be all things for all people for all situations, so you have to choose your battles.
> The issue is that if you leave any sort of lever that reduces security, it will be abused by bad actors.
As you can see in this discussion, there already are some obtuse ways to disable/ignore the signing. It's just way worse if people have to disable the signing instead of adding a trust for their own certificate, so that only mozilla and user's addons are truste instead of all the malicious garbage out there on the web.
Fortunately, it is no longer necessary to run malicious software on user computers. With latest "advancements" in Firefox security everyone can publish malware directly in Firefox addon center [1]. No review needed!
"We accidentally uploaded all your HTTP requests to our servers, but we will definitely fix that in next addon version!~"
Makes me think of this ticket no work around for the error other than closing the browser and editing a file or modifying the browser and recompiling. No options at all in the advance settings.
It's stuff like this that makes me unhappy with mozzilla. User's who know what they are doing should be permitted to do so. Warn them here be dragons or whatever, but it's ultimately their choice.
You can, though, on both accounts? Use the dev version of Firefox and you get the power to ignore addon certificates, and you can already self-sign your own extensions as much as you like?
I just ran into this. All my extensions were immediately turned off in the middle of a browsing session, from UI conveniences to important safeguards, with no apparent option to turn them back on again.
Warning about something that can't be verified is one thing, but automatically shutting things off -- particularly things that could affect security and privacy -- is a Windows 10 level of unacceptable interference.
And there are often bootstrapping problems where e.g. the push that distributes the new intermediate cert to clients is rejected because of the same expiry issue.
A new certificate can be generated if you have to in a couple of minutes, sure. Of course, you probably defined some procedures to do it properly and securely that require more time.
Then the issue becomes: how to get the new certificate to a few hundreds million users?
If it was a certificate on some server, just replace it there, done. Client software will just pick it up. But not here. A copy of the certificate is shipped in every add-on package file. Oops. Now you have to re-sign all add-ons with the new certificate. And get those resigned files to the users.
Essentially it works like this (which is a slightly modified jar/apk signing mechanism):
- An add-on package is a zip file and other than the actual files there is also a list of known-good hashes of those files in a file called "manifest.mf" in the META-INF folder
- Then there is a file "META-INF/mozilla.sf" giving hashes of "manifest.sf"
- And finally, there is "META-INF/mozilla.rsa", which is a DER-encoded pkcs7 signature and two certificates. The signature verifies "mozilla.sf" was not tampered with and still is the same as when it was signed by mozilla. Which in turn verifies the known-good hashes are still proper.
- The signature is made with a generated certificate, the first one included in "mozilla.rsa". E.g. "CN=uBlock0@raymondhill.net" in case of uBlock.
- The "CN=uBlock0@raymondhill.net" certificate was issued by an intermediate certificate "CN=signingca1.addons.mozilla.org". This "CN=signingca1.addons.mozilla.org" is the second certificate in mozilla.rsa. It says "Validity Not After : May 4 00:09:46 2019 GMT". Oops. This is where the chain breaks now!
- "CN:signingca1.addons.mozilla.org" was issued by "CN=root-ca-production-amo". This root certificate is baked straight into the browser and not part of mozilla.rsa.
Therefore, it is not enough to issue another intermediate certificate (e.g. "CN=signingca-number-two.addons.mozilla.org"), but you have to actually generate a new "CN=uBlock0@raymondhill.net" (or whatever) signed by this new certificate, put those two certificates and a new signature of "mozilla.sf" based on those new certificates into a new mozilla.rsa FOR EACH add-on and ship updated add-on files.
PS:
Try it yourself... Extract some addon package (it's a zip file). Then:
openssl pkcs7 -in META-INF/mozilla.rsa -inform DER -print
The root certificate is also part of the mozilla.rsa. Addons have it included, it can be extracted and then imported into the browser Certificates > Authorities and it will be used to validate addons, including those which are not updated.
This only makes sense if you really need to fix it while the browser is running, otherwise you can simply restart it after the new certificate is imported.
There's a workaround that involves going to about:config and setting xpinstall.signatures.required to false.
However, if you're running the Stable or Beta version, it will only work under Linux. On Windows and MacOS you'll need to download Nightly or the Developer Edition.
To fix this on MacOS I did the following:
1. Downloaded and installed Firefox Nightly
2. Ran /Applications/Firefox\ Nightly.app/Contents/MacOS/firefox-bin --profilemanager
3. Changed the profile to "default" so my normal Firefox profile would be used
4. Started up Firefox Nightly, opened about:config, then set xpinstall.signatures.required to false
Not sure if it's a good idea to use my default profile in Nightly. It might be a wiser idea to copy it instead.
What timezone are you in? I'm in UTC-4 (Detroit), and haven't seen any problems so far. (Also running Nightly on Arch Linux - I haven't made any previous changes to the addon signing either)
To clarify, by 'not working' I meant none of the addons with signing issues are re-enabled after changing xpinstall.signatures.required. I might have wrongly assumed this would happen. However, I tried installing a new addon I had never installed before and that works, but reinstalling one that I had previously installed still doesn't, even after uninstalling it (uBlock Origin).
My timezone is America/Los_Angeles.
EDIT: Sorry, I'm dumb. I actually have two versions of FF installed and I chose the one that wasn't Nightly.
Upgrading your profile from Release to Nightly, which occurs automatically when you open it with Nightly, is a one-way irreversible step. This could prevent your profile from being used with Release without crashes, or lose profile data such as bookmarks or saved passwords when later used with Release, depending on what work is underway in Nightly and if it happens to be backwards-compatible. Be sure to backup your profile if you choose to switch channels.
Note: I am told that Developer channel uses a separate profile, but there are instructions below showing people how to override that, at which point this warning becomes relevant once again.
That’s a good point. However, some of the instructions below specifically tell people how to force any channel onto using the existing Release profile. I’ll update my post.
And I told the developer edition to use my regular profile because that's the one that has all my settings and add-ons and I didn't realize the risk was there. Guess at this point all I can really do is hope and cross the bridge when I get there.
Yes, the risk remains. If I read this right (from my phone), Release is 66, Developer is 67, Nightly is 68. This isn’t guaranteed to be a problem, but it’s not guaranteed okay either. YMMV.
FWIW I started using beta, nightly and the old "UX" channel, first on Mac and then on Linux, and before I knew it could be a problem I switched between them with the same profile all the time. Maybe there were subtle bugs I wasn't aware of, but nothing I ever noticed.
I haven’t run into any issues in a while, but you only have to get hit by lightning one time to lose your profile data. Best to be consciously careful about it.
I do agree, and I'm more careful now. Always keep a backup, at the very least. I now symlink ~/bin/firefox to nightly because some apps seem to have it hardcoded to open "firefox" rather than what's set as default.
Looks like it would have been better to copy the profile instead. I managed to get most of my profile back using Firefox Sync, though for some reason it didn't transfer across my preferences and I had to redo those.
It is probably safer to use an unbranded build with the same version as the currently installed Firefox (take note that it will not update). Page with links to the latest release builds: https://wiki.mozilla.org/Add-ons/Extension_Signing
And frankly, this an extra absurdity on top of that. If you’re going to require signatures for all extensions, regardless of user preference, shouldn’t you be keeping an eye on the signing process?
Why does Mozilla do this? Same with removing the option to not update. Why not let users choose (in the case of update maybe with an about config setting)?
Because (stable) users are dumb, are easily manipulated and can't be trusted. Thus the mothership has to be in control for the greater good.
They also argue that enduser computers are already effectively "compromised" from a mozilla perspective because adware runs installers with admin privs and thus could insert things into the program folders. Thus anything the user can do adware could do too and therefore they can't give them any choice.
They put it in nicer words though.
To their credit, you can opt out but only if you switch to dev edition, nightly or custom builds, which either is a one-way road since downgrades corrupt profiles or tedious because you don't receive auto-updates.
But what they should really have done is allowing additional signing roots. Even secure boot does that.
I get the ostensible justification, but attacking this way requires the user to dig into the obscure dev settings and load an xpi from outside the browser[1]. Is there even one case of a user compromised that way?
[1] or at least they could have allowed that as a compromise
I updated my previous comment. They say there exist crapware installers that use elevated privileges that do inject stuff into the browser and that's why we can't have nice things, yes.
But I disagree with their value tradeoffs. They want to add a little "protection" - which is really flimsy since there is no privilege separation - for users who already compromised their systems with adware at the expense of the freedom of everyone else.
I'm totally fine with software already running on my machine being able to install addons into my browser. It can also already install a keylogger and record the screen, what's the big deal?
This sounds like a threat model and mitigation developed by a college intern.
How, exactly, is a user land application going to protect itself from modification by a computer admin? I think DRM, anti-virus, and os vendors everywhere would love an answer to this.
This threat model completely fails to account for live patching, trusted cert root modification, dll hooking, etc. Either the Mozilla security folks are incompetent / winging it, or this isn't the real reason.
It is not possible for a user land application to prevent root processes from hijacking / modifying it. Such protection requires the protecting mechanism to run at a higher level of trust / security ring than the attacker.
Is it perhaps a good time to remind folks that the same thing could happen to all your "secure" HTTPS websites that are completely unavailable via HTTP, where the only thing served over HTTP are the 301 Moved redirects, even for sites that don't collect any user information at all, and only serve static and public content, which really hardly benefit from the mandatory encryption?
Or is HTTPS / LetsEncrypt too big to fail? HTTPS still always a good choice? I see…
For a piece of open source software you really have very little control with firefox. It really sucks that the alternatives are worse.
This, likely for almost all of their users, creates more of a security problem than signature checking actually solves. For me noscript no longer works which is (IMO) a critically important extension (between mozilla taking away the disable javascript button and spector.)
OTOH, it's opensource, and you can re-compile it in like 20-30 minutes with whatever changes you desire. So you have the control. You can probably even add some hacked up support for multiple signing certificates (and add yours there), if you tried.
It's just that it's more work than having what you want implmeneted and maintained by others.
Firefox requires extensions installed via their "store" be signed with a certificate to make sure they're actually from there. That certificate has an expiry date. It expired, so now all of its signatures are invalid -- and Firefox no longer trusts the associated extensions.
A minor added detail is that it wasn't the leaf certificate that expired. I've heard that that would have been handled properly. It was an intermediate cert, and I guess that possibly wasn't fully taken into account? (This is 3rd hand knowledge and speculation, note.)
They have acknowledged the defect and are working on a fix. While this is a severe impact, I am still with Firefox. The are enough alternative browsers to tide over the problem for now. The fact that alternatives exist is the reason why we should support projects like Firefox.
Just curious, should we expect that the fix (issuing a new signing cert and re-signing all the addons and whatnot) will result in the addons being automatically updated and re-enabled? They certainly seem to have streamlined disabling the addons, I wonder if it is equally simple from a users perspective to bring them back. Also now wondering just how hard their network/CDN is going to get slammed when those new re-signed addons go live and every user automatically redownloads them.
that's kinda the problem. there's plenty of reasons to be "with" firefox still, but you shouldn't need reasons other than it's the best browser. when it starts requiring loyalty to be a user, that's a big problem.
For me, it is the best browser. Yes, this is a big fuck up, but it's not like this has caused me material harm. It's easy for me to switch over to Chrome until this is fixed, and I doubt the same mistake will be repeated in Mozilla.
I expect perfection from plane and car manufacturers, and I pay for that. My browser, I can live with an occasional hiccup.
This helped me discover Firefox's Content Blocking setting, which is set to Standard by default, but now I set it to Strict. Works better than an ad block!
I thought that the way signing works in general is that the signer issues a certificate for the thing being signed (domain, code, whatever) that contains identifying information for the thing signed (host name for an SSL certificate, checksum of the code for a code signing certificate), the valid from and valid to dates of that certificate, and assorted other information, and either a reference to or a copy of the signer's certificate, and it signs the whole issued certificate with the signer's certificate.
Someone checking the signed thing is supposed to consider it validly signed if:
1. The date is in the valid range for the signed thing's certificate,
2. A check of the signature of that certificate against the signing certificate passes,
3. The signing certificate is recognized as being from an issuer considered trusted by the checker,
4. Neither the signed thing's certificate nor the signing certificate have been revoked, and
5. The signing took place during the valid date range of the signing certificate.
Note there is no "the date of the check is in the valid date range of the signing certificate". A signing certificate expiring should not invalidate things signed by it. It should just prevent signing anything else with it.
So why is a signing certificate expiring for Firefox breaking already signed extensions? Shouldn't it just be stopping new versions of extensions from being signed?
If a signing certificate expires and is stolen, it can backdate signatures. On the theory that expiration is useful because either people keep poorer track of key material over time or algorithms get weaker over time (which is not an unassailable theory, but it's a coherent model), you want expiration to prevent future use just as if it were revocation. Because you can't trust the date of a possibly-forged signature, you have to check the current date.
The model you're suggesting is closer to the "timestamping" one commonly used in code signing (IIRC Windows and Mac both do this) where a third party that's particularly trusted to handle key material well long-term gives you a second signature over the message "I saw this signature at this time" (effectively they are analogous to a notary or witness for real-world signatures). Then you can trust that signatures from expired signing certs were actually made in the past, and not by an attacker who got hold of the key. That is, without timestamping you have no proof of #5 in your list.
(I suppose you could do this now for the SSL PKI with Certificate Transparency logs.... it isn't exactly what they were built for but it's probably sound.)
As noted, in practice, without additional info, there's no way to tell when a signature was created and so all signatures die when the signing cert expires.
That said, existing signatures that have already been verified, for existing extensions, should still be trusted.
954 comments
[ 3.0 ms ] story [ 331 ms ] threadI think this fail-closed behavior is more of a security issue than the one it is trying to solve. All of my security add-ons - Privacy Badger, NoScript, Decentraleyes, and many more were disabled. Even worse, it happened without notice to the user.
One moment I was browsing the internet (just barely) secured by these add-ons, and the next moment, all of them disappeared (without warning) and I only noticed when I saw my password manager was missing.
Probably the correct behavior is to have some sort of semi-annoying popup when it expires, and then only a week later do the full blocking. You need to strike the right balance of making it annoying enough that it can't be ignored by everyone (otherwise you just have the exact same problem, just delayed a week) and that fear of it happening is a sufficient motivator to stop people lazily relying on the grace period, but also not too annoying that it makes a lot of people quit. You also want to avoid permission fatigue.
With this reduced threat model, it's easy to simply keep existing pre-installed extensions available, and disable updates. Your only problem is if a pre-installed extension is malicious or has a vulnerability, it will remain.
This is why a signature can also be accompanied by a trusted time stamp which can confirm that the signature was made while the certificate was valid.
This is the common way to sign all Windows software to avoid this exact kind of problem.
Yes, that implies this is a known and solved problem. It’s embarrassing for Mozilla to not have prepared for this.
On a positive note, it's been a while since I browsed without a lot of extensions. Ads are still annoying and I noticed some extensions apparently had more of a performance impact than you'd hope.
I like FF, don't get me wrong, but this is going to absolutely fucking destroy user trust in Mozilla. This kind of incompetence, on a browser scale, is breathtaking.
I'm not sure this cert is used with the PW manager?
Firefox password storage isn't even encrypted by default, last I checked.
Example: https://subdavis.com/Tusk/
While I agree with you two assuming the bridge is over water and not too high, there are real consequences that cannot be reversed. I cannot unsee the ads I saw in the past few minutes before switching to nightly.
This should not be possible.
Worse, had they not taken the paternalistic, nanny-like stance that you can't even disable the signing checks, I could roll out a script that would make this a non-issue for my users. But no, thanks Mozilla for ruining my Monday.
Might not be the most substantive comment I could possibly make in the circumstances, but I'm pissed. The only appropriate response feels like a string of infuriated profanity directed at their incompetence and decision-making.
Luckily, I can still type "make install" without debian informing me that "random_dangerous_untrusted_code_from_interwebs" is not approved.
I think we'll all live. No need for the chicken little act.
My concern is around non-technical users (the group, mind you, that Firefox has been spending marketing dosh on courting recently with Quantum and all) who don't have as compelling reasons for not just switching back to Chrome. In the last hour, I've gotten several phone calls from family members asking me why the browser I convinced them to use is broken. I don't have a good answer, because platitudes about surveillance and muh freedoms don't count for shit when your grandma just wants to get rid of the ads on the local newspaper site.
I'm personally going nowhere and deeply appreciate Mozilla for all the work on FF and friends, occasional fuckups aside, but I don't think this is going to be a non-event for a browser that's been desperately fighting to regain market/mind-share.
Very much this. People are often too quick to forget who their customers are and what they really want.
We check our warning system (set up to detect suspicious logins, incidentally also catches any users who've been locked out because they forgot their password), and his last login attempt took a total of two tries.
Sadly, I have to agree that this feels like a big blow to user trust.
User trust is not really just about respect or values; it definitely also includes things like performance and reliability. The average user, right now feeling powerless, might even feel anger towards Mozilla for this - after all, they already downloaded the extension, why would they all just stop working behind their backs? They don't understand what CAs are or why certificates expire. People don't frankly care what place your heart is in when they are angry about something. Perhaps people are being dramatic, but that's normal. People are pretty darn dramatic about Chrome, too.
Meanwhile... I use Firefox everywhere, and I've lost my password manager, adblocking, security-related extensions, etc. all in one go, and the only solutions I'm aware of involve disabling extension signing. Gotta admit, even though I will probably continue using Firefox after this, that it certainly is a bummer.
And yet every other major browser vendor has punched their users with far worse catastrophes of privacy, security, ripping away features, breaking features, and general shitheaddedness.
Switching browsers because of this incident is like ordering a burger at your favourite restaurant and one time it comes out without the meat patty, so in protest you switch to a crappy alternative restaurant that has had a long history of health code violations.
I'm glad you have software/vendors you feel you can trust. I definitely don't feel that way about most software anymore. I do think you are being a bit hyperbolic regarding other browser vendors, but to each their own, I don't know what trying to argue about that would solve for anyone.
I think what is a real tough dillema is being sad about nonfunctioning adblocker while working for the biggest internet ads company.
So are you working in chrome marketing department?
I have computers other than my work computer(s.) I, indeed, do not have Chrome or Chromium installed on my home boxes running NixOS. I do not use my work devices for personal web browsing. I'm currently posting this message with Firefox 66.0.3 on NixOS 19.03.
>I think what is a real tough dillema is being sad about nonfunctioning adblocker while working for the biggest internet ads company. > >So are you working in chrome marketing department?
I'm a software engineer. I'm also over at Github:
https://github.com/jchv
I work at Google because it's an excellent place to work. I'm far from elite; I didn't finish college (couldn't afford) and I grew up in the suburbs of Detroit, so being able to work at any large SV company is something I don't take for granted. I don't think any single employee can claim to love 100% of the things Google does, and that's fine. Nobody is required to.
As for why I would use an adblocker, practically speaking it's both for reducing annoyances and increasing security. Malware (and 0days!) delivered via ads is not unheard of, sadly.
I think HN penalizes non-link posts (or people are flagging it because they think it's just someone asking for tech support).
Still, this type of oversight seems all too common even in large companies. I remember several cases from Fortune 500 companies in the past few years alone. What would be a good way to automate checking for them? Has anyone developed a tool designed specifically to avoid certificate expiry disasters?
You don't ship the signing keys with the certs, as that would be bad. ;)
My employer (Kynd.io) currently monitors public web sites for customers so we can flag e.g. "Hey this site cert expires in a week! If it's dead probably just switch it off, otherwise renew the certificate" and we're in the process of integrating CT but mostly so we can say "You already have a newer cert but need to go install it" in our How To Fix instructions.
I've never seen a bulletproof solution for organizational tasks that need to be done yearly.
If someone's in charge... and both they and their manager happen to leave in the same year... and whatever system they had in place to remember (probably their personal calendars) is gone... and the manager's manager has 1,000 other things to remember...
...how does an organization ensure the task still gets done?
With something almost stupidly simple and low-tech: checklists.
(I'm reading "The Checklist Manifesto" right now, and the points it makes seem to fit perfectly with everything you mention.)
Once you do this, the only checklist that matters are procedural checklists to add a new client or new cert to the renewal notification list. When you use a standard group email for all cert purchases, that one becomes tough to miss.
In my 7 years of being involved, we never missed a cert renewal with this process for ~300 client sites with multiple or wildcard certs.
I'm willing to bet Mozilla already does something like this but an engineer didn't set it up correctly for this certificate.
edit: not Europe, just UK and Japan apparently: https://www.zdnet.com/article/ericsson-expired-certificate-c...
It took like two days before there was any kind of fix available, and they couldn't even roll it out automatically because the expiration had also disabled the auto-updating.
Certificate expiry really only exists to make money for CAs. It doesn’t solve any security problem that CRLs don’t already solve (and solve better). There’s lots of unsolved problems relating to ‘how do you make a reliable PKI’, but cert expiry is really just an unrelated business requirement for CAs.
* Very short lifetimes get people to automate, preventing problems where one cert lasts long enough to lose the institutional knowledge around it.
* CRLs don't work. For performance you don't want to check for a revocation in serial with the request, and you don't want to block all browsing if the revocation list server is down. Revoking a cert will cover some users, but lots will still get "https://" and no warnings.
CRLs require maintenance and distribution of a list by a 3rd party. Creating an accurate, all-inclusive CRL of all website keys that your browser should reject is far, far from easy. (Case in point: "how many web sites are there?" Is not an easy question. )
Properly propagating such a list to any browser that might need it is another daunting task - less than 100% propagation means end users are exposed to security risks.
Certificate expiry is much more elegant: the client can check the certificate's validity himself, without relying on input from 3rd parties.
If certificates didn't expire, CRLs would (by now) be huge and growing enormously every day. They'd be so big that by the time you'd have downloaded one, it'd be outdated.
But, this sharing carries a cost for user privacy, if I shard certs 16 ways then each CRL download gives me 4 bits of info about which sites you were visiting.
OCSP effectively takes this to the extreme, each lookup is tiny because it's just for one cert, but it gives away exactly which cert you cared about each time.
Failing closed means failure of a third party immediately breaks your site. Failing open means a MitM can simply block the CRL check.
OCSP stapling and the 'must staple' header are a lot better for privacy, and OCSP responses have some validity so at least a 5 hour outage of your CA doesn't bring your site down immediately.
It is still vulnerable to a DOS and trust on first use though.
Apache and nginx both shipped OCSP stapling implementations that are very bad, awful enough that for almost anyone I'd say "No, don't enable that" rather than try to explain how they need to use it and get them to a place where it's useful and safe. Adam Langley wrote years ago about how to do this correctly, and there does seem to be a little bit of movement in the correct direction at Apache, but the situation remains pretty poor.
For any method, fail closed is user hostile and often a DOS vulnerability whilst fail open is another way for an attacker to use a revoked cert.
This is a big issue with on-line methods like OCSP as a MitM using a bad cert can probably block OCSP traffic as well.
CSLRs grow out of proportion, and leak information to the outside world.
Cert expiry serves as a backstop to these other revocation methods, and as a bonus ensures that simply forgetting about a cert cannot bite you 10 years later.
Short lived certs are quite obviously better from a security perspective, but the security difference between a certificate that expires in five years, and one that expires never is irrelevant.
The information leakage of CRLs is stating to the public that a cert needed to be revoked.
Obviously, a compromised cert that will expire in 5 years is horrible. However, a non compromised cert you are no longer using that will never expire is more off a risk than a disused cert that will expire in a year. Not to say you should leave the one year cert lying around. However, there is no desire to put the one year cert on a pre-shipped CLR.
Not sure that's viable for a signing certificate like this, but that's the way to solve it for the web PKI.
The problem comes if your keys ever get compromised or cracked all your historical traffic becomes vulnerable instead of just the most recent window.
All it being new means is that depending on your risk ratio you need to decide whether updates to the software need testing or whether you need to invest in your own solution - or, how about just wait until it matures and keep the old process until then.
Waiting doesn't invalidate the premise either. It just means you lack the resources to implement it safely and that's ok.
Register a domain, get a certificate lasting forever, let the domain expire and somebody buy it. Then somehow redirect all or part of the traffic to that domain to your own server with a valid certificate. Chances are that few people will notice something has changed in the details of the certificate.
However you'll have left traces all over the place: credit cards, phone numbers, etc.
Very few things check revocation, unfortunately - it puts an extra hop on the fast path of connecting to a server. OCSP stapling is pretty much the only thing a browser would care about - having the server fetch a signed OCSP response that is good for a limited period of time (say, hours), and send that along with the certificate during negotiation.
Or, you could just have the server fetch a certificate thats good for a limited period of time.
[1] https://tools.ietf.org/html/rfc8555#section-7.6
Firefox still does normal OCSP requests, Chromes does not. So if you are a Chrome user, to my understanding, there is now way to know if the server certificate was revoked or not, other than OCSP stapling together with OCSP Must Staple. Additionally, both Chrome and Firefox ship a list of revoked certificates, but it may not be updated quickly enough and as far as i can tell it mostly contains roots and intermediates.
The problem is that Tree Style Tabs relies on userchrome.css edit to hide the tab bar, and when TST is forcibly removed there is no way to access the tabs, because that edited userchrome.css is still there. This is very disruptive. At least with the pre WebExtension addon TST itself hides the tab bar, so if TST is removed then the original tab bar comes back on automatically
https://github.com/eoger/tabcenter-redux/wiki/Custom-CSS-Twe...
LetsEncrypt renewal is supposed to be automated. [1]
I know of a company that hosted blogs for thousands of customers. They used LetsEncrypt, but the CTO considered automatic renewals a possible security risk, so they did it manually. Problem is, the expiration happened in a weekend and they "forgot" to update the certificates before that. Suffice to say that the next Monday wasn't pleasant. They automated after that.
[1] https://letsencrypt.org/about/
But now that you mention it, I wonder what's the opinion of security experts like tptacek on cert renewal automation.
To mitigate that you end-up building a series of privilege-restricted jobs flowing from the DMZ back into the internal network. And maintaining that might be more complicated than just manually renewing, depending upon the processes and architecture of the company.
I run Caddy (which uses acme-go/lego as its ACME provider) as a non-root user with no access to /etc at all. It seems to be running fine.
If you don't trust it your automation, you rotate the keys manually, as you would normally.
There are no valid reasons to throw the baby away with the bathwater.
acme-go/lego doesn't use HTTP validation unless you disable just about every other form of validation first. TLS-ALPN validation is much more likely, so port 443.
That said, it is very easy to allow software to bind to privileged ports without providing it root access; this has been solved for a very, very long time.
You (normally) don't want downtime in your website, so you just let your regular webserver serve the acme challenge instead of stopping it.
I used manual renewal for LetsEncrypt for about 4 websites on other shared hosts & renewing them every 3 months was a pain; had to keep reminders and schedules just not to miss renewals until I synchronised their renewal schedules to batch (manual) renewing them.
I had automated renewal for 1 website on a cloud server, it was a one time effort, I never had to bother about SSL cert for that site and the most favourable of them all.
[1] https://caddyserver.com
[2] https://httpd.apache.org/docs/2.4/mod/mod_md.html
https://github.com/icing/mod_md/wiki/Migration
https://traefik.io
We issue certificates automatically if none is existing when connecting to a website and renew the certificates in batches 30 days before they expire. When renewing, we merge certificates/hostnames into bigger certificates with 90 hostnames so we don't have so many moving parts.
If renewal would break, however (as it did once or twice before), nothing bad would happen because on page load there would be a new certificate issued.
Another use case for the app I am developing! The basic idea: You can enter an item (i.e. "MyOwnShop Cert") into the list. From that time on, it will be tracked how much time passed since the item was entered or renewed (by clicking the renew button). The item with the longest time since entering/renewing is at the top of the list.
Compared to schedules and reminders it has the advantage that the item is not out of our mind once the reminder or schedule pasts. It just sits there dutifully and its timer keeps increasing.
I use it for keeping up with middle-term contacts ("Wow, I have not written Carl for 3 weeks?") and health-related issues. Logging in stuff that easily spoils would be another use case. And, apparently, cert renewals :)
Their FAQ [1] recommends exactly that: renewing every 60 days.
[1] https://letsencrypt.org/docs/faq/#what-is-the-lifetime-for-l...
similar to saying that you could do it with a raspberry pi
Of course that is also a kind of misconfiguration. The site has Debian security auto updates on but certbot is not among them. It should be forced to be updated. Furthermore there was no monitoring of errors in its log file.
Still it's not as simple as one believes Letsencrypt to be.
At the end of the day, someone needs to verify that new certificates gets acquired and installed before the old ones expire. Automation makes acquiring them less tedious, but not much for making sure someone pays attention.
https://pypi.org/project/check-tls-certs/
I run one daily from cron and have it email me a report with the days to expiration for the certs I’m responsible for, even for certs that auto renew. I don’t filter the email. Daily is not too frequent for it to go to my inbox, but frequent enough that I’ll notice if it doesn’t mail me. YMMV.
Discipline? Experience? PIP?
Certificates that aren't from the Web PKI almost invariably won't be logged. Most logs explicitly refuse everything except certs from the Web PKI so as not to be burdened storing garbage. So this won't find certs issued by the custom OpenSSL CA on that one guys Linux laptop.
Not all Web PKI certs are logged. There is no BR obligation and no root store programme rule that requires logging. The only things in place that strongly encourage logging are the Chrome and Safari policies. For systems that aren't designed to be accessed with a web browser or, much more rarely, enterprises that have persuaded themselves only IE is authorised anyway, the certs might deliberately not be logged. Yes there are (small) CAs doing this in the Web PKI, on purpose, in 2019.
(But seriously, sure you’re right but for my audience (which is essentially Latacora’s and HN’s), CT is fine.)
[0] https://ctadvisor.lolware.net/
Is anything more than a calendar reminder on the phone of someone important enough to shake the Earth and get it fixed For. Certain. needed? Like, say, the CEO, CTO, and CFO should at a minimum get a notification so they can ask if the refresh was done when necessary?
Corporations are often not very good at putting the right people in these roles but good ones are invaluable. Since the Marvel Universe is everywhere, Pepper Potts is the archetype in that setting to give you an idea of why you'd need people like this. Tony Stark would be "too busy" to renew the certificates, but Pepper would make sure it gets done.
Also, Chrome is not immune to "crashes for everyone at the same time" bugs. Like that time when the start of daylight saving time made it crash for a full day (a quick search tells me it probably was https://bugs.chromium.org/p/chromium/issues/detail?id=287821).
What else would you expect for auto-updating software that relies on the internet to work? It's a monoculture attached to a firehose of disease.
This is exactly the same as "pushing out a security fix to all users," except it apparently wasn't intentional. You can't have one without the other.
The npm self-signed certificate fiasco of early 2014 springs immediately to mind.
Taking a look around, you’ll find lots of service providers, or tools you could use. But the main issue is all they do is tell a human being to do something, which they can still fail to do. Which is why automating cert rotation (with things like let’s encrypt or ACM) is arguably a better solution than monitoring it.
Not perfect, but I've added a TLS certificate extraction tool into a DPI that displays all visible certificates ordered by expiry date.
One could then mirror all one's site traffic to it and let it run in the background. Coupled with some alerting tool it would catch most of those cases I guess.
I could polish the tool a bit more if there is some interest, but anyone could do it as well.
See
https://github.com/rixed/junkie
and more specifically the plugin called 'sslogram'.
Quixotically: make cert failure a randomised number, linearly related to how long ago the cert expired. This slowly introduces more and more failures, over a certain “grace period”, which makes the problem less of an extinction level event. It’s not a solution but it definitely would help.
Edit: wow, downvotes? Care to explain what I'm missing?
I'm referring to traditional code signing, which I assume Firefox extensions are more similar to --- the goal being to ensure that some data has not changed since it was signed, and only the validity of the certificate at the time the data was signed is meaningful; even after the certificate expires, a signature created when it was valid still asserts that the data it signed has not changed.
Without timestamping the expired cert always would have caused problems, even if it was replaced early and correctly: Every add-on would still need to be signed again with the new replacement certificate and shipped to all users. It's not as easy as just replacing the certificate on some server.
Well, this is still what has to happen: replace the certificate, ship that new certificate[1], re-sign every add-on, ship every add-on to every user.
Now, in order to ship new versions of the add-ons, you probably will have to bump the add-on version numbers as well. Which can have further unintended consequences.
[1] Incorrect, see blow; it is my understanding that the certificate in question is baked into the browser itself, with no way to push updates just for the certificate remotely other than shipping an entire new Firefox build. Well 6 new builds: esr, stable, dev, beta, nightly, unbranded. Gonna be a fun night for a lot of mozilla folks... Well, a night is not gonna be enough...
I might be wrong tho, and misunderstood something.
EDIT I was wrong (https://news.ycombinator.com/item?id=19824520), the expired cert is not baked into the browser, just into the add-on package files. No need for new Firefox binaries, after all. Still, they have to resign all add-ons and ship new versions.
They'd better have the best post mortum ever, possibly with someone being fired.
Arguably these two goals are incompatible. :)
Or you fire the scapegoat because of a broken system that allowed one person to make a mistake?
But we're outsiders looking in and don't know what's going on at this point. That's why I used the qualifier "possibly." It's quite possibly it wasn't incompetence.
That's true, sort of. How often do you let people make huge mistakes before you decide that maybe they are just not apt for the position that they've been promoted to and Peter was right? Once? Twice? And unlimited amount, as long as it's never the exact same mistake?
Thanks :)
That might seem rather extreme, but the fact that this situation was even possible was a consequence of a series of bad decisions over an extended period of time about the required behaviour of new versions of Firefox, combined with technical failures that betray fundamental weaknesses in the whole system design. Whoever was ultimately responsible for those failings demonstrably isn't competent to run something of this importance and should probably either implement immediate and dramatic changes to the relevant policies and technical details or consider their position. Anything less is surely going to damage trust, which is something Firefox can ill afford when it's already in danger of being reduced to a niche product rather than a mainstream browser.
The fundamental problem here is the system (code signing.) It's a political thing with security being the excuse. They want control of a platform for business reasons.
No one needs to be fired for a single instance of a particular mistake. If this happened multiple times, then I would be on board with firing someone.
What I'd like to see is a post-mortem, followed by an explanation of how they'll prevent the mistake from being made again in future.
This could have been prevented by someone putting the expiration date on the team shared calendar with a 60 day alert.
This has probably happened to every major cloud provider and countless companies at least once. Certs are hard.
Should Mozilla have had monitoring on their cert expiration? Yes. Will they after this? Probably. Is any one person ever at fault for something like this? No.
Firefox is an open source project. You're welcome to contribute and make things better.
Well no because they won't accept a patch that lets us plebs turn off the signed extension requirement.
Everyone's extensions broke. Including security ones. Including the ones bundled into the TOR browser. And end-users can't fix it. Because Mozilla decided that it was too dangerous to let users choose what extensions to run for themselves. This is an excellent moment to be upset.
Honestly that's one of the most successful things you can expect out of a failure of this magnitude.
Edit: I think I know why. It checks the signatures daily, and the timing works out so it hasn't checked since the cert expired for me. Just luck, it will break within the next 21 hours for everyone. From the source code:
If it hasn’t broken yet for you, I think (but I’m not very much not sure) setting that preference to 1556940100 should keep it working until 24 hours from now. And if you keep updating that value every 23 hours to the output of date '+%s' until it is fixed via a firefox update it should keep working forever.
I think you need to restart the browser as well after updating the preference for the above idea to work.
Temporary work around till the cert gets fixed: set "xpinstall.signatures.required" to false
Mistakes happen, it's okay. But users should be empowered to work around them.
Before the forced code signing, before the automatic updates, Mozilla or any other organisation's mistakes would not have such dramatic effects; now, they have the power to basically break almost all their userbase nearly instantly, and that is what worries me the most.
The issue is that if you leave any sort of lever that reduces security, it will be abused by bad actors. This is why browsers are having ever decreasing ways to bypass security and have full access. It is annoying, but at the end of the day, protecting 99.999% of the users trumps what us power users want.
It is horribly paternalistic to advocate for keeping users ignorant, unlearning, and --- dare I say it --- easily manipulated.
I will refrain from mentioning again that infamous Franklin quote. I am frankly very fucking pissed off by this authoritarian walled-garden trend, and vehemently oppose anyone who helps this industry put the nooses around the necks of others as well as their own.
I still don’t want to have to understand everything I ever touch, even if I could.
I do think that in the future, it will be imperative for everyone to have some level of technological literacy above what is currently the average. And I'd like to work to get to that point, instead of taking all the tools away because they're too dangerous.
Also, sensible defaults are good! Hiding dangerous settings is also good! What's not okay is making those settings completely unavailable. At least in Firefox's case you have the option to recompile the source code, but that should not be the only recourse...
If we're going to be authoritarian I would rather ban anyone who doesn't understand that from connecting to the internet then have a broken walled garden.
That is absolutely complicated for the vast majority of the world's internet users. No one else is my family would understand what the hell "privileged code" means and shouldn't have to.
Adjust the qualifier at the end depending on your platform. On Windows, it might be apps that present a UAC dialogue—or maybe just remove the qualifier, since Windows doesn't do much sandboxing by default.
If you don't understand it, don't touch it. The default settings should work for most users. There can even be a warning against touching without understanding, like with Firefox's about:config. The offensive thing is preventing users from touching even if they do understand.
I'm not disagreeing with you, but the right mechanism is not straightforward to figure out, and you'll always be in a game of cat and mouse. One that sucks resources from whatever other useful stuff you might be spending your (or Mozilla's) time on.
If you want your freedom from reviewed extensions: fine, get an unbranded Firefox, or Developer edition, and you get that.
If you can recommend an fork that allows extension sideloading but is kept up to date, please do so, I’ve been looking...
Unbranded Firefox is actually a specific version of Firefox distributed by Mozilla, which allows you to disable extension signing requirements. I am very glad to see that they offer this, and I will be using it from now on.
https://wiki.mozilla.org/Add-ons/Extension_Signing#Unbranded...
If we're going to assume that software is right and the user is wrong 100% of the time, then the software needs to actually be right 100% of the time. Unfortunately, our software isn't that robust, and it never will be.
In this case, dropping the extra control/ignoring power users is probably saving a lot of non-power users from shooting themselves in the foot in the vast majority of cases. Pilots (should be) 100% power users. The average operator of a browser is somewhere on the opposite end of the spectrum.
Any real system will have things go horribly wrong for some subset of users on a regular basis. It's impossible to be all things for all people for all situations, so you have to choose your battles.
As you can see in this discussion, there already are some obtuse ways to disable/ignore the signing. It's just way worse if people have to disable the signing instead of adding a trust for their own certificate, so that only mozilla and user's addons are truste instead of all the malicious garbage out there on the web.
> Temporary work around till the cert gets fixed: set "xpinstall.signatures.required" to false
https://news.ycombinator.com/item?id=19823879
If there's a privilege level that allows for one but not the other, that sounds like something Mozilla should fix.
"We accidentally uploaded all your HTTP requests to our servers, but we will definitely fix that in next addon version!~"
[1]: https://arstechnica.com/?post_type=post&p=1340459
https://bugzilla.mozilla.org/show_bug.cgi?id=1528738
It's stuff like this that makes me unhappy with mozzilla. User's who know what they are doing should be permitted to do so. Warn them here be dragons or whatever, but it's ultimately their choice.
Warning about something that can't be verified is one thing, but automatically shutting things off -- particularly things that could affect security and privacy -- is a Windows 10 level of unacceptable interference.
Then the issue becomes: how to get the new certificate to a few hundreds million users?
If it was a certificate on some server, just replace it there, done. Client software will just pick it up. But not here. A copy of the certificate is shipped in every add-on package file. Oops. Now you have to re-sign all add-ons with the new certificate. And get those resigned files to the users.
Essentially it works like this (which is a slightly modified jar/apk signing mechanism):
- An add-on package is a zip file and other than the actual files there is also a list of known-good hashes of those files in a file called "manifest.mf" in the META-INF folder
- Then there is a file "META-INF/mozilla.sf" giving hashes of "manifest.sf"
- And finally, there is "META-INF/mozilla.rsa", which is a DER-encoded pkcs7 signature and two certificates. The signature verifies "mozilla.sf" was not tampered with and still is the same as when it was signed by mozilla. Which in turn verifies the known-good hashes are still proper.
- The signature is made with a generated certificate, the first one included in "mozilla.rsa". E.g. "CN=uBlock0@raymondhill.net" in case of uBlock.
- The "CN=uBlock0@raymondhill.net" certificate was issued by an intermediate certificate "CN=signingca1.addons.mozilla.org". This "CN=signingca1.addons.mozilla.org" is the second certificate in mozilla.rsa. It says "Validity Not After : May 4 00:09:46 2019 GMT". Oops. This is where the chain breaks now!
- "CN:signingca1.addons.mozilla.org" was issued by "CN=root-ca-production-amo". This root certificate is baked straight into the browser and not part of mozilla.rsa.
Therefore, it is not enough to issue another intermediate certificate (e.g. "CN=signingca-number-two.addons.mozilla.org"), but you have to actually generate a new "CN=uBlock0@raymondhill.net" (or whatever) signed by this new certificate, put those two certificates and a new signature of "mozilla.sf" based on those new certificates into a new mozilla.rsa FOR EACH add-on and ship updated add-on files.
PS:
Try it yourself... Extract some addon package (it's a zip file). Then:
A sample procedure doing exactly this, and a fix for Firefox <= 56.0.2 can be found here: https://www.velvetbug.com/benb/icfix/
The same procedure can be used on newer versions, but the syntax is a bit different (import cert, go to about:addons, open console):
This only makes sense if you really need to fix it while the browser is running, otherwise you can simply restart it after the new certificate is imported.However, if you're running the Stable or Beta version, it will only work under Linux. On Windows and MacOS you'll need to download Nightly or the Developer Edition.
To fix this on MacOS I did the following:
1. Downloaded and installed Firefox Nightly
2. Ran /Applications/Firefox\ Nightly.app/Contents/MacOS/firefox-bin --profilemanager
3. Changed the profile to "default" so my normal Firefox profile would be used
4. Started up Firefox Nightly, opened about:config, then set xpinstall.signatures.required to false
Not sure if it's a good idea to use my default profile in Nightly. It might be a wiser idea to copy it instead.
Saved me tons of ultimately pointless thrashing.
I OWE you, dude.
My timezone is America/Los_Angeles.
EDIT: Sorry, I'm dumb. I actually have two versions of FF installed and I chose the one that wasn't Nightly.
Note: I am told that Developer channel uses a separate profile, but there are instructions below showing people how to override that, at which point this warning becomes relevant once again.
(See reply about Developer, though.)
The workaround also works if you're running Firefox Extended Support Release on MacOS. Thankfully.
For me missing extensions aren't just an inconvenience. I simply don't browse with JS on. Firefox is dead to me without NoScript.
https://youtube.com/watch?v=taGARf8K5J8
And frankly, this an extra absurdity on top of that. If you’re going to require signatures for all extensions, regardless of user preference, shouldn’t you be keeping an eye on the signing process?
They put it in nicer words though.
To their credit, you can opt out but only if you switch to dev edition, nightly or custom builds, which either is a one-way road since downgrades corrupt profiles or tedious because you don't receive auto-updates.
But what they should really have done is allowing additional signing roots. Even secure boot does that.
[1] or at least they could have allowed that as a compromise
But I disagree with their value tradeoffs. They want to add a little "protection" - which is really flimsy since there is no privilege separation - for users who already compromised their systems with adware at the expense of the freedom of everyone else.
How, exactly, is a user land application going to protect itself from modification by a computer admin? I think DRM, anti-virus, and os vendors everywhere would love an answer to this.
This threat model completely fails to account for live patching, trusted cert root modification, dll hooking, etc. Either the Mozilla security folks are incompetent / winging it, or this isn't the real reason.
Or is HTTPS / LetsEncrypt too big to fail? HTTPS still always a good choice? I see…
This is just plain bad.
For a piece of open source software you really have very little control with firefox. It really sucks that the alternatives are worse.
This, likely for almost all of their users, creates more of a security problem than signature checking actually solves. For me noscript no longer works which is (IMO) a critically important extension (between mozilla taking away the disable javascript button and spector.)
It's just that it's more work than having what you want implmeneted and maintained by others.
that's kinda the problem. there's plenty of reasons to be "with" firefox still, but you shouldn't need reasons other than it's the best browser. when it starts requiring loyalty to be a user, that's a big problem.
I expect perfection from plane and car manufacturers, and I pay for that. My browser, I can live with an occasional hiccup.
Preferences > Privacy and Security > Strict
I thought that the way signing works in general is that the signer issues a certificate for the thing being signed (domain, code, whatever) that contains identifying information for the thing signed (host name for an SSL certificate, checksum of the code for a code signing certificate), the valid from and valid to dates of that certificate, and assorted other information, and either a reference to or a copy of the signer's certificate, and it signs the whole issued certificate with the signer's certificate.
Someone checking the signed thing is supposed to consider it validly signed if:
1. The date is in the valid range for the signed thing's certificate,
2. A check of the signature of that certificate against the signing certificate passes,
3. The signing certificate is recognized as being from an issuer considered trusted by the checker,
4. Neither the signed thing's certificate nor the signing certificate have been revoked, and
5. The signing took place during the valid date range of the signing certificate.
Note there is no "the date of the check is in the valid date range of the signing certificate". A signing certificate expiring should not invalidate things signed by it. It should just prevent signing anything else with it.
So why is a signing certificate expiring for Firefox breaking already signed extensions? Shouldn't it just be stopping new versions of extensions from being signed?
The model you're suggesting is closer to the "timestamping" one commonly used in code signing (IIRC Windows and Mac both do this) where a third party that's particularly trusted to handle key material well long-term gives you a second signature over the message "I saw this signature at this time" (effectively they are analogous to a notary or witness for real-world signatures). Then you can trust that signatures from expired signing certs were actually made in the past, and not by an attacker who got hold of the key. That is, without timestamping you have no proof of #5 in your list.
(I suppose you could do this now for the SSL PKI with Certificate Transparency logs.... it isn't exactly what they were built for but it's probably sound.)
That said, existing signatures that have already been verified, for existing extensions, should still be trusted.