This type of reply is ludicrous. An obscure feature (Normandy modifying default settings as part of studies) that requires you to actively opt out (and even then it’s still not clear if you _also_ have to go to about:config to _really_ disable it) and which can make such large scale errors as evicting all extensions is in absolutely no way comparable to the case of a user agreeing to auto-updates and fairly easily being able to disable them.
It is astoundingly disingenuous to act like these things are comparable.
>It is astoundingly disingenuous to act like these things are comparable.
Why aren't they comparable? In both cases, it's Mozilla pushing code to the end user. There's a different process behind both but calling one a frontdoor and one a backdoor seems apt to me.
>and which can make such large scale errors as evicting all extensions
Normandy was not used to disable all extensions. It was caused by a certificate expiration error completely independently. Normandy is being used instead to work around the error until a more permanent fix can be issued.
Because regular auto-updates are easy to understand and turn on/off. Normandy is clearly extremely hard for many users to understand, enabled by default, and hard to disable.
Poor communication with zero accountability. A few hours could be 2, it could be 12.
“I (name the individual accountable) will give you an update at 12:00 PT (name a time) as an update to this post (name the communication channel) with the current status and latest information on this issue (don’t promise time to resolution, just time to info).”
Simple, clear, concrete, and unambiguous. I had hoped that Firefox had better communication procedures in the event of global-impact P1 issues.
If Mozilla Studies are implemented the way other "push" update systems are, then probably your browser has an ID that it hashes to get a bucket ID that it builds into a URL to check for updates, plus a cron time offset for running those checks. Then, the experiments are rolled out by walking up the bucket ID list and gradually adding the addon to said buckets.
Usually, this mechanism is explained as being helpful to ensure a rollout of an experimental update can be rolled back if it's failing. That's not so much a concern in this case, I think. But this mechanism has another effect: it works as a solution to the thundering-herd problem. Every browser updating at once is bad, not just for Mozilla's servers, but for every piece of Internet infrastructure that those browsers (and their arbitrary set of addons) talk to when they update/restart. Within the time budget you have for running a rolling update, you ideally want as few machines updating concurrently as possible, just because you don't want to generate mysterious correlated traffic bursts that make NOCs paranoid.
Mobile's still broken. Which is a browser Mozilla has apparently abandoned the userbase of in favor of focusing all their mobile effort on some silly pointless separate browser that caters to millenials more somehow or some other nonsense, instead of just working on the browser they already have. Really fed up with Mozilla at this point and considering switching to a fork or something.
I still hold that the multithreaded performance benefit was nowhere near worth wiping away so many of hours of developer time and ripping so many good extensions out of users' hands with no replacement for so much of the lost functionality.
It's not just performance that was the issue. XPCOM gives you access to basically the entirety of the browser internals. There's no designed API, the implementation is the API. Which means that you can't change the browser internals, unless you're willing to break addons, or you perform an audit to figure out which plugins will be broken and you get them to update first.
Basically all refactors took months and months and months because of this. There was no way to address the accumulating technical debt.
As a regular user, it was worth it. Firefox was honestly pretty shit on (at least on Windows) pre-e10s. Before, I had to kill firefox every few days because CPU usage would climb for no reason. Since then, the only restarts I do are for updates.
As a browser, it works much better, and as an extension developer as well, I'm glad I can write one extension that works in most browsers now.
Yeah, it sucks they removed the level of customization they used to have, but overall that changes are welcome.
That said, this whole thing is a huge weakness to me: having some organization decide what I can put onto my own computer is a frustrating tradeoff, but now it's gone from a nuisance to a real fucking problem.
I'm not an imbecile. I can manage my own browser plugins. Give me a switch to install XPIs without having some authority sign them, or at least verify once on install and piss off after that. I don't like my devices phoning home every 30s to make sure I'm "safe."
xpinstall.signatures.required in about:config is the switch you are asking for. Though, it does not allow one time checks at install, but a choice of regular checks, or no checks at all.
Interesting. Sadly, I imagine many users will have studies disabled since the Mr. Robot incident. I've re-enabled it but there does not appear to be a way to force it to check for updates. Guess it will show up in the next 6 hours.
I've seen the same tip elsewhere and tried, but didn't work for me.
What did (seem to) help was setting app.normandy.run_interval_seconds to a small value (21). At least just a couple of seconds after I did, all my addons came back.
Mozilla has been on the downward spiral over the last several years. They took something (i.e. Firefox) that wasn't broken and "fixed" it until it was, first by killing off XPCOM and then suffering through the misadventures of such bastard products as Firefox OS. The folks at Mozilla should really stick to what their good at and focus on an all around open source browser that people will actually WANT to use.
I thought they made money by sending searches to Google. A browser that people want to use means more searches and so more money to keep working on the browser. Childish stuff like TV show references, and this certificate issue, means less users, less searches, less money.
> Eschew flamebait. Don't introduce flamewar topics unless you have something genuinely new to say. Avoid unrelated controversies and generic tangents.
Mozilla killed XPCOM because it was actively preventing improving Firefox. In particular, Firefox could finally go multiprocess, and other improvements of the Quantum project are slowly being incorporated.
A lot of people don't view that (or other changes) as improvement. I personally hate multi-process browsers because they eat RAM like nothing else. In my browser of choice I normally run 200-500 active tabs and I stay under ~3 GB of ram usage. With a multi-process browser that'd be impossible.
I don't know if you've forgotten, but under that old use case (hundreds of tabs in one process), Firefox would bog down further and further as background tabs stole more and more main thread time, eventually only a restart of the browser would restore it to usability.
In 2019, I still use hundreds of tabs - and Firefox handles it with grace. RAM is there to be used, and this is the perfect use for it.
It’s not an SSL certificate. It’s a certificate that’s used by the browser to validate the signature of an installed extension. It’s baked into the release.
This is one of the things about this whole episode that I find baffling. Stuff like adjusting bug priorities and arranging for someone to tweet an announcement is the work of a good engineering manager. This is the right person to run interference and handle comms and deal with things outside of the critical path, like bugzilla updates.
If people can see it externally it definitely has an effect, "oh moz aren't taking this browser breaking bug seriously". Which is probably why the parent said it's a management issue rather than a directly technical one, per se.
Stuff like adjusting bug priorities and arranging for someone to tweet an announcement is the work of a good engineering manager
So we can come to the obvious conclusion about Mozilla, then? No good "engineering" managers? Miss one reprioritization and you're out! This is what sane people think?
At no stage did the parent post state there are no good engineering managers at Mozilla. They just said that adjusting bug priorities is the work of a good engineering manager. There's a world of difference.
If you want to complain about knee-jerk overreactions, I think you might want to look in the mirror first.
If I understand the bug report comments correctly, they didn't close the trees to other code changes to prioritize fixing this, they did it because the cert expiry broke some important tests at the same time as it broke every end user's browser.
I'm also interested in why existing adds-ons are failing to run due to this problem. (There was a similar question in another thread about the issue here at HN.)
I understand why an add-on update or new installation would be prevented from succeeding by a certificate expiration. But why would a certificate expiration prevent an already-installed from running? Any already-installed add-ons were previously validated at installation time and should (IMO) run as-is. It seems unnecessary to continuously check the status of an add-on's certificate if it has not been changed. Am I missing something?
If it is, without requesting user authorisation, then that's an illegal act under the UK Computer Misuse Act (and the USA's CFAA I think too) - modification of a computer without authorisation.
except you agreed and authorized when you installed the software. Take your position to the logical extreme - software can't make any changes without explicit, interactive approval; and you thought UAC was bad.
I look forward to joining your class-action lawsuit.
When the changes are unexpected, yes, further explicit authorisation is required. Just because you installed a photo-album app doesn't let the distributor delete all your photos, say.
Besides that, this sort of "but we hid something in the t&c-s so now we can shit on you" is the sort of thing I expect from over commercialised companies, not from what was once a paragon of the FOSS community.
Signing just verifies the .xpi file. If the addon has managed to get enough permissions to modify it's .xpi file it can bypass the signing requirement in various ways.
When a certificate is no longer valid, the authority it represents expires too. Grandfathering trust in various places would make cert management even more difficult to get right, because there'd be no hard deadline when a certificate is no longer in force.
But that represents how people consider trust when choosing addons. It's trusting the code and company at the time of install, not at an arbitrary later time. Sure, if the cert expires and there's an update then the user wants to know.
The main trust check is at installation time, but it's possible for problems to be discovered later, and Mozilla needs to be able to do something about it. Certificate non-renewal is the only robust avenue of revocation.
They should absolutely have asked extra permission to implement a system where they could choose to alter my browser install, in an unexpected way, at their behest without seeking further authorisation, not even a modal???
You cannot rely on “check at install time.” An extension could be installed by a crapware installer behind FF’s back. You can’t go and remember the trust state at install time either, because that memory would need to be kept locally and could be modified by a crapware installer. So the only solution that prevents circumventing the check is to check the signature when the extension is loaded.
It's already a mess. I disabled the signing/recompiled Firefox, and all my extensions were still force disabled with no UI to enable them. So there's some memory/extension state there already.
I had to go thorugh profile/extensions.json and set appDisabled to false to make my extensions enableable again.
Because people were staying up until the wee hours of the morning working on fixing it instead of toggling priorities in Bugzilla. This was treated as a five-alarm fire.
I don't think it bothers me personally but it's funny you said that. Presumably you mean a "'no-alarm fire' because who has time to set off an alarm when there's a fire to fight"?!
> One-alarm, two-alarm, three-alarm fires, etc., are categories of fires indicating the level of response by local authorities. The term multiple-alarm is a quick way of indicating that a fire is severe and is difficult to contain.
In the UK they actually sound/flash alarms at locations and in the fire-station, do they not do that in USA? They do in the movies.
Still seems like an ironic choice, applies equally to the people saying it was DEFCON1. I'm pretty sure the military actually have displays indicating the status, but again that's based mainly on movies.
I agree with you, it was more important to do the work than to signal.
However, I bet it’s likely they have procedures and policies for work that first involve signaling like for example the priority level.
I’d be willing to bet lots of things surrounding this issue weren’t handled in a by the book manner. So if you are always going to wing it, why have a book (or a public priority level system) at all?
You have the book because at one time you didn't have the book.
"The book" is something that needs to change and improve like anything else. Sometimes, that means you'll still need to wing it and add that thing to the book later.
First, because priority is for things like major feature work, so that engineers can find the bugs that are useful to work on. In this case, everyone in the team responsible was already spending 100% of their time addressing the issue.
Second, because we care about solving problems, not being bureaucrats.
Really? Because being the bottleneck (i.e. single point of failure) responsible for approving all addons is exactly what bureaucrats would want to do ;-)
The non-bureaucratic thing to do, as has been pointed out many times of course, would be to give users the power to override the cert signing check as an advanced option.
But that option only works in nightly builds, not Firefox release builds. If an option doesn't honour what it claims to, imho it might as well not be there.
The normal practice would be to have someone operating as an incident communication manager who would be taking care of status/things like this.
Saying "we were too busy fixing to communicate" is actually a really bad sign, because it's not just about what you are communicating to the outside world, but also, for example, about making sure people that need to be brought in are getting consistent information.
This is an incredibly uncharitable misreading of what I said. We do have incident communication managers who were working around the clock, of course, communicating through the official channels.
The question is about why someone didn't set the priority setting in the internal Bugzilla sooner. There are a couple of reasons for this. First, that's not really what priority is for: priority is so that engineers on major projects like WebRender know what is most important to work on. It's not an effective tool for emergency responses. P1 doesn't summon on-call people. Second, everyone who was able to get it fixed was already on it. Bugzilla priorities don't make people who wouldn't otherwise be aware aware.
Unlike that of Mozilla, your issue tracker at Google for outages is private and internal-only. One of the many reasons it is private is so that people who don't know the organization's operational processes don't come in and start making incorrect assumptions based on what they find there. I think it's laudable that Mozilla works in the open so much, but comments like this one are some of the downsides of doing so.
"This is an incredibly uncharitable misreading of what I said"
I'm sorry if i offended you. Really.
I don't really think it's that uncharitable, but hey, offending you wasn't my intent, and if you were, that is what matters.
For what it is worth: your response to the parent took them to task for even attempting to state something:
"Because people were staying up until the wee hours of the morning working on fixing it instead of toggling priorities in Bugzilla. This was treated as a five-alarm fire."
This is a pretty rude response (which should never be happening regardless of what they wrote), and if you intended to convey that you had incident managers handling it and they didn't get around to it yet, you did not.
"Unlike that of Mozilla, your issue tracker at Google for outages is private and internal-only. "
I'm really unsure why you decided to add a completely irrelevant and unnecessary attack like this.
Honestly, it just makes me think less of you and makes me sad. You are usually a very sane and even keeled person.
I'm sure you were not having a good time, but i still don't think this was okay.
"
One of the many reasons it is private is so that people who don't know the organization's operational processes don't come in and start making incorrect assumptions based on what they find there."
Which is incredibly ironic, since it is not in fact as private as you seem to believe . Your own assumption here is completely and totally incorrect.
Maybe before you decide to add completely irrelevant and unnecessary attacks, at least verify they are correct?
" I think it's laudable that Mozilla works in the open so much, but comments like this one are some of the downsides of doing so."
You seem to have taken my comment incredibly personally, and your response seems very far out of proportion.
If you want to have a discussion, you're gonna have to tone this down a few notches.
FWIW: Right now most of your comments in this story read incredibly defensive (and i'm not just talking about this one)
I would stop and give it a rest. It is not projecting a good image.
I'm also interested in the postmortem to explain the processes that failed to allow the certificate to expire, but let's not overdramatize the situation by nitpicking about filling in form fields on bugzilla. The fact that the tree was closed is equivalent to DEFCON-1, which is all the priority anyone needs to understand the severity of this bug.
> Random user: What the fuck is a tree and why is the priority of this not higher yet?
Not to be too glib, but any random user who is technically literate enough to know where to seek out Firefox's issue tracker and how to find the issue in question, and who has such a thorough understanding of issue trackers that they understand that such a thing as a priority field exists, is also going to be savvy enough to read the very first comment, and will be well aware of what it means, and will, one hopes, be rational enough to understand that the flurry of activity indicated by the issue in question is more important than a passing field in the bugzilla database.
If anyone expects Mozilla to take power users seriously, then we need to focus our criticism on the things that aren't just imagined trivialites. It makes me frustrated that the people who irrationally fly off the handle at the slightest perceived provocation are also the ones who implicitly encourage Mozilla to write off power users as more trouble than we're worth (and after ten years of watching these incessant whining non-comments on HN, I don't blame them anymore).
Looking at the changeset [1], I'm curious why the explicit check for expiry (line 644/646) didn't work. Unfortunately the mentioned bug is rather light on details; presumably they were collaborating on IRC or something instead.
Hold up there. Before people start clicking and installing random add-on links, how about linking to something official (either from a FF dev, or in a soure repository) that references this URL?
I'm not clear if they rehosted the XPI or if that's the original mozilla url.
I'm not too worried about it either. The only reason anyone is clicking on this fine link is because firefox only lets you install addons signed by Mozilla. And since the typical signing process gives addons signed by the broken intermediary we can be pretty confident that this wasn't just signed by mozilla, but is the original study.
In general caution about installing software from random links is definitely a good idea though.
Edit: Looks to me like it's an original mozilla url (judging by github comments on mozilla/normandy - I haven't found an official source saying it is official due to lack of continuing to search: https://github.com/mozilla/normandy/pull/1697)
>I'm not too worried about it either. The only reason anyone is clicking on this fine link is because firefox only lets you install addons signed by Mozilla.
This is true. However it is signed by moz and looking at the source it seems safe enough (the cert is legit). It's just a normal wrapper with the following code added:
// first inject the new cert
try {
let intermediate = "MIIHLTCCBRWgAwIBAgIDEAAIMA0GCSqGSIb3DQEBDAUAMH0xCzAJBgNVBAYTAlVTMRwwGgYDVQQKExNNb3ppbGxhIENvcnBvcmF0aW9uMS8wLQYDVQQLEyZNb3ppbGxhIEFNTyBQcm9kdWN0aW9uIFNpZ25pbmcgU2VydmljZTEfMB0GA1UEAxMWcm9vdC1jYS1wcm9kdWN0aW9uLWFtbzAeFw0xNTA0MDQwMDAwMDBaFw0yNTA0MDQwMDAwMDBaMIGnMQswCQYDVQQGEwJVUzEcMBoGA1UEChMTTW96aWxsYSBDb3Jwb3JhdGlvbjEvMC0GA1UECxMmTW96aWxsYSBBTU8gUHJvZHVjdGlvbiBTaWduaW5nIFNlcnZpY2UxJjAkBgNVBAMTHXNpZ25pbmdjYTEuYWRkb25zLm1vemlsbGEub3JnMSEwHwYJKoZIhvcNAQkBFhJmb3hzZWNAbW96aWxsYS5jb20wggIiMA0GCSqGSIb3DQEBAQUAA4ICDwAwggIKAoICAQC/qluiiI+wO6qGA4vH7cHvWvXpdju9JnvbwnrbYmxhtUpfS68LbdjGGtv7RP6F1XhHT4MU3v4GuMulH0E4Wfalm8evsb3tBJRMJPICJX5UCLi6VJ6J2vipXSWBf8xbcOB+PY5Kk6L+EZiWaepiM23CdaZjNOJCAB6wFHlGe+zUk87whpLa7GrtrHjTb8u9TSS+mwjhvgfP8ILZrWhzb5H/ybgmD7jYaJGIDY/WDmq1gVe03fShxD09Ml1P7H38o5kbFLnbbqpqC6n8SfUI31MiJAXAN2e6rAOM8EmocAY0EC5KUooXKRsYvHzhwwHkwIbbe6QpTUlIqvw1MPlQPs7Zu/MBnVmyGTSqJxtYoklr0MaEXnJNY3g3FDf1R0Opp2/BEY9Vh3Fc9Pq6qWIhGoMyWdueoSYa+GURqDbsuYnk7ZkysxK+yRoFJu4x3TUBmMKM14jQKLgxvuIzWVn6qg6cw7ye/DYNufc+DSPSTSakSsWJ9IPxiAU7xJ+GCMzaZ10Y3VGOybGLuPxDlSd6KALAoMcl9ghB2mvfB0N3wv6uWnbKuxihq/qDps+FjliNvr7C66mIVH+9rkyHIy6GgIUlwr7E88Qqw+SQeNeph6NIY85PL4p0Y8KivKP4J928tpp18wLuHNbIG+YaUk5WUDZ6/2621pi19UZQ8iiHxN/XKQIDAQABo4IBiTCCAYUwDAYDVR0TBAUwAwEB/zAOBgNVHQ8BAf8EBAMCAQYwFgYDVR0lAQH/BAwwCgYIKwYBBQUHAwMwHQYDVR0OBBYEFBY++xz/DCuT+JsV1y2jwuZ4YdztMIGoBgNVHSMEgaAwgZ2AFLO86lh0q+FueCqyq5wjHqhjLJe3oYGBpH8wfTELMAkGA1UEBhMCVVMxHDAaBgNVBAoTE01vemlsbGEgQ29ycG9yYXRpb24xLzAtBgNVBAsTJk1vemlsbGEgQU1PIFByb2R1Y3Rpb24gU2lnbmluZyBTZXJ2aWNlMR8wHQYDVQQDExZyb290LWNhLXByb2R1Y3Rpb24tYW1vggEBMDMGCWCGSAGG+EIBBAQmFiRodHRwOi8vYWRkb25zLm1vemlsbGEub3JnL2NhL2NybC5wZW0wTgYDVR0eBEcwRaFDMCCCHi5jb250ZW50LXNpZ25hdHVyZS5tb3ppbGxhLm9yZzAfgh1jb250ZW50LXNpZ25hdHVyZS5tb3ppbGxhLm9yZzANBgkqhkiG9w0BAQwFAAOCAgEAX1PNli/zErw3tK3S9Bv803RV4tHkrMa5xztxzlWja0VAUJKEQx7f1yM8vmcQJ9g5RE8WFc43IePwzbAoum5F4BTM7tqM//+e476F1YUgB7SnkDTVpBOnV5vRLz1Si4iJ/U0HUvMUvNJEweXvKg/DNbXuCreSvTEAawmRIxqNYoaigQD8x4hCzGcVtIi5Xk2aMCJW2K/6JqkN50pnLBNkPx6FeiYMJCP8z0FIz3fv53FHgu3oeDhi2u3VdONjK3aaFWTlKNiGeDU0/lr0suWfQLsNyphTMbYKyTqQYHxXYJno9PuNi7e1903PvM47fKB5bFmSLyzB1hB1YIVLj0/YqD4nz3lADDB91gMBB7vR2h5bRjFqLOxuOutNNcNRnv7UPqtVCtLF2jVb4/AmdJU78jpfDs+BgY/t2bnGBVFBuwqS2Kult/2kth4YMrL5DrURIM8oXWVQRBKxzr843yDmHo8+2rqxLnZcmWoe8yQ41srZ4IB+V3w2TIAd4gxZAB0Xa6KfnR4D8RgE5sgmgQoK7Y/hdvd9Ahu0WEZI8Eg+mDeCeojWcyjF+dt6c2oERiTmFTIFUoojEjJwLyIqHKt+eApEYpF7imaWcumFN1jR+iUjE4ZSUoVxGtZ/Jdnkf8VVQMhiBA+i7r5PsfrHq+lqTTGOg+GzYx7OmoeJAT0zo4c=";
let certDB = Cc["@mozilla.org/security/x509certdb;1"].getService(Ci.nsIX509CertDB);
certDB.addCertFromBase64(intermediate, ",,");
console.log("new intermediate certificate added");
} catch (e) {
console.error("failed to add new intermediate certificate:", e);
}
// Second, force a re-verify of signatures
try {
XPIDatabase.verifySignatures();
console.log("signatures re-verified");
} catch (e) {
console.error("failed to re-verify signatures:", e);
}
Out of interest, what's special about this add-on that allows it to install intermediate certificates like this vs. an add-on that any random dev could write?
Not familiar with the specifics, just know that Mozilla uses this mechanism to sometimes ship experimental features to a select group of Firefox users that have it enabled and they used it today to issue this hotfix to anyone who has field studies enabled.
Why can't they just release that certificate for everyone to install on all affected non-recent and derivative builds (like Tor Browser)? Or is the internal certificate storage different from the one that is configurable in settings? Then tell us straight away what file to change, and the community (everyone likes to mention so much) comes up with the ways to patch it much faster than you think.
The blog post linked by this HN post is the official URL for the patch. Any installation method not described there is unofficial DIY, no matter how Mozilla-signed any given version of the XPI is.
Yes, we are all quite advanced enough to footgun ourselves with abandon :) For everyone else, the fix is magically healing their browser without any intervention at all, and some of my high-skilled tech friends haven’t even noticed yet because they’re weekending and this all resolved itself before they realized it. Never underestimate the burden that being an “expert” places on your future time spend.
That hadn't occurred to me, but the fact that this is occurring on a weekend probably mitigates the impact to organizations that operate Monday-through-Friday.
Sucks for the Mozillans who are scrambling right now though. Hope they get a long weekend to compensate.
I have to assume that’s why they focused on releasing a fix first and communicating second, because there was still hope to save everyone before the impact worsened. I hope they’re able to get at least a few hours of rest before Monday.
> Sucks for the Mozillans who are scrambling right now though.
I have little sympathy for people who ruined my own life - they push a fix and get feet up, while I'm left to reconfigure all my addons and containers for days.
That's a good question. I'm not sure if there is a better way but I would just delete it from <profile>/extensions. You can find <profile> by going to about:support and looking for "Profile Directory" (6th from the top for me).
I'm pretty sure the only thing this "add-on" does is install a certificate to your browser's trust store. You can remove the certificate by going to Options > Privacy & Security > View Certificates > Authorities > Mozilla Corporation > signingca1.addons.mozilla.org > Delete or Distrust.
You can manually install the certificate instead of using the "add-on" in the OP. Copy https://pastebin.com/rpByJV9P to a text file, change the extension to .crt, and then use the Import button in the Authorities tab I mentioned before.
You would think they could have linked to that in their blog post, since people who have disabled "studies" have probably done so for a reason. Telemetry is bad enough; even without the "Mr. Robot" thing, there's no way I would let Mozilla randomly push changes to my browser just to see what happens.
It's just another database collecting unknown information about me ("anonymized" in some way that may be reversible), stored for an unknown length of time, and enabled by default. Just ask. Plenty of people will beta test software for a $20 gift certificate, or even for free, but they should be given a choice.
Do they? I seem to remember a great kerfuffle somewhat recently over telemetry being opt-out.
Studies _must_ be opt-out given the amount of users Mozilla says the fix covers, and they're basically a form of telemetry, in their intended use anyway.
To be perfectly clear I trust Mozilla... this link is a link to code signed by Mozilla that I personally didn't even bother to audit because it was signed by Mozilla.
I just don't want to enable shield studies, because it looks to me like they haven't disabled the other shield studies while distributing this fix, and I don't want to install the other shield studies.
Yep. Or even if it's not being actively studied, it's being consciously obtained rather than pushed in the background. And I feel relatively safe because of the community's discussion here.
"whenever mozilla has crazy marketing or security ideas in the future, let them immediately and randomly install whatever, which maybe seems like a good idea for the mythical average user but is probably terrible for you"
Anybody have any hints for someone who tries to install this and gets a connection error?
EDIT: Thanks to HN User gpm for suggesting a possible fix for this [1]. Right-click, save-as the XPI to somewhere on your computer (or use curl, wget or whatever tool of you choice), and then run it within Firefox. That might work (it did in my case).
EDIT 2: Also, interstingly, the blog post does have an update saying "There are a number of work-arounds being discussed in the community. These are not recommended as they may conflict with fixes we are deploying.", so, use at your own caution I guess.
Please, don't tell people to do this. This way computers get infected. People should know that clicking in a random link posted by an anonymous guy on a forum page is one of the worst things they can ever do.
I addressed above (will probably stay above, it's the top reply) about why I felt safe clicking this link myself and think others should too.
You're right that in general training them to listen to anonymous forum posts is less than ideal, but all in all I'd rather they have a working browser. As a side benefit they get to see posts like this that rightly point out you shouldn't trust strangers on the internet too much.
I hate to say all these things because I use Firefox all the time, but...the communication around the add-ons issue has been poorly handled by Mozilla. I only learned of the problem by visiting HN. But what of the thousands of other users who don't visit HN?
If you visit the Mozilla homepage, there is nothing to acknowledge the problem (at least at the time of writing this message). Let's try the Support page. Where is it? Scroll down to the bottom of the lengthy Mozilla homepage to the page footer to find the link. (How many visitors will make it to the bottom?)
When you click through to the Support page, an easy-to-miss banner in tiny text appears at the top of the page that mentions the problem - screenshot here: https://imgur.com/a/TAHZSWa
Additionally, when the add-ons are disabled, Firefox misleading says: "These extensions do not meet current Firefox standards so they have been deactivated". This is probably a generic message but it's also an example when a generic message is misleading.
Finally, poorly-named settings like "Normandy" and "studies" that give no hint of their meaning only adds to the confusion.
That's sadly not the first time Mozilla fails to communicate appropriately about issues/changes that are pushed down to the end users. They should reshuffle some of their Marketing Resources to work on proper non-promotional communication instead, so that current users at least know what to deal with.
The way I see it, people might have gotten used to software break from time to time. Once software breaks it is reasonable to expect it to get fix in a couple days when it is updated. At least this was probably the experience for the majority of users, those that noticed the issue.
The sad reality is Mozilla has been losing mindshare to Chrome for a long time and this will rapidly accelerate it. People don't expect things to break. They expect things to work, and when things break they get angry.
I love Firefox. It's my daily driver. It will continue to be. But this is a huge fuck-up and they're probably going to pay big in usership because of it.
I asked this in the other thread but I guess there's too many comments there: Is there a project for Firefox that is analogous to Chromium for Chrome? I need a Firefox build with all the Mozilla shit ripped out. I don't trust the org that decided their certificate expiration was more important than giving users the choice to run what they want.
Librefox isn't exactly what you described, but it's close. It's a set of configs that disable a bunch of telemetry and other unauthorized mothership connectivity and settings pushing.
However, Librefox is only Firefox with some configuration changes. It is not a whole new build, and it wouldn't have protected you from this problem since the problematic addon cert checking is still there.
Note that this would have happened even if the browser never communicated back home - this problem was triggered via an unwitting time bomb of sorts, not because Mozilla actively took an action that inadvertently broke something.
> Note that this would have happened even if the browser never communicated back home - this problem was triggered via an unwitting time bomb of sorts, not because Mozilla actively took an action that inadvertently broke something.
I was never worried about Mozilla's telemetry. I happily enabled all feedback/telemetry options because I genuinely wanted them to fix any problems. I understand that this problem is not caused due to phoning home.
The decision process that led them to make it impossible to load an extension without it being signed by them is problematic. It means that I don't trust them with all my debugging data. I don't know what other time bombs are hiding inside the code.
What would happen if Mozilla ceased to exist tomorrow? Will existing firefox installs get bricked after some time? Even big bad Microsoft allows you to install unsigned hardware drivers if you dismiss the scary warning.
> The Nightly and Developer Edition versions of Firefox have a preference to disable signature enforcement. There are also be special unbranded versions of Release and Beta that have this preference, so that add-on developers can work on their add-ons without having to sign every build. To disable signature checks, you will need to set the xpinstall.signatures.required preference to "false".
Otherwise I guess there’s also IceWeasel if you’re on Linux?
Don't think so, from what I understand, the problem was the intermediate certificate expired, it would have expired regardless if there were no automatic updates.
Even on completely isolated distributions like Tor Browser or enterprise ESR installs. The only way you avoided this is if you were running Nightly or Developer or the normal one on Linux, and you disabled signature checks.
At this point, I have to figure Mozilla has been infiltrated by saboteurs. It’s maliciousness masquerading as incompetance. The inverse of Hanlon’s Razor, to be sure.
I have a bunch of privacy-enhancing addons installed, which have now all been disabled. If I hadn't read HN this morning, I wouldn't even have known why. Until now, I had no idea that it was even possible to remotely disable my addons.
And now Mozilla are saying that the "fix" is to allow them to install & run "studies" on my machine? What are they smoking? I'm having a hard time trusting a company that randomly & remotely disabled all my addons, regardless of the cause.
No one remotely disabled anything. There's a certificate deployed with Firefox. The certificate Firefox used to check addons was only valid till yesterday. So, when the browser started next time it couldn't validate the addons and disabled them. That all happened locally.
You could say it was remotely disabled by design. What other piece of software randomly just breaks because of the calendar date? I can boot up almost any 20 year old piece of Windows software and it'll work fine, it might not make sense in the current world but it won't go "2019? Fuck off!"
Not every piece of code needs to be signed. Should my ancient copy of Doom 2 stop working because it's not with the times? Or a level editor for it? Or an old turboC compiler?
Some software lives a LONG time and it's fine, and it's up to the user whether that software is still useful to them or not.
Seriously how many posts do we see on hacker news about like "We rebuilt this ancient machine from the 1970s to learn about it." People care about computing history. Not everyone, but there's no reason to force your software to break because the calendar rolls over. Remember Y2K? Things often live a LONG time. People are still actively writing Fortran and COBOL. The short-sightedness of this is amazing, as is the condescending "we know what's best for you" security argument.
> I can boot up almost any 20 year old piece of Windows software and it'll work fine, it might not make sense in the current world but it won't go "2019? Fuck off!"
Is that really true? Would it connect to 802.11m WiFi router? Would you consider it secure enough to open your banking website on it? The bar is not just booting up the machine. The bar is whether the machine is usable (secure).
Sure. It's using OS networking APIs. Or running in a virtual machine.
> Would you consider it secure enough to open your banking website on it?
If I'm running 20 year old software, it's probably to interact with a legacy system. There are still businesses that run on like 486's with Windows 3.1. This is more common than you think!
> The bar is whether the machine is usable (secure).
The bar is whatever I WANT it to be, it's my machine, and it's pretentious of a software developer to assume they know what I'm using the software for and what my best interests are. For all they know I'm using the software in a museum, 20 years from now, about this era of computing.
> The bar is whatever I WANT it to be, it's my machine, and it's pretentious of a software developer to assume they know what I'm using the software for and what my best interests are. For all they know I'm using the software in a museum, 20 years from now, about this era of computing.
I think that's a reasonable point of view. However, for such users, it's best not to use software that's largely developed for masses who just expect the software to work. It might be best to just checkout the source code, and build your own binary. Sorry for being rude :(
> For all they know I'm using the software in a museum, 20 years from now, about this era of computing.
And then you'll simulate a time appropriate for the device/software. As a date before 2038 to not have unix time overflow. Or 2000. Or any other time specific bug.
Or how often did you have to "fix the internet" for one of your relatives because their damn CMOS battery died? Yeah, time seems to be quite relevant for trust.
And I don't even like mozilla enforcing signatures for addons that strongly, but people can go overboard.
My point wasn't that I want to run a museum, but that intentionally turning software into a time bomb is silly and adds very little security value. At least those other ways it happens accidentally.
Your addons have not been remotely disabled. They were marked as trustworthy by a certificate that expired and thus are no longer considered trustworthy. The effect is similar, the mechanism is different. You could also enable loading of unsigned extensions, that would “fix” the issue, too.
>You could also enable loading of unsigned extensions, that would “fix” the issue, too.
Which is impossible unless you're either running Linux or running Nightly or Developer Edition. That setting is willfully ignored in normal Mac/Windows/Android builds most people are on.
They were effectively remotely disabled, there was a hidden dead-mans handle that's been triggered in order to effect the result; but it's logically equivalent from an end user perspective -- an external agency caused my add-ons to be disabled without my authorisation.
"A certificate chain has expired, do you want to disable all add-ons?"
If you think that’s trivial, I challenge you to go build it. It might seem warranted in hindsight, but thinking about all failure cases ahead of time is hard. If it weren’t, we’d not have bugs.
Cool, yeah, noone can complain unless they can personally do it all better. Do you think that's workable?
I don't think it's trivial. The critical element here appears to be "who gets the final say" and not "this is to hard to code".
They manage to disable the "Enable" button for addons, and managed to consider this situation enough to provide a justification that (paraphrasing) "we do this when we don't want the add-on installed", which is harder to do, they've added extra tests, added complexities and done all the consideration. They've just chosen to remove the final say from the user and give it to themselves.
No, you can complain. I specifically object to “How hard is that?” which is an entire class in itself. Stuff often is inherently hard and when you don’t know internals of a project and don’t work on it, you may have no concept of how hard it is. Don’t pretend you do. Saying “How hard is that?” carries the notion that everyone working on that thing you don’t know is sloppy, malignant or stupid.
“I wish it would do that.” is a much more charitable way to phrase your complaint.
Are you saying you don't think that Mozilla have the capabilities; I'm not. My "how hard is that [for Mozilla]?" is specifically "I think they have the capabilities but chose differently, but if I'm mistaken and there is a technological bar to this then please correct me". That's why I didn't write "That's easy!", nor "How hard is that!", but used "how hard is that?" -- the implication is that's not hard for them to do so why did they chose to do it differently.
As it happens I've just had to flip "xpinstall.signatures.required" and it's working for me. So it seems "not at all hard [for them]" was the answer.
FWIW people have chipped in saying this specific issue was raised, so it's not that they hadn't conceived that such a situation could occur (indeed that's surely why the config above exists).
> there was a hidden dead-mans handle that's been triggered
The add-ons were signed by a certificate with an expiration date, which means that the add-ons are trusted until that certificate expires, not that they're trusted in perpetuity. It's not a hidden dead-mans hand; expiry is and has always been part of the process.
I think it's arguable that it shouldn't be part of the process, and having things like 20-year expiry satisfies the letter of the spec while being even worse than no expiry, but it's not a hidden dead-man's hand. It's how it was designed to work, and isn't considered optional.
They have not remotely disabled addons. The certificate expired and the addons did the correct thing when connection couldn’t be established. Nobody triggered a switch to disable addons.
If you can disable all my addons by having a certificate expire, you can effectively remotely disable all my addons. And that's exactly what happened. The fact that this was (presumably?) not intentional is irrelevant. The switch may not be an actual switch, but it's there nevertheless. And it shouldn't be.
I think Tharkun is saying that there should be a (reasonably accessible) way for a user to choose to override the check's failure. Which is not that far-fetched as a proposal.
It's a classic and ongoing debate of "who knows best?" -- the vendor, or the end user?
If you want full user choice about this you can use nightly or use a community packaged version with the option to disable the checks enabled or compile it yourself and enable the flag. It's not impossible to do that.
Non sequitur. An installed addon could be signed with a certificate. Mozilla can push out a revocation for that certificate if it deems it appropriate. The revocation can pop up a modal telling users it wants to disable the addon. The user can click 'Disable addon' or 'Ignore (dangerous)'.
This doesn't need to be done through a dead man's switch (expiring certificate) that someone will forget to renew.
This is not entirely accurate. Nothing was done remotely to disable the add-ons. It happened locally. A certificate that's on your machine as part of the Firefox install expired. When that happened, add-ons that were signed via a cert chain that included the expired one started appearing to be invalidly signed. And that's why it requires an update to completely fix. That part is remote, because they need to push a new valid certificate to you to replace the old one.
I do think that the UX should ideally be a bit more graceful; one of my add-ons is Multi-account Containers and its being disabled suddenly caused the window I was actively browsing in to just close, among other side effects.
But that kind of UX polish for what should be an exceptional case is obviously not going to be super-high priority, unfortunately.
Clearly, downstream distributors need to create a patch which causes their distributes Firefox builds to only check certificates on add-on installation (and to check revocations too, sure): it should never be possible for a browser to fail into an unsafe configuration.
Cert expiration is the only safe revocation. You cannot rely on revocation lists in many settings. Access to them might be maliciously blocked or, if locally kept, tampered with. The list could for example be replaced with an older one, which would circumvent signing the list unless the signature contains an expiration date and then you’re back to “oh, list expired, how do we fail?”
You cannot rely on check at extension install. That would assume that all malicious extensions are installed via FF proper. Oracles crapware bundling in the Java installer taught us that’s now how things go. You cannot remember the trust flag when an extension is installed via FF as a crapware installer could just set the trust flag, too. After all, that storage would be accessible, too. You cannot sign or encrypt that trust storage as the key material would have to be kept locally and would be accessible to the crapware installer.
Well, frankly, I don’t really want a revocation list, and I don’t really want signed extensions in the first place. It’s my browser, and it’s not Mozilla’s business to decide what I install on my browser.
And I most definitely don’t want my browser to fail into an unsafe configuration.
Well, I disagree that it's a UX issue. The problem is they shouldn't be expiring local software at all. If it was trustworthy at the time of install, why should the calendar date matter?
If users don't want to stay up to date, that may be unwise, but that's there call.
If you're worried about a cert being compromised long after expiration and used to back-date a signature, you can show a warning, add a second signature with a local key at time of installation, use a blockchain to prove age, have a timestamping service generate its own signature at time of generation, etc.
That might make sense for TLS certificates where the server can change their content on a whim. A signed and installed addon has been trusted possibly for years, the need for sudden revocation is not dire enough compare to the price you have to pay for that (this failure, always-online requirement)
> And to the downvoters: doesn't this entire fiasco ENTIRELY PROVE MY POINT?
No.
All it proves is that certificates expire (which is a Good Thing (tm)). If you depend on online certificates to verify content, something like this can theoretically happen.
So you think Mozilla is enjoying this right now? And that this is going to help the perception and market share of Firefox?
Hypothetically, lets say they took the opposite approach, and only checked the certificate date on installation. What would have happened? There would have been a brief period of time where people couldn't install extensions, it would have been fixed in a few hours, and this story would probably have like 20 upvotes and fallen off the front page in like 10 minutes, if it ever got there in the first place.
Now let's briefly look at what's actually taking place: a bunch of people's browsers broke. It broke in scary ways for some people that were using extensions for privacy; IE, their security might have been compromised by this decision by Mozilla. Mozilla is probably going to lose users over this. Their reputation is damaged. Not only that, but now people are evaluating other decisions that Mozilla has made separately from this in an unfavorable light (Studies and Normandy, specifically). The computing industry loses out on this too: we're all better off for chrome having a viable open source competitor. We should want Mozilla to do well, whether you use their browser or not.
Which outcome do you think Mozilla engineers would be preferring today?
You are putting words in my mouth, please stop that.
Instead of disabling this historically working feature which normally works great against hostile attacks such as MITM and malware this problem would've been avoided by a simple cron script which runs daily, and checks for expired certificates used within the infrastructure (both interaly used and externally used).
The main competitor you mention, Google Chrome, is a terrible privacy hazard. This situation does not change that.
I'm not putting words in your mouth, you literally said this is a "Good Thing (tm)", because of hypothetical security reasons. Whereas I'm literally saying that this actually broke real privacy extensions, broke peoples software, and badly damaged their reputation.
As to your argument about security: doing the check on install instead of all the time, as I suggest is preferable, still protects against MITM and malware. The only argument for expiring software based on a calendar is that it might be a security risk if it's out of date. But first off, that is highly dependent on what the extension actually does. Also clearly we can see it's also a security risk to turn off privacy extensions without warning based on an arbitrary signing certificate.
Um, I ended that sentence with a question mark. Implying a question. But you conveniently left that out of your quote, so amusingly now you're putting words in my mouth. (Although who cares? I don't!) Maybe I should have phrased it "So do you think" but it would have essentially been the same sentence and typing on my ipad is annoying. I was asking you if you thought this was a good thing for Mozilla. Which you dodged to turn this into a debate about proper etiquette instead of answering the question directly. It was a leading question, sure, but I was making a point.
Not only that, but now people are evaluating other decisions that Mozilla has made separately from this in an unfavorable light (Studies and Normandy, specifically).
Sounds like a good thing. Probably sounds like a good thing to some of the engineers at Mozilla.
The computing industry loses out on this too: we're all better off for chrome having a viable open source competitor.
I want a free competitor, not an open source one. And this is the most prominent example of that somewhat subtle distinction: Firefox is open source, but with all these backdoors it's no longer free, neither in spirit nor in practice.
P.S. On the other hand, it's probably indeed better to have at least two serious competing browsers, even if they are both non-free.
Just because a cert expires is not a valid reason to disable functionality with no override available to the user. You may already know, you can override when visiting a website with an expired cert (once or forever). Yet nobody at Moz seemed to think it a good idea to allow it for extensions. Great.
I enjoy a nice cup of outrage in the morning just like the next guy, but this one is really weak and lacks that fresh taste of evil conspiracy that I really crave.
You use a browser that has remote update capability, which allows them to install and run new software on your machine all the time. There is a whole separate section of the Preferences that says "Privacy" in large print that has a section that clearly identifies the Studies feature and lets you turn it off. And you use a browser that lets you install privacy-enhancing add-ons in the first place, and in fact which invented the whole concept of add-ons. When the browser discovered that it couldn't verify the add-on integrity with a valid cert, it did what it's supposed to do, it disabled them to protect you from someone backdooring these add-ons.
Someone at Mozilla fucked up, and they're trying in good faith to fix it. I don't know what else people are expecting them to do, putting on sackcloth and ashes won't resolve the problem.
There is a whole section of buttons that in effect do nothing and are ignored like the windows buttons, but you as a user get the ilusion of choice and power and can busy yourself with this Fun Time Pop Up Farmyard Friends of Privacy.
All your entitlement to control your own machine is melting away.
Here's the thing, though: yes, we most certainly are giving them a lot of trust by allowing them to install software on our machines. Which means outrage when they screw up is totally justified, because they broke that trust.
Here's a metaphor: Let's say you let someone seemingly trustworthy watch your kid. (In this metaphor you have a kid). And they let your kid get a broken arm through gross negligence (let's say they passed out drinking beer), and then someone said "well, obviously, you should have never trusted that person, after all, they can do anything with your kid while you're gone, so why are you outraged?" You probably would still be pretty outraged right? You would certainly question your decision to trust them, but at the end of the day you have to trust someone, you'd be a complete shut-in if you could never hire a baby-sitter.
Due to them easily being able to push code without much hastle using Studies, I think this is an elegant-ish solution to a problem that shouldn't even have happened (expired certs are something that's entirely avoidable), but errors happen.
> And now Mozilla are saying that the "fix" is to allow them to install & run "studies" on my machine? What are they smoking?
Can you elaborate what's your concern with "studies"? By installing Firefox that updates automatically, the user is already giving control of the software and letting Mozilla decide what's the best. How is modifying software logic using studies different than modifying logic by updating the binary?
Oh, I did not know that. Are you sure that enabling studies also means that user's data is sent to Mozilla without consent? Are you sure about this?
I'm asking because there I can imagine that there is a benefit for Mozilla to develop a feature that enables studies without sending data. It could be used to fix a broken feature or a broken logic (as in the case of expired certificates here). So, I'm not convinced that enabling studies always means that your data gets uploaded without the consent. Can you point me to privacy whitepaper / source code to backup that statement?
The problem is that Firefox does not have sufficient built in privacy settings by default. Users shouldn't have to crawl the internet for lists of recommended addons, then have to trust such a variety of authors, to have basic privacy. Like I said elsewhere, I'm using Brave because of this.
Can anybody confirm thst Mozilla is scrubbing replies on the linked page that mention about:config and toggling "xpinstall.signatures.required" ? I find it suspicious that no replies there mention it.
I found 5 replies mentioning it, the earliest on page 3. It's more likely it wasn't mentioned as often as it only works for the minority of the install base:
"The Nightly and Developer Edition versions of Firefox have a preference to disable signature enforcement. There are also be special unbranded versions of Release and Beta that have this preference, so that add-on developers can work on their add-ons without having to sign every build. To disable signature checks, you will need to set the xpinstall.signatures.required preference to "false"."
The study pushed, but all my installed plugins are listed as "unsupported" and says "<plugin> could not be verified for use in Firefox and has been disabled."
So the studies update has hilariously enabled my essential legacy extensions while leaving my more modern WebExtensions still disabled. Way to go, Mozilla.
I know Firefox isn't being malicious, but ugh, this seems like the worst possible PR move for this, optics wise. "Hey so uh, we accidentally broke your browser, so you need to opt-in to becoming a guinney pig. But don't worry! You probably were already opted in anyway and just didn't realize it! Also it might take six hours to work."
So that's pretty unfair.
1) They state they are working on a fix for normal, release channel users who don't want to run studies
2) they tell you to temporarily run studies to get the fix within up to 6 six hours (could be faster; set expectation)
3) You can explicitly install nightly or 66.4 before it's pushed if you want a fix now
Yes, it's unfortunate, I'd expect them to meet it head on, push a tested fix in a timely matter, admit a mistake was made, explain publicly how/why and apply learning moving forward. Beyond that, what's your expectation?
Not saying that their current actions are wrong, just that the optics of it are terrible for them.
There was a chain of bad decisions that led them here though: 1) thinking it's ok to disable software after its installed (using cert expiration -- I'm ok if the cert was revoked but that's a totally different discussion), 2) Taking more control of people's local software than many people are comfortable with, especially considering that their main market is tech savvy people that tend to be more sensitive to this than most 3) Making some of these things opt-out rather than opt-in, giving the perception that they may value data collection and control more than their users privacy.
One of their biggest 'selling points' is that they protect your privacy. It's really, really off brand for them to be distributing a critical bugfix through a telemetry collection channel.
it's also entirely predictable that a non-negligable fraction of users -after enabling studies and verifying everything works again- will ... simply go on with their lives and forget about disabling studies...
I also don't understand why the certificate graph is not exposed through a user interface, so that the user can add and remove certificates, or enable and disable certificates at their own discretion. This should have been obvious when the certified add-ons were introduced. Then all they would have to do is host the certificate file on their own domain and everyone could follow the simple steps in the GUI to replace the expired certificate...
I might be wrong, I don't have data, but, as far as I can tell most users either use the installed browser or chrome (or whatever their tech savvy friends/relatives install for them).
Being a former ops guy the items you list resonate with me. On the one hand I do feel for the developers and hope they come up with a fix soon. On the other side, this is frustrating and there were some bad decisions made that a typical ops person would have pointed out and been ignored. The ignoring of ops guys until something breaks is something that has been consistent in my experience. Anyway for the sake of having an alternative to Chrome I hope they fix this yesterday.
For what it's worth, the (initial) mechanism for disabling add-ons (your 1) has been present since before Firefox 1.0. It was designed to quickly deactivate any malicious add-on as soon as it was detected, before it had a chance to do too much damage. In my books, that's a good thing.
Here, the mechanism that kicked in was the protection against add-ons that could have been signed with stolen credentials, which would make them clearly malicious.
Of course, it turns out that the problem was an expired cert, so a bug/human error. But generally speaking, I think that 1 is good.
> It was designed to quickly deactivate any malicious add-on as soon as it was detected, before it had a chance to do too much damage. In my books, that's a good thing.
I hate this attitude from security people so much. If for the sake of fighting malicious code you are crippling the software usability or my user experience, you are the malicious code.
I hate that attitude from entitled users so much. If you don't want security, you're welcome to have a malware-ridden system, but don't think that this means all users should have to put up with malware-ridden systems.
I wish that was true, but in fact I have no way to disable this and similiar amazing security entrenchments. The monthly device bricking windows updates, for instance.
If I can't do anything with my hardened computer, I don't care if is eaten alive by malware, it is useless anyways.
At work, as the guy who have to fight on behalf of the sysadmins and the users dozens of clueless security advisors who are hardening everything according to security best-practices written by similarily clueless experts, I'm seriously astonished by the common backward thinking. If you are blocking access to all users pdf files, for an instance, you are the malware, you are causing disturbance to the business operation and annoying everyone.
This petulant antagonism ("You are the malware!", "No YOU ARE!") between users and security is contrary to everyone's interests.
Go sit in separate corners, both of you. Think really, really hard about how both of your jobs are critical to the long-term success of the business. Don't come back until you've meaningfully internalized that.
I'm having hard time understanding what are you referring to. All I'm saying is that I had a perfectly working system and now it is no more functioning properly. Why would anyone see this as more secure is beyond me.
Similarily, when "security best-practices" are leading to hundreds of my users losing SSO access to their BI system, or hundreds of printers rendered unusable because of security update that requires administrative permissions for reinstalling the same drivers that were perfectly working so far, I don't care for your security benefits. They suppose to defend against the thing that you are causing. You are already damaging the organization with thousands of working hours lost, and everyone is frustrated, as a bonus.
Sorry, but where is the difference from a secure-system that allows central control - and a male-ware backdoored system?
All that is diffrent is the promise of non-maliciousness. Which often does not hold up. Cause money is corrosive to those little centralized empires of "all-can-fail-but-me".
Security is diversity, as in having a non-centrally controllable ecosystem, that is not a mono-culture. Your updates are the danger, your urge for control is the forrest fire.
Linux is not secure because its updated often. As package maintainer take-overs have show- that is even a vector.
Its secure, because its fragmented into a thousand small populations, which offer no real financially interesting attack vector for a large scale take over.
Linux's diversity also makes it difficult to package and distribute non-malicious software, so I'm not sure that's the poster child for how to do security without compromising usability.
(The situation is admittedly getting better with Flatpak.)
> If you don't want security, you're welcome to have a malware-ridden system
No, I am apparently not. Microsoft, Apple, and others insist on making it difficult. At least I found out today that I can install unsigned Firefox extensions once I switch to a special "unbranded" build. I'm glad Mozilla, at least, still offers that.
Here's the thing: I disable a lot of the security stuff you're not supposed to disable, when I can. I use a Jailbroken iPhone. My Mac has SIP and Gatekeeper turned off. Windows Defender is turned off on my gaming PC, and I lower Microsoft's driver signing requirements to the greatest extent allowed. I also ran an unpatched day-1 build of Windows 10 for around four years, with the autoupdate system forcibly neutered. (I now run LTSB, instead.)
I have never been bitten by a virus, ever†. I don't know if that's because of all the security measures I'm not able to turn off or because I've been lucky or something else. I suspect it's because I don't run dodgy software. Or maybe my life is a lie and all my devices have been infected for the past decade, and I never noticed.
In the meantime, I'm not seeing the upside to software forcing hardened security.
---
P.S. While I share their frustrations, I don't endorse the GP's attitude. I know that a lot of people really are doing difficult work with the best of intentions.
† Except for a handful of times when I was testing suspicious software in a disposable VM. That doesn't count for obvious reasons.
Just because you haven't been affected by a virus doesn't mean you never will. For example, the patch for the zero-day exploited by WannaCry was sent over windows update a few months before WannaCry existed. I personally use Linux, so Windows Update doesn't exactly exist for me, but I still update my system whenever such updates are available. Both SIP and driver signing are both mechanisms to prevent the installation of rootkits. If you do get a virus, such mechanisms would prevent it from hiding itself or causing more damage to the system.
Not running dodgy software is indeed a very effective way to not get viruses, but that doesn't mean you shouldn't take more security precautions if they are available. What if, for example, malware exploits a zero-day in your browser and successfully installs itself without any interaction from the user? Windows Defender and related antivirus could detect malicious activity from such malware and remove it on windows and Gatekeeper/SIP/driver signing and related systems could severely limit its impact.
Mozilla probably introduced extension signing to prevent less technically inclined users from having adware installed into their browser. X.509 introduces certificate expiration, and X.509 is the most widely used mechanism for signing things. The only reason you have this problem is because the folks at Mozilla forgot to renew that intermediate certificate. While I agree that it should be possible for users to disable extension signing, as users should be able to do whatever they want on their own system, you shouldn't blame them for forgetting to renew a certificate associated with an additional security measure built to protect users from malware.
> The only reason you have this problem is because the folks at Mozilla forgot to renew that intermediate certificate. While I agree that it should be possible for users to disable extension signing, as users should be able to do whatever they want on their own system, you shouldn't blame them for forgetting to renew a certificate associated with an additional security measure built to protect users from malware.
Well, we agree on everything, in that case. :) I have no problem with sensible defaults that can be adjusted, nor do I take great issue with Mozilla's specific mistake in letting the certs expire.
Edit: Also, just noting that I only lessen security measures when I have a reason, not because I'm rebelling against security practices or something. I use a Jailbroken iPhone because I run Jailbroken software, and I disable driver signing because I use unsigned drivers. It's not that I don't see the risks, so much as I'm very unconvinced the risks are worth the downsides, for me.
It's just the IBM mainframe priesthood reasserting its dominance, lurching from the tomb of computing history to save us all from ourselves. Nothing to see here.
Today, more than half of my open tabs disappeared in an instant, and were not even an option to re-open until either I waited around ("up to six hours...") or manually installed the workaround. All of my in-progress work in any of those tabs? Gone.
That absolutely qualifies as crippled usability. The mere fact of such a thing being possible is a usability defect. On what basis do I trust that my work is not going to disappear on me like that again?
Firefox will continue to have bugs. All software will continue to have bugs. I'm so sorry that you lost some tabs in your browser but shit really does happen and acting like this is some violation due to overzealous security controls is inane.
I did not say "all trust". Please don't presume to inflate my explicitly stated position — especially while also minimizing the impact this incident had on me, and others. I did not merely "lose some tabs"; those, I could just re-open. I lost work. That data, effort, and time are gone.
If you think this clownshoery hasn't cost Firefox any trust, then you're being as naïve as you accuse me of being "absurdly overdramatic" and "inane".
Bugs are a thing, totally conceded. Sloppy certificate management is, too, but it's an entirely other class of thing. Deliberately conflating them is at least as disingenuous a debating tactic as pointing at Chrome, which is utterly irrelevant to this incident. That's straight-up "whataboutism".
Full stop, this was foreseeable. This was preventable.
EDIT: Phrasing.
EDIT 2: I won't respond further to the same kind of tone.
Alright, I apologize for the tone. It's unnecessary to make something like this into a heated discussion.
That said, the part I was referring to is:
> The mere fact of such a thing being possible is a usability defect. On what basis do I trust that my work is not going to disappear on me like that again?
The possibility of a bug happening is hardly a usability defect in my mind. Or if you want to call it one, it seems like a perfectly reasonable one - this was a defense born out of necessity when malicious extensions were more of a problem.
And I think that the "On what basis" question definitely implies a total lack of trust, but sure, maybe not. The basis is that this is a single instance of a failure over the course of the features' lifetime, for a feature that has existed for absolutely ages.
I pointed to Chrome as an example of similar issues cropping up across codebases to show that these sorts of bugs do happen. I don't consider that whataboutism.
All bugs are foreseeable and preventable. Systems are complex. I think you're putting the issue in a very unfair light, even though it's very reasonable to be upset about time and effort that is lost because of the issue.
First, thank you for responding in a manner that invites a response, rather than demands refutation.
I understand your perspective, and appreciate your recognition of my own. That said, if you think I'm putting the situation in an unfair light, I think you're downplaying it at least as much.
In my eyes, this is no mere "bug"; it's an abject process failure. As a reply to another of my comments in this discussion suggests, this is more on the level of, "Oops, we forgot to renew our domain name...", than it is, "Gosh, we didn't validate the pointer returned by the frobnitz function, when the whoozle isn't initialized yet..."
Dealing with expiring certificates before they expire is covered in like the second week of Certificate Management 101, as it were. If it's necessary to stick an intermediate cert in there, then it's doubly so to keep it current.
> The basis is that this is a single instance of a failure over the course of the features' lifetime, for a feature that has existed for absolutely ages.
The plural of "anecdote" isn't "data", but an existence proof is an existence proof. That the problem has gone from zero occurrences to one, no matter over what period, literally makes it infinitely more likely to recur, if you want to be that reductive...
Public perception. For instance, one of the first comments on their post is this:
> Why not just post a link to the fix that can be installed WITHOUT enabling Studies? This sounds like a clever plan to get more people to share their data via Studies…
I definitely don't agree with that guy, and I doubt that's a majority opinion, but asking people to use a workaround that benefits them (Mozilla) after they broke things for a lot of people is bad publicity for sure. For what it's worth I think Mozilla is doing the right thing here, just it's not going to make them look great.
> using cert expiration -- I'm ok if the cert was revoked but that's a totally different discussion
CAs can delete certificates from their revocation lists after expiration, which means that you can't tell the difference between a certificate that was never revoked but merely expired and a revoked-and-then-expired certificate.
As an alternative perspective, I'm totally fine with FF disabling the extensions when the cert went invalid, and I'm also happy that it auto-updated itself to fix the issue. To me the optics are pretty good: a mistake happened and they were able to recover pretty fast, and my browser wasn't exploited by bad actors in the meantime.
No, just this one because it took me more than 5 minutes to not find a working fix, and this was such a massive fuck-up that I don't feel like sticking around.
I appreciate the condescension of both your comment and the person I initially replied to, but I honestly see your comments as saying, in more words, "Fuck the user." And that's fine, but why don't you just say it? Go ahead and type it, I want you to type what you really think about the users who are so dumb and fickle that they can't handle something so trivial as not being able to use their precious stupid add-ons like HTTPS Everywhere and uBlock.
Hahahah look at all the Firefox fanboys coming out of the woodwork to try intellectual bullying because they're mad I admitted to uninstalling a web browser.
I am positive my first comment on this thread was not an order of magnitude over the line, if we can quantifiably measure such a thing, and it got censored anyway. "I just switched my browser. Bye bye Firefox."
I mean if you want me to be reflective it's really not going to work if we're refusing to admit that either both of these comments should have been censored or the one above should not have been censored. But I get it, life's not fair and the squeaky wheel gets the grease.
Firefox zealots get to railroad commenters on HN, noted, I certainly won't engage in these discussions again.
No, just this one because it took me more than 5 minutes to not find a working fix, and this was such a massive fuck-up that I don't feel like sticking around.
I appreciate the condescension of both your comment and the person I initially replied to, but I honestly see your comments as saying, in more words, "Fuck the user." And that's fine, but why don't you just say it? Go ahead and type it, I want you to type what you really think about the users who are so dumb and fickle that they can't handle something so trivial as not being able to use their precious stupid add-ons like HTTPS Everywhere and uBlock..
Flag it again, for the F word. Flag the comment I responded to as well, for consistency. :)
No, we’re not flagging you for your use of swear words; we’re flagging you for your lack of construction to the conversation. You clearly are not interested in having a conversation
Switch to what? Chrome? Because you don't like having to re-opt-in to studies? That would be ludicrous given Google's privacy track record. Opera? They're owned by a Chinese investment firm now. Edge? MS's whole OS is based on data collection.
CVE-2016-9179 was published about Lynx in November 2016. Lynx took more than 5 minutes to release an update, with the fix included [1] in 2.8.9dev.11 not reaching a production release until July 2018 — almost two years after the CVE was published.
With a response time like that, I don't see how Lynx will satisfy their "5 minute fix" need any more than Firefox did.
Instead of pretending you're perplexed about my desire for a "5 minute fix", you can go back to the comment that said everyone here who has a problem with the issue Firefox caused is crafting a "narrative that appeals to emotion."
Read that, I don't know maybe ten times, and then read my comment which said I just switched my browser, no emotion involved.
I'm a man of action, baby. You can deliberate over my thought process all day, wondering if I would pick this browser or that one, or maybe even if I felt bad that I violated my loyalty to Mozilla Firefox (blessed be thy name). But don't think too hard, I've already moved on to the next decision in my life.
Lynx, Midori, Safari, IE6. Maybe Palemoon, Samsung browser is another possibility. Maxthon was popular for a bit. Maybe you're super 1337 and run in only curl or wget.
The handling, or the bug itself? Sound like the damage control is fine (although worrying that they have no way to distribute hotfixes more rapidly than this).
The bug in the first place, on the other hand, seems pretty negligent. Not that it's incomprehensible, just pretty stupid.
Have they stopped whitelisting Facebook and Twitter in their "tracking blocker" yet?
The BS coming from their blog post surrounding this whitelist makes me distrust them completely: "Loading a script from an edge-cache does not track a user without third-party cookies or equivalent browser-local storage" (...) "Given that most users on the web share IP addresses with other users because of NAT, it is unlikely this can be used to reliably track users"
Not only it's quite possible to know if the user is behind CGNAT or not, meaning the tracking works just fine for millions of users, but carriers have been known to inject user IDs in the replies of users behind CGNAT.
I'm not sure I care how unfair the characterization is. I heavily use container tabs — ahem, 'usecontainers — and all of my open container tabs disappeared at once, with no indication of why or what to do about it, when this happened. I lost an absurd amount of work and state because of that. I only knew what caused it by inference, because I'd just previously read The Fine Article (which, btw, gave no indication that losing state like that was something I should expect, merely, "No active steps need to be taken to make add-ons work again"...)
I still prefer Firefox over all the other browsers, and will continue to use it, but the project has lost a lot of trust and goodwill over this.
The optics are indeed awful, and this was fully preventable. Firefox fucked up, full stop.
Based on the timing of initial tweets and blog posts on this fiasco, I'm pretty sure I was in the first 10%, if not first 1%, of people who experienced this. And I was in a plane at 36,000 feet trying to work on a cross country (U.S.) flight when suddenly about 130 tabs in 7 windows disappeared. Really, REALLY bad. Panic, frustration, confusion...
I was more than 50% sure that all was not lost forever, that it was some "glitch" (Extensions all showed the same bloody red status), but I was tweaked. I work in security (embedded systems, not computers/IT) so I have a very good understanding of certificates, TLS, PKI, etc. There are many ways things can get out of whack if the people in charge screw up.
Regardless, this is embarrassing, dare I say shameful (pretty much almost up there with "Ooooppsss... we just lost our domain - it expired and no one thought to renew it)
Come on, guys, get it together. Have a procedure, document it, practice it, stay in front of it.
EDIT: after installing the fixed XPI, I have to sadly report that all data has gone. All my carefully-managed containerized life was wiped clean. Heads should roll.
Complete shambles. And the worst thing is, I suspect it's all a plot to have more people opt-in to the shitty telemetry. Otherwise, why not push an update through the usual channels? Had it been a security-related fix, would have they used "studies"? I bet not.
I am sure an update will be pushed quickly. I'm waiting too, as a testing user.
You can fix this temporarily via setting "xpinstall.signatures.required" to false. Toggle it back to true once update is released and you install it.
Meanwhile I'm hijacking this comment that is to the upper parts of the tree to state this: the way the community treats Mozilla and Firefox is horribly, inexplicably, unacceptably unfair.
This is nothing compared to innumerable other fuckups in software history, and even recent ones like goto fail, heartbleed, or Chrome logging you into Sync w/o notice.
This is a mistake, an easily recoverable one, and is not intentional or malicious. Firefox is developed in the out and open, all the processes are public. And people, with an absurd entitlement and malice, go as far as to call things backdoors or malware. Meanwhile the alternative actually is a backdoor ridden malware.
> This is a mistake, an easily recoverable one, and is not intentional or malicious.
While I agree with most of your comment, you're downplaying the severity here, especially since, IIUC, this situation also affected the Tor browser, disabling NoScript. If regimes like China were on the ball, and succeeded in escalating the remote code execution vulnerability into into deanonimization, this debacle may end up having a death toll attached to it.
They only point out the opt-in instructions for the few people that voluntarily opt out of Shield studies and wish to get the fix sooner.
Most Firefox users have that checkbox enabled by default, and so most Firefox users received the fix within 0-6 hours of the blog post's publication.
HN readers often take special care to prevent Mozilla from updating Firefox, but that in no way represents the wider population of either all addons users or all Firefox users.
Then it seems it’s a good thing they had a system in place that could deliver the quick fix in less than a day, on a Friday evening, when shipping a normal fix could have — is — taking longer to ship than that quick fix did.
> it’s a good thing they had a system in place that could deliver the quick fix
No, it's not. "Studies" is not a security-related mechanism and it didn't exist in the past, when fixes were rolled out very quickly anyway for security reasons. "Studies" should not be relied on to be a fix-delivery mechanism, because it just isn't.
This is not even about privacy, it's simple good engineering sense.
It seems like your objection to the Normandy system is that the UX surrounding it includes the word “Studies”. I am grateful they chose in this instance to prioritize repairing addons worldwide over the confusion that word has caused you and potentially others. I assume, having seen this and other such comments delivered with outrage rather than thankfulness today, that they will re-evaluate the UX surrounding the Normandy system to ensure that it more clearly designates non-study changes as such.
It's not about designation, it's about control. If Mozilla really cares about trust, they shouldn't mix their update delivery system, which should care for timely security-related material, with general telemetry, data-gathering, and experiments.
I use FF because I care about principles. Otherwise I might as well just let myself be exploited by Google, MS, Apple and friends.
Ah, you object to Normandy’s design in some manner. That’s being hashed out in today’s Normandy thread, and if you haven’t already read that link you’ll definitely want to:
Why does it take longer to ship a general fix / update? They don’t do a full regression for the studies fix? Update mechanism doesn’t check for updates as often? I couldn’t find any information on this yet but would love to know.
“How long does it take to build, unit test, performance test, and QA check a new Firefox release on every supported release of macOS, Windows, and Linux platform?” is absolutely a question that outraged users are trying not to confront. You’re right to ask it, so don’t let the downvotes get you down.
Presumably the testing burden for a preference update using Normandy is smaller, as (and I’m guessing wildly here) fewer things can be altered with Normandy and therefore testing can be simplified to exclude, for made-up example, “the code-signed binary can be executed on all platforms”.
Sounds reasonable to me. Would love too see that information on the official Mozilla blog for the post-mortem. I personally think it is great to have a mechanism to push fixes quickly - whatever the name is. I just don't understand why this mechanism can't be the regular update mechanism.
Another workaround if you don't want to enable "studies" is to manually re-load the add-ons in Debug Mode. I don't know the full consequences of this, but Firefox seems to be behaving normally having done it.
Go to about:debugging from the address bar. Right at the top is a button to "Load Temporary Add-on", with a checkbox "Enable add-on debugging". (On a Mac, the add-ons are in ~/Library/Application Support/Firefox/Profiles/«ID».default/extensions (assuming that you have only a single profile).)
They should stay enabled until Firefox relaunches.
Granted I'm using Nightly (and previously disabled extension signing in about:config), but now all my themes are disabled, even the default one apparently, though that is what it is using. Cannot be re-enabled. When do I get my dark theme back?
Also...my default search engine is now Amazon.com?? WTF is going on.
EDIT: Also my only search engine. Heck of a job Mozilla.
Nope. Maybe I should restart Nightly again but...I just added Google back, and it was not as easy as I'd expect. The Firefox "add search engines" page is a mess and straight Google search was like 3 pages down.
Also I found another dark theme that is somewhat similar to what I had (but not official Mozilla) and installed that. All the default ones are still disabled.
Thanks.. Same here on every machine.. The hotfix update has been installed and add-on functionality is indeed recovered, but still i only have "amazon.com" as a pre-installed search provider and have been unable to restore them to their default settings without creating a completely new profile. Besides, i would be rather hesitant to install search providers from that list, imho.
503 comments
[ 2.8 ms ] story [ 244 ms ] threadIt is astoundingly disingenuous to act like these things are comparable.
>It is astoundingly disingenuous to act like these things are comparable.
Why aren't they comparable? In both cases, it's Mozilla pushing code to the end user. There's a different process behind both but calling one a frontdoor and one a backdoor seems apt to me.
>and which can make such large scale errors as evicting all extensions
Normandy was not used to disable all extensions. It was caused by a certificate expiration error completely independently. Normandy is being used instead to work around the error until a more permanent fix can be issued.
Because regular auto-updates are easy to understand and turn on/off. Normandy is clearly extremely hard for many users to understand, enabled by default, and hard to disable.
HN has this problem too.
“I (name the individual accountable) will give you an update at 12:00 PT (name a time) as an update to this post (name the communication channel) with the current status and latest information on this issue (don’t promise time to resolution, just time to info).”
Simple, clear, concrete, and unambiguous. I had hoped that Firefox had better communication procedures in the event of global-impact P1 issues.
Usually, this mechanism is explained as being helpful to ensure a rollout of an experimental update can be rolled back if it's failing. That's not so much a concern in this case, I think. But this mechanism has another effect: it works as a solution to the thundering-herd problem. Every browser updating at once is bad, not just for Mozilla's servers, but for every piece of Internet infrastructure that those browsers (and their arbitrary set of addons) talk to when they update/restart. Within the time budget you have for running a rolling update, you ideally want as few machines updating concurrently as possible, just because you don't want to generate mysterious correlated traffic bursts that make NOCs paranoid.
You go a bit in, wait to see if the canary, then go further.
Of everything mozilla has done recently, Servo is one of the things I'm most positive about.
Basically all refactors took months and months and months because of this. There was no way to address the accumulating technical debt.
As a browser, it works much better, and as an extension developer as well, I'm glad I can write one extension that works in most browsers now.
Yeah, it sucks they removed the level of customization they used to have, but overall that changes are welcome.
That said, this whole thing is a huge weakness to me: having some organization decide what I can put onto my own computer is a frustrating tradeoff, but now it's gone from a nuisance to a real fucking problem.
I'm not an imbecile. I can manage my own browser plugins. Give me a switch to install XPIs without having some authority sign them, or at least verify once on install and piss off after that. I don't like my devices phoning home every 30s to make sure I'm "safe."
The hotfix xpi link mentioned elsewhere in this thread works on mobile.
Out of curiosity, how did you know that? What does this actually do?
I stumbled upon this myself earlier
What did (seem to) help was setting app.normandy.run_interval_seconds to a small value (21). At least just a couple of seconds after I did, all my addons came back.
Edit: plugins -> addons
It was a private email service. The complaint was that the private email service also happens to be used by Antifa members. Which is unsurprising.
This is like complaining that they shouldn't give money to the Tor project because it gets used by unsavory people, too.
https://news.ycombinator.com/newsguidelines.html
In 2019, I still use hundreds of tabs - and Firefox handles it with grace. RAM is there to be used, and this is the perfect use for it.
Also why it took 6 hrs to assign P1 to the bug
I would assume the delay in assigning P1 is really just a result of assigning P1 not being as high priority as fixing the damn problem.
So we can come to the obvious conclusion about Mozilla, then? No good "engineering" managers? Miss one reprioritization and you're out! This is what sane people think?
If you want to complain about knee-jerk overreactions, I think you might want to look in the mirror first.
I understand why an add-on update or new installation would be prevented from succeeding by a certificate expiration. But why would a certificate expiration prevent an already-installed from running? Any already-installed add-ons were previously validated at installation time and should (IMO) run as-is. It seems unnecessary to continuously check the status of an add-on's certificate if it has not been changed. Am I missing something?
I look forward to joining your class-action lawsuit.
Besides that, this sort of "but we hid something in the t&c-s so now we can shit on you" is the sort of thing I expect from over commercialised companies, not from what was once a paragon of the FOSS community.
FWIW class-actions don't exist in UK.
Revocation is, I think, the actual reason.
See profile/blocklist-addons.json
Not sure how not renewing a certificate and letting everything get disabled is useful. It's only useful if cert key leaks.
No, they do not need to. They decided that they want to. Remember that there was a time before certificate signing.
And now the decision to be able protect those who install crapware is also harming those who never had those issues.
It's exceedingly poor ethics.
I had to go thorugh profile/extensions.json and set appDisabled to false to make my extensions enableable again.
Because people were staying up until the wee hours of the morning working on fixing it instead of toggling priorities in Bugzilla. This was treated as a five-alarm fire.
I don't think it bothers me personally but it's funny you said that. Presumably you mean a "'no-alarm fire' because who has time to set off an alarm when there's a fire to fight"?!
https://en.wikipedia.org/wiki/Multiple-alarm_fire
The 5th level or a five-alarm fire, is the top level where you're basically talking all hands on deck.
Still seems like an ironic choice, applies equally to the people saying it was DEFCON1. I'm pretty sure the military actually have displays indicating the status, but again that's based mainly on movies.
However, I bet it’s likely they have procedures and policies for work that first involve signaling like for example the priority level.
I’d be willing to bet lots of things surrounding this issue weren’t handled in a by the book manner. So if you are always going to wing it, why have a book (or a public priority level system) at all?
"The book" is something that needs to change and improve like anything else. Sometimes, that means you'll still need to wing it and add that thing to the book later.
Second, because we care about solving problems, not being bureaucrats.
The non-bureaucratic thing to do, as has been pointed out many times of course, would be to give users the power to override the cert signing check as an advanced option.
Saying "we were too busy fixing to communicate" is actually a really bad sign, because it's not just about what you are communicating to the outside world, but also, for example, about making sure people that need to be brought in are getting consistent information.
The question is about why someone didn't set the priority setting in the internal Bugzilla sooner. There are a couple of reasons for this. First, that's not really what priority is for: priority is so that engineers on major projects like WebRender know what is most important to work on. It's not an effective tool for emergency responses. P1 doesn't summon on-call people. Second, everyone who was able to get it fixed was already on it. Bugzilla priorities don't make people who wouldn't otherwise be aware aware.
Unlike that of Mozilla, your issue tracker at Google for outages is private and internal-only. One of the many reasons it is private is so that people who don't know the organization's operational processes don't come in and start making incorrect assumptions based on what they find there. I think it's laudable that Mozilla works in the open so much, but comments like this one are some of the downsides of doing so.
I'm sorry if i offended you. Really. I don't really think it's that uncharitable, but hey, offending you wasn't my intent, and if you were, that is what matters.
For what it is worth: your response to the parent took them to task for even attempting to state something: "Because people were staying up until the wee hours of the morning working on fixing it instead of toggling priorities in Bugzilla. This was treated as a five-alarm fire."
This is a pretty rude response (which should never be happening regardless of what they wrote), and if you intended to convey that you had incident managers handling it and they didn't get around to it yet, you did not.
"Unlike that of Mozilla, your issue tracker at Google for outages is private and internal-only. "
I'm really unsure why you decided to add a completely irrelevant and unnecessary attack like this.
Honestly, it just makes me think less of you and makes me sad. You are usually a very sane and even keeled person. I'm sure you were not having a good time, but i still don't think this was okay.
" One of the many reasons it is private is so that people who don't know the organization's operational processes don't come in and start making incorrect assumptions based on what they find there."
Which is incredibly ironic, since it is not in fact as private as you seem to believe . Your own assumption here is completely and totally incorrect.
Maybe before you decide to add completely irrelevant and unnecessary attacks, at least verify they are correct?
" I think it's laudable that Mozilla works in the open so much, but comments like this one are some of the downsides of doing so."
You seem to have taken my comment incredibly personally, and your response seems very far out of proportion.
If you want to have a discussion, you're gonna have to tone this down a few notches.
FWIW: Right now most of your comments in this story read incredibly defensive (and i'm not just talking about this one)
I would stop and give it a rest. It is not projecting a good image.
Of course, that's just my perspective.
Really hope tomorrow is a better day for you.
Random user: What the fuck is a tree and why is the priority of this not higher yet?
Not to be too glib, but any random user who is technically literate enough to know where to seek out Firefox's issue tracker and how to find the issue in question, and who has such a thorough understanding of issue trackers that they understand that such a thing as a priority field exists, is also going to be savvy enough to read the very first comment, and will be well aware of what it means, and will, one hopes, be rational enough to understand that the flurry of activity indicated by the issue in question is more important than a passing field in the bugzilla database.
If anyone expects Mozilla to take power users seriously, then we need to focus our criticism on the things that aren't just imagined trivialites. It makes me frustrated that the people who irrationally fly off the handle at the slightest perceived provocation are also the ones who implicitly encourage Mozilla to write off power users as more trouble than we're worth (and after ten years of watching these incessant whining non-comments on HN, I don't blame them anymore).
[1] GitHub mirror to not stress their infra: https://github.com/mozilla/gecko-dev/commit/1d1260c7615f1d9a...
https://storage.googleapis.com/moz-fx-normandy-prod-addons/e...
https://news.ycombinator.com/item?id=19825921
I'm not clear if they rehosted the XPI or if that's the original mozilla url.
I'm not too worried about it either. The only reason anyone is clicking on this fine link is because firefox only lets you install addons signed by Mozilla. And since the typical signing process gives addons signed by the broken intermediary we can be pretty confident that this wasn't just signed by mozilla, but is the original study.
In general caution about installing software from random links is definitely a good idea though.
Edit: Looks to me like it's an original mozilla url (judging by github comments on mozilla/normandy - I haven't found an official source saying it is official due to lack of continuing to search: https://github.com/mozilla/normandy/pull/1697)
unzip *.xpi
nano META-INF/manifest.mf
gives me
Manifest-Version: 1.0
Name: background.js Digest-Algorithms: MD5 SHA1 MD5-Digest: pcBRGwbuhPz06VrGWmAitQ== SHA1-Digest: szDd6YcB3bpF+NusZhEHhmMDi5U=
Name: content.js Digest-Algorithms: MD5 SHA1 MD5-Digest: CGOATrflEiq+QEu1IZlFvQ== SHA1-Digest: ps2bMGGRQdb4E7VOakqQEhJ8M5c=
Name: content.js.map Digest-Algorithms: MD5 SHA1 MD5-Digest: FY98a5hwQKH3g1fKcGK04A== SHA1-Digest: bAzZBP+YQ3EDWUXpqzKcTUw35Y0=
Name: manifest.json Digest-Algorithms: MD5 SHA1 MD5-Digest: eEm4sDKemttFN7G7JeLo0g== SHA1-Digest: 5W8OY1mk3QjECHzHna00iNXo9mM=
Name: experiments/skeleton/api.js Digest-Algorithms: MD5 SHA1 MD5-Digest: 0RBtD2TRmeE30v9+4TxXYA== SHA1-Digest: 2Uq9PO2H1iks/Cb7VAkfGrrD6hA=
Name: experiments/skeleton/schema.json Digest-Algorithms: MD5 SHA1 MD5-Digest: nSzuviuP+VtUvjE4IyIVhQ== SHA1-Digest: W311W+MXcHSsHIVFP15zxGUmQS8=
===
The hashes that certify the integrity of the files are under rigorous protection of ... MD5 and SHA1 (!)
There might be a very difficult preimage attack on MD5.
There's no evidence of a preimage attack on SHA1.
There is absolutely no way you're doing both at once.
And that gives it access to use `Cc`/Components.classes?
Sucks for the Mozillans who are scrambling right now though. Hope they get a long weekend to compensate.
I have little sympathy for people who ruined my own life - they push a fix and get feet up, while I'm left to reconfigure all my addons and containers for days.
You can manually install the certificate instead of using the "add-on" in the OP. Copy https://pastebin.com/rpByJV9P to a text file, change the extension to .crt, and then use the Import button in the Authorities tab I mentioned before.
Studies _must_ be opt-out given the amount of users Mozilla says the fix covers, and they're basically a form of telemetry, in their intended use anyway.
Has the "privacy" community finally jumped the shark?
I just don't want to enable shield studies, because it looks to me like they haven't disabled the other shield studies while distributing this fix, and I don't want to install the other shield studies.
vs
"whenever mozilla has crazy marketing or security ideas in the future, let them immediately and randomly install whatever, which maybe seems like a good idea for the mythical average user but is probably terrible for you"
EDIT: Thanks to HN User gpm for suggesting a possible fix for this [1]. Right-click, save-as the XPI to somewhere on your computer (or use curl, wget or whatever tool of you choice), and then run it within Firefox. That might work (it did in my case).
EDIT 2: Also, interstingly, the blog post does have an update saying "There are a number of work-arounds being discussed in the community. These are not recommended as they may conflict with fixes we are deploying.", so, use at your own caution I guess.
[1]: https://news.ycombinator.com/item?id=19828669
Install from a random web link to file on a "cloud" server.
What could possibly go wrong!
You're right that in general training them to listen to anonymous forum posts is less than ideal, but all in all I'd rather they have a working browser. As a side benefit they get to see posts like this that rightly point out you shouldn't trust strangers on the internet too much.
If you visit the Mozilla homepage, there is nothing to acknowledge the problem (at least at the time of writing this message). Let's try the Support page. Where is it? Scroll down to the bottom of the lengthy Mozilla homepage to the page footer to find the link. (How many visitors will make it to the bottom?)
When you click through to the Support page, an easy-to-miss banner in tiny text appears at the top of the page that mentions the problem - screenshot here: https://imgur.com/a/TAHZSWa
Additionally, when the add-ons are disabled, Firefox misleading says: "These extensions do not meet current Firefox standards so they have been deactivated". This is probably a generic message but it's also an example when a generic message is misleading.
Finally, poorly-named settings like "Normandy" and "studies" that give no hint of their meaning only adds to the confusion.
I love Firefox. It's my daily driver. It will continue to be. But this is a huge fuck-up and they're probably going to pay big in usership because of it.
https://github.com/intika/Librefox
However, Librefox is only Firefox with some configuration changes. It is not a whole new build, and it wouldn't have protected you from this problem since the problematic addon cert checking is still there.
Note that this would have happened even if the browser never communicated back home - this problem was triggered via an unwitting time bomb of sorts, not because Mozilla actively took an action that inadvertently broke something.
I was never worried about Mozilla's telemetry. I happily enabled all feedback/telemetry options because I genuinely wanted them to fix any problems. I understand that this problem is not caused due to phoning home.
The decision process that led them to make it impossible to load an extension without it being signed by them is problematic. It means that I don't trust them with all my debugging data. I don't know what other time bombs are hiding inside the code.
What would happen if Mozilla ceased to exist tomorrow? Will existing firefox installs get bricked after some time? Even big bad Microsoft allows you to install unsigned hardware drivers if you dismiss the scary warning.
> The Nightly and Developer Edition versions of Firefox have a preference to disable signature enforcement. There are also be special unbranded versions of Release and Beta that have this preference, so that add-on developers can work on their add-ons without having to sign every build. To disable signature checks, you will need to set the xpinstall.signatures.required preference to "false".
Otherwise I guess there’s also IceWeasel if you’re on Linux?
Custom firefox builds offered by some linux distros are a better choice, yes.
Guess we'll see a post-mortem soon and get to know how did this even came to be.
And now Mozilla are saying that the "fix" is to allow them to install & run "studies" on my machine? What are they smoking? I'm having a hard time trusting a company that randomly & remotely disabled all my addons, regardless of the cause.
Some software lives a LONG time and it's fine, and it's up to the user whether that software is still useful to them or not.
Seriously how many posts do we see on hacker news about like "We rebuilt this ancient machine from the 1970s to learn about it." People care about computing history. Not everyone, but there's no reason to force your software to break because the calendar rolls over. Remember Y2K? Things often live a LONG time. People are still actively writing Fortran and COBOL. The short-sightedness of this is amazing, as is the condescending "we know what's best for you" security argument.
So a secure system would let you run Doom, but could forbid:'
- Access to the filesystem outside the application domain due to potential for exfiltrating or destroying user data
- Likewise, access to global system data may be limited
- Access to the network due to (raw TCP/UDP) traffic not having been audited for security, and the network connectivity being usable for exfiltration
- Access to run full-screen due to the ability to perform user phishing attacks by presenting fake UI.
- The ability to disable system-registered keyboard sequences (on windows, such as the windows key or sticky keys)
- Access to mouse events outside its window
- Access to key scan data, although this likely will be emulated
- Access to change display color modes to e.g. 256 color indexed, although this will likely be emulated as well
Is that really true? Would it connect to 802.11m WiFi router? Would you consider it secure enough to open your banking website on it? The bar is not just booting up the machine. The bar is whether the machine is usable (secure).
Sure. It's using OS networking APIs. Or running in a virtual machine.
> Would you consider it secure enough to open your banking website on it?
If I'm running 20 year old software, it's probably to interact with a legacy system. There are still businesses that run on like 486's with Windows 3.1. This is more common than you think!
> The bar is whether the machine is usable (secure).
The bar is whatever I WANT it to be, it's my machine, and it's pretentious of a software developer to assume they know what I'm using the software for and what my best interests are. For all they know I'm using the software in a museum, 20 years from now, about this era of computing.
I think that's a reasonable point of view. However, for such users, it's best not to use software that's largely developed for masses who just expect the software to work. It might be best to just checkout the source code, and build your own binary. Sorry for being rude :(
And then you'll simulate a time appropriate for the device/software. As a date before 2038 to not have unix time overflow. Or 2000. Or any other time specific bug.
Or how often did you have to "fix the internet" for one of your relatives because their damn CMOS battery died? Yeah, time seems to be quite relevant for trust.
And I don't even like mozilla enforcing signatures for addons that strongly, but people can go overboard.
Which is impossible unless you're either running Linux or running Nightly or Developer Edition. That setting is willfully ignored in normal Mac/Windows/Android builds most people are on.
"A certificate chain has expired, do you want to disable all add-ons?"
How hard is that?
If you think that’s trivial, I challenge you to go build it. It might seem warranted in hindsight, but thinking about all failure cases ahead of time is hard. If it weren’t, we’d not have bugs.
I don't think it's trivial. The critical element here appears to be "who gets the final say" and not "this is to hard to code".
They manage to disable the "Enable" button for addons, and managed to consider this situation enough to provide a justification that (paraphrasing) "we do this when we don't want the add-on installed", which is harder to do, they've added extra tests, added complexities and done all the consideration. They've just chosen to remove the final say from the user and give it to themselves.
It seems consistent with their recent behaviour.
“I wish it would do that.” is a much more charitable way to phrase your complaint.
As it happens I've just had to flip "xpinstall.signatures.required" and it's working for me. So it seems "not at all hard [for them]" was the answer.
FWIW people have chipped in saying this specific issue was raised, so it's not that they hadn't conceived that such a situation could occur (indeed that's surely why the config above exists).
The add-ons were signed by a certificate with an expiration date, which means that the add-ons are trusted until that certificate expires, not that they're trusted in perpetuity. It's not a hidden dead-mans hand; expiry is and has always been part of the process.
I think it's arguable that it shouldn't be part of the process, and having things like 20-year expiry satisfies the letter of the spec while being even worse than no expiry, but it's not a hidden dead-man's hand. It's how it was designed to work, and isn't considered optional.
It's a classic and ongoing debate of "who knows best?" -- the vendor, or the end user?
This doesn't need to be done through a dead man's switch (expiring certificate) that someone will forget to renew.
I do think that the UX should ideally be a bit more graceful; one of my add-ons is Multi-account Containers and its being disabled suddenly caused the window I was actively browsing in to just close, among other side effects.
But that kind of UX polish for what should be an exceptional case is obviously not going to be super-high priority, unfortunately.
You cannot rely on check at extension install. That would assume that all malicious extensions are installed via FF proper. Oracles crapware bundling in the Java installer taught us that’s now how things go. You cannot remember the trust flag when an extension is installed via FF as a crapware installer could just set the trust flag, too. After all, that storage would be accessible, too. You cannot sign or encrypt that trust storage as the key material would have to be kept locally and would be accessible to the crapware installer.
And I most definitely don’t want my browser to fail into an unsafe configuration.
(And yes, I know browsers should stay up to date etc. etc., but come on, no software should just stop functioning because of a calendar)
And to the downvoters: doesn't this entire fiasco ENTIRELY PROVE MY POINT?
If users don't want to stay up to date, that may be unwise, but that's there call.
If you're worried about a cert being compromised long after expiration and used to back-date a signature, you can show a warning, add a second signature with a local key at time of installation, use a blockchain to prove age, have a timestamping service generate its own signature at time of generation, etc.
I also did a bunch of sketchy registry hacks to fix that. (I keep up to date, but I don't need microsoft restarting my system at random hours)
No.
All it proves is that certificates expire (which is a Good Thing (tm)). If you depend on online certificates to verify content, something like this can theoretically happen.
Hypothetically, lets say they took the opposite approach, and only checked the certificate date on installation. What would have happened? There would have been a brief period of time where people couldn't install extensions, it would have been fixed in a few hours, and this story would probably have like 20 upvotes and fallen off the front page in like 10 minutes, if it ever got there in the first place.
Now let's briefly look at what's actually taking place: a bunch of people's browsers broke. It broke in scary ways for some people that were using extensions for privacy; IE, their security might have been compromised by this decision by Mozilla. Mozilla is probably going to lose users over this. Their reputation is damaged. Not only that, but now people are evaluating other decisions that Mozilla has made separately from this in an unfavorable light (Studies and Normandy, specifically). The computing industry loses out on this too: we're all better off for chrome having a viable open source competitor. We should want Mozilla to do well, whether you use their browser or not.
Which outcome do you think Mozilla engineers would be preferring today?
Instead of disabling this historically working feature which normally works great against hostile attacks such as MITM and malware this problem would've been avoided by a simple cron script which runs daily, and checks for expired certificates used within the infrastructure (both interaly used and externally used).
The main competitor you mention, Google Chrome, is a terrible privacy hazard. This situation does not change that.
As to your argument about security: doing the check on install instead of all the time, as I suggest is preferable, still protects against MITM and malware. The only argument for expiring software based on a calendar is that it might be a security risk if it's out of date. But first off, that is highly dependent on what the extension actually does. Also clearly we can see it's also a security risk to turn off privacy extensions without warning based on an arbitrary signing certificate.
> > I'm not putting words in your mouth [..]
Yes, you did:
> "So you think [..]"
>> "So you think [..]"
Um, I ended that sentence with a question mark. Implying a question. But you conveniently left that out of your quote, so amusingly now you're putting words in my mouth. (Although who cares? I don't!) Maybe I should have phrased it "So do you think" but it would have essentially been the same sentence and typing on my ipad is annoying. I was asking you if you thought this was a good thing for Mozilla. Which you dodged to turn this into a debate about proper etiquette instead of answering the question directly. It was a leading question, sure, but I was making a point.
Sounds like a good thing. Probably sounds like a good thing to some of the engineers at Mozilla.
The computing industry loses out on this too: we're all better off for chrome having a viable open source competitor.
I want a free competitor, not an open source one. And this is the most prominent example of that somewhat subtle distinction: Firefox is open source, but with all these backdoors it's no longer free, neither in spirit nor in practice.
P.S. On the other hand, it's probably indeed better to have at least two serious competing browsers, even if they are both non-free.
You use a browser that has remote update capability, which allows them to install and run new software on your machine all the time. There is a whole separate section of the Preferences that says "Privacy" in large print that has a section that clearly identifies the Studies feature and lets you turn it off. And you use a browser that lets you install privacy-enhancing add-ons in the first place, and in fact which invented the whole concept of add-ons. When the browser discovered that it couldn't verify the add-on integrity with a valid cert, it did what it's supposed to do, it disabled them to protect you from someone backdooring these add-ons.
Someone at Mozilla fucked up, and they're trying in good faith to fix it. I don't know what else people are expecting them to do, putting on sackcloth and ashes won't resolve the problem.
All your entitlement to control your own machine is melting away.
Here's a metaphor: Let's say you let someone seemingly trustworthy watch your kid. (In this metaphor you have a kid). And they let your kid get a broken arm through gross negligence (let's say they passed out drinking beer), and then someone said "well, obviously, you should have never trusted that person, after all, they can do anything with your kid while you're gone, so why are you outraged?" You probably would still be pretty outraged right? You would certainly question your decision to trust them, but at the end of the day you have to trust someone, you'd be a complete shut-in if you could never hire a baby-sitter.
Eorum est humanum.
Can you elaborate what's your concern with "studies"? By installing Firefox that updates automatically, the user is already giving control of the software and letting Mozilla decide what's the best. How is modifying software logic using studies different than modifying logic by updating the binary?
I'm asking because there I can imagine that there is a benefit for Mozilla to develop a feature that enables studies without sending data. It could be used to fix a broken feature or a broken logic (as in the case of expired certificates here). So, I'm not convinced that enabling studies always means that your data gets uploaded without the consent. Can you point me to privacy whitepaper / source code to backup that statement?
"The Nightly and Developer Edition versions of Firefox have a preference to disable signature enforcement. There are also be special unbranded versions of Release and Beta that have this preference, so that add-on developers can work on their add-ons without having to sign every build. To disable signature checks, you will need to set the xpinstall.signatures.required preference to "false"."
https://wiki.mozilla.org/Add-ons/Extension_Signing
Preferably someone who doesn't go to meetings and installd updates.
Yes, it's unfortunate, I'd expect them to meet it head on, push a tested fix in a timely matter, admit a mistake was made, explain publicly how/why and apply learning moving forward. Beyond that, what's your expectation?
There was a chain of bad decisions that led them here though: 1) thinking it's ok to disable software after its installed (using cert expiration -- I'm ok if the cert was revoked but that's a totally different discussion), 2) Taking more control of people's local software than many people are comfortable with, especially considering that their main market is tech savvy people that tend to be more sensitive to this than most 3) Making some of these things opt-out rather than opt-in, giving the perception that they may value data collection and control more than their users privacy.
Is that so, though? Firefox is still being used by millions of users, and I doubt those are only the tech savvy internet users.
(Then again, this mostly applies to Firefox users using add-ons, which probably has a higher share of technical users.)
it's also entirely predictable that a non-negligable fraction of users -after enabling studies and verifying everything works again- will ... simply go on with their lives and forget about disabling studies...
I also don't understand why the certificate graph is not exposed through a user interface, so that the user can add and remove certificates, or enable and disable certificates at their own discretion. This should have been obvious when the certified add-ons were introduced. Then all they would have to do is host the certificate file on their own domain and everyone could follow the simple steps in the GUI to replace the expired certificate...
Here, the mechanism that kicked in was the protection against add-ons that could have been signed with stolen credentials, which would make them clearly malicious.
Of course, it turns out that the problem was an expired cert, so a bug/human error. But generally speaking, I think that 1 is good.
I hate this attitude from security people so much. If for the sake of fighting malicious code you are crippling the software usability or my user experience, you are the malicious code.
If I can't do anything with my hardened computer, I don't care if is eaten alive by malware, it is useless anyways.
At work, as the guy who have to fight on behalf of the sysadmins and the users dozens of clueless security advisors who are hardening everything according to security best-practices written by similarily clueless experts, I'm seriously astonished by the common backward thinking. If you are blocking access to all users pdf files, for an instance, you are the malware, you are causing disturbance to the business operation and annoying everyone.
Go sit in separate corners, both of you. Think really, really hard about how both of your jobs are critical to the long-term success of the business. Don't come back until you've meaningfully internalized that.
Similarily, when "security best-practices" are leading to hundreds of my users losing SSO access to their BI system, or hundreds of printers rendered unusable because of security update that requires administrative permissions for reinstalling the same drivers that were perfectly working so far, I don't care for your security benefits. They suppose to defend against the thing that you are causing. You are already damaging the organization with thousands of working hours lost, and everyone is frustrated, as a bonus.
All that is diffrent is the promise of non-maliciousness. Which often does not hold up. Cause money is corrosive to those little centralized empires of "all-can-fail-but-me".
Security is diversity, as in having a non-centrally controllable ecosystem, that is not a mono-culture. Your updates are the danger, your urge for control is the forrest fire.
Linux is not secure because its updated often. As package maintainer take-overs have show- that is even a vector. Its secure, because its fragmented into a thousand small populations, which offer no real financially interesting attack vector for a large scale take over.
(The situation is admittedly getting better with Flatpak.)
No, I am apparently not. Microsoft, Apple, and others insist on making it difficult. At least I found out today that I can install unsigned Firefox extensions once I switch to a special "unbranded" build. I'm glad Mozilla, at least, still offers that.
Here's the thing: I disable a lot of the security stuff you're not supposed to disable, when I can. I use a Jailbroken iPhone. My Mac has SIP and Gatekeeper turned off. Windows Defender is turned off on my gaming PC, and I lower Microsoft's driver signing requirements to the greatest extent allowed. I also ran an unpatched day-1 build of Windows 10 for around four years, with the autoupdate system forcibly neutered. (I now run LTSB, instead.)
I have never been bitten by a virus, ever†. I don't know if that's because of all the security measures I'm not able to turn off or because I've been lucky or something else. I suspect it's because I don't run dodgy software. Or maybe my life is a lie and all my devices have been infected for the past decade, and I never noticed.
In the meantime, I'm not seeing the upside to software forcing hardened security.
---
P.S. While I share their frustrations, I don't endorse the GP's attitude. I know that a lot of people really are doing difficult work with the best of intentions.
† Except for a handful of times when I was testing suspicious software in a disposable VM. That doesn't count for obvious reasons.
Not running dodgy software is indeed a very effective way to not get viruses, but that doesn't mean you shouldn't take more security precautions if they are available. What if, for example, malware exploits a zero-day in your browser and successfully installs itself without any interaction from the user? Windows Defender and related antivirus could detect malicious activity from such malware and remove it on windows and Gatekeeper/SIP/driver signing and related systems could severely limit its impact.
Mozilla probably introduced extension signing to prevent less technically inclined users from having adware installed into their browser. X.509 introduces certificate expiration, and X.509 is the most widely used mechanism for signing things. The only reason you have this problem is because the folks at Mozilla forgot to renew that intermediate certificate. While I agree that it should be possible for users to disable extension signing, as users should be able to do whatever they want on their own system, you shouldn't blame them for forgetting to renew a certificate associated with an additional security measure built to protect users from malware.
Well, we agree on everything, in that case. :) I have no problem with sensible defaults that can be adjusted, nor do I take great issue with Mozilla's specific mistake in letting the certs expire.
Edit: Also, just noting that I only lessen security measures when I have a reason, not because I'm rebelling against security practices or something. I use a Jailbroken iPhone because I run Jailbroken software, and I disable driver signing because I use unsigned drivers. It's not that I don't see the risks, so much as I'm very unconvinced the risks are worth the downsides, for me.
Malware that does obvious bad things like cripple the machine or encrypt the drive is minority.
But it's actually not funny, it's dangerous. This behavior endangers others.
Likely your computers have been a zombie in a botnet for many years - I'm always wondering how these large botnets can exist at such scale.
This is why one day we all will not be allowed to use computers without a "driving license".
Today, more than half of my open tabs disappeared in an instant, and were not even an option to re-open until either I waited around ("up to six hours...") or manually installed the workaround. All of my in-progress work in any of those tabs? Gone.
That absolutely qualifies as crippled usability. The mere fact of such a thing being possible is a usability defect. On what basis do I trust that my work is not going to disappear on me like that again?
https://bugs.chromium.org/p/chromium/issues/detail?id=952287
Chrome has bugs too.
Firefox will continue to have bugs. All software will continue to have bugs. I'm so sorry that you lost some tabs in your browser but shit really does happen and acting like this is some violation due to overzealous security controls is inane.
If you think this clownshoery hasn't cost Firefox any trust, then you're being as naïve as you accuse me of being "absurdly overdramatic" and "inane".
Bugs are a thing, totally conceded. Sloppy certificate management is, too, but it's an entirely other class of thing. Deliberately conflating them is at least as disingenuous a debating tactic as pointing at Chrome, which is utterly irrelevant to this incident. That's straight-up "whataboutism".
Full stop, this was foreseeable. This was preventable.
EDIT: Phrasing.
EDIT 2: I won't respond further to the same kind of tone.
That said, the part I was referring to is:
> The mere fact of such a thing being possible is a usability defect. On what basis do I trust that my work is not going to disappear on me like that again?
The possibility of a bug happening is hardly a usability defect in my mind. Or if you want to call it one, it seems like a perfectly reasonable one - this was a defense born out of necessity when malicious extensions were more of a problem.
And I think that the "On what basis" question definitely implies a total lack of trust, but sure, maybe not. The basis is that this is a single instance of a failure over the course of the features' lifetime, for a feature that has existed for absolutely ages.
I pointed to Chrome as an example of similar issues cropping up across codebases to show that these sorts of bugs do happen. I don't consider that whataboutism.
All bugs are foreseeable and preventable. Systems are complex. I think you're putting the issue in a very unfair light, even though it's very reasonable to be upset about time and effort that is lost because of the issue.
I understand your perspective, and appreciate your recognition of my own. That said, if you think I'm putting the situation in an unfair light, I think you're downplaying it at least as much.
In my eyes, this is no mere "bug"; it's an abject process failure. As a reply to another of my comments in this discussion suggests, this is more on the level of, "Oops, we forgot to renew our domain name...", than it is, "Gosh, we didn't validate the pointer returned by the frobnitz function, when the whoozle isn't initialized yet..."
Dealing with expiring certificates before they expire is covered in like the second week of Certificate Management 101, as it were. If it's necessary to stick an intermediate cert in there, then it's doubly so to keep it current.
> The basis is that this is a single instance of a failure over the course of the features' lifetime, for a feature that has existed for absolutely ages.
The plural of "anecdote" isn't "data", but an existence proof is an existence proof. That the problem has gone from zero occurrences to one, no matter over what period, literally makes it infinitely more likely to recur, if you want to be that reductive...
> Why not just post a link to the fix that can be installed WITHOUT enabling Studies? This sounds like a clever plan to get more people to share their data via Studies…
I definitely don't agree with that guy, and I doubt that's a majority opinion, but asking people to use a workaround that benefits them (Mozilla) after they broke things for a lot of people is bad publicity for sure. For what it's worth I think Mozilla is doing the right thing here, just it's not going to make them look great.
CAs can delete certificates from their revocation lists after expiration, which means that you can't tell the difference between a certificate that was never revoked but merely expired and a revoked-and-then-expired certificate.
To me adding a new plugin signing cert through a side loaded plugin is pretty much the definition of exploited.
All this tells me is that their plugin signing solution is utterly worthless.
The only way this should have been fixed was through official update channels.
Then it's only a matter of time until you come back to Firefox, or maybe you'll end up making your own web browser?
I appreciate the condescension of both your comment and the person I initially replied to, but I honestly see your comments as saying, in more words, "Fuck the user." And that's fine, but why don't you just say it? Go ahead and type it, I want you to type what you really think about the users who are so dumb and fickle that they can't handle something so trivial as not being able to use their precious stupid add-ons like HTTPS Everywhere and uBlock.
This kind of toxicity is exactly why many of us have "uninstalled" Reddit and the like.
Please refrain from bringing that toxicity to HN.
“Toxicity.”
Goes both ways.
Your comment, my comment, that comment, none of them are substantive. Why isn't that comment removed yet?
I mean if you want me to be reflective it's really not going to work if we're refusing to admit that either both of these comments should have been censored or the one above should not have been censored. But I get it, life's not fair and the squeaky wheel gets the grease.
Firefox zealots get to railroad commenters on HN, noted, I certainly won't engage in these discussions again.
I appreciate the condescension of both your comment and the person I initially replied to, but I honestly see your comments as saying, in more words, "Fuck the user." And that's fine, but why don't you just say it? Go ahead and type it, I want you to type what you really think about the users who are so dumb and fickle that they can't handle something so trivial as not being able to use their precious stupid add-ons like HTTPS Everywhere and uBlock..
Flag it again, for the F word. Flag the comment I responded to as well, for consistency. :)
“Everyone who disagrees with me is an emotional idiot!”
“Hey I just uninstalled Firefox.”
“Flag him!”
I await your apology.
With a response time like that, I don't see how Lynx will satisfy their "5 minute fix" need any more than Firefox did.
[1] https://lynx.invisible-island.net/current/CHANGES
Read that, I don't know maybe ten times, and then read my comment which said I just switched my browser, no emotion involved.
I'm a man of action, baby. You can deliberate over my thought process all day, wondering if I would pick this browser or that one, or maybe even if I felt bad that I violated my loyalty to Mozilla Firefox (blessed be thy name). But don't think too hard, I've already moved on to the next decision in my life.
The bug in the first place, on the other hand, seems pretty negligent. Not that it's incomprehensible, just pretty stupid.
Anyhow, good luck with brave!
The BS coming from their blog post surrounding this whitelist makes me distrust them completely: "Loading a script from an edge-cache does not track a user without third-party cookies or equivalent browser-local storage" (...) "Given that most users on the web share IP addresses with other users because of NAT, it is unlikely this can be used to reliably track users"
Not only it's quite possible to know if the user is behind CGNAT or not, meaning the tracking works just fine for millions of users, but carriers have been known to inject user IDs in the replies of users behind CGNAT.
I still prefer Firefox over all the other browsers, and will continue to use it, but the project has lost a lot of trust and goodwill over this.
The optics are indeed awful, and this was fully preventable. Firefox fucked up, full stop.
Based on the timing of initial tweets and blog posts on this fiasco, I'm pretty sure I was in the first 10%, if not first 1%, of people who experienced this. And I was in a plane at 36,000 feet trying to work on a cross country (U.S.) flight when suddenly about 130 tabs in 7 windows disappeared. Really, REALLY bad. Panic, frustration, confusion...
I was more than 50% sure that all was not lost forever, that it was some "glitch" (Extensions all showed the same bloody red status), but I was tweaked. I work in security (embedded systems, not computers/IT) so I have a very good understanding of certificates, TLS, PKI, etc. There are many ways things can get out of whack if the people in charge screw up.
Regardless, this is embarrassing, dare I say shameful (pretty much almost up there with "Ooooppsss... we just lost our domain - it expired and no one thought to renew it)
Come on, guys, get it together. Have a procedure, document it, practice it, stay in front of it.
EDIT: after installing the fixed XPI, I have to sadly report that all data has gone. All my carefully-managed containerized life was wiped clean. Heads should roll.
Complete shambles. And the worst thing is, I suspect it's all a plot to have more people opt-in to the shitty telemetry. Otherwise, why not push an update through the usual channels? Had it been a security-related fix, would have they used "studies"? I bet not.
Given that it has happened, I expect them to provision a new certificate and push a fixed version within an hour or two to all release channels.
What I would emphatically ‘not’ expect, is a hack that might take up to 6 hours to be applied.
You can fix this temporarily via setting "xpinstall.signatures.required" to false. Toggle it back to true once update is released and you install it.
Meanwhile I'm hijacking this comment that is to the upper parts of the tree to state this: the way the community treats Mozilla and Firefox is horribly, inexplicably, unacceptably unfair.
This is nothing compared to innumerable other fuckups in software history, and even recent ones like goto fail, heartbleed, or Chrome logging you into Sync w/o notice.
This is a mistake, an easily recoverable one, and is not intentional or malicious. Firefox is developed in the out and open, all the processes are public. And people, with an absurd entitlement and malice, go as far as to call things backdoors or malware. Meanwhile the alternative actually is a backdoor ridden malware.
Please don't be this ungrateful.
While I agree with most of your comment, you're downplaying the severity here, especially since, IIUC, this situation also affected the Tor browser, disabling NoScript. If regimes like China were on the ball, and succeeded in escalating the remote code execution vulnerability into into deanonimization, this debacle may end up having a death toll attached to it.
Most Firefox users have that checkbox enabled by default, and so most Firefox users received the fix within 0-6 hours of the blog post's publication.
HN readers often take special care to prevent Mozilla from updating Firefox, but that in no way represents the wider population of either all addons users or all Firefox users.
No, it's not. "Studies" is not a security-related mechanism and it didn't exist in the past, when fixes were rolled out very quickly anyway for security reasons. "Studies" should not be relied on to be a fix-delivery mechanism, because it just isn't.
This is not even about privacy, it's simple good engineering sense.
I use FF because I care about principles. Otherwise I might as well just let myself be exploited by Google, MS, Apple and friends.
https://news.ycombinator.com/item?id=19825830
(While of course you’re welcome to continue pursuing your issues with it here, your opinions will receive more views there.)
Presumably the testing burden for a preference update using Normandy is smaller, as (and I’m guessing wildly here) fewer things can be altered with Normandy and therefore testing can be simplified to exclude, for made-up example, “the code-signed binary can be executed on all platforms”.
Go to about:debugging from the address bar. Right at the top is a button to "Load Temporary Add-on", with a checkbox "Enable add-on debugging". (On a Mac, the add-ons are in ~/Library/Application Support/Firefox/Profiles/«ID».default/extensions (assuming that you have only a single profile).) They should stay enabled until Firefox relaunches.
You'd be looking for a file C:\Users\YOUR_USER_NAME\AppData\Roaming\Mozilla\Firefox\Profiles\YOUR_PROFILE\containers.json
and also ...\YOUR_PROFILE\browser-extension-data\@testpilot-containers
Also...my default search engine is now Amazon.com?? WTF is going on.
EDIT: Also my only search engine. Heck of a job Mozilla.
Also I found another dark theme that is somewhat similar to what I had (but not official Mozilla) and installed that. All the default ones are still disabled.