At that size of a company there's usually blast radius restrictions and per-role permissions. I don't expect anyone has enough rights to "delete Facebook" on their own.
I guarantee you that there are 100+ people who could take Facebook down for 24+ hours if they went rogue.
For example the people responsible for the bootup scripts of Facebook infra could sneak in a "0 0 1 * * /bin/rm -rf ${TEMPDIR}/*" into crontab... They'd set the commit message as "clear out temp monthly" and it would get deployed across the entire fleet till in the first of next month every disk at Facebook gets erased because TEMPDIR isn't defined...
I guess they have enough pending stock to deter them...
How though? Every past author of that script would be notified of such a change. It'd be insane if all of them would pretend they didn't see it and accept that change.
It would be hidden amongst a big refactor, and it would have a bunch of unit and integration tests that all work correctly (because they do set the environment variables correctly).
Writing code that appears to do one thing and actually does another is very doable - you only need to hide one malicious line amongst thousands in a code review, while the reviewer needs to inspect every line.
Besides, the vast majority of code reviews are 10 minutes or less.
I think you oversimplified the code merge process at Facebook. Their internal code review system (Phabricator) would automatically add tens of reviewers to your PR because you changed an important script authored by those people. Also, there are linters that would call out the use of potentially hazardous commands. Actually deploying the thing is another process that requires jumping through a series of other review hoops.
That’s...now how any of this works. You can’t just change integrity-bearing things without FIM systems kicking in. And you’d need collusion to get something mainlined that would bypass that.
This wouldn't "delete" Facebook or many much smaller companies. It would result in maybe a small outage and get restored immediately in most cases. It's also an infra change you'd need across many systems - this isn't possible as a single change "across entire fleet".
Anecdotally, whatsapp worked for me (message was sent and received successfully) though Facebook gave an error. I was already logged into both services.
Every company's actual status page (and news feed) is on Twitter, no matter what else they say. I don't know where Twitter hosts their real status page. Probably Facebook.
> every company's actual status page is on Twitter
Clearly you have not experienced the absolute joy of a mission-critical locked-in-vendor B2B SaaS with no status page, no active Twitter presence, and the effective status page being a chat widget that routes to a person overseas who says "oh yeah this is a known outage on our main product please stay tuned" but there's no attempt to proactively make it visible to clients that they know there is an outage.
EDIT: fun fact, said vendor lets us use a subdomain we own, so we just route them through a Cloudflare Worker that injects Sentry into their HTML, so that we can monitor errors ourselves and raise tickets with them pretending that we know less than we actually know, because somehow we have a better observability culture than a SaaS vendor that's been around for 20 years. Don't underestimate the difference between vendor.mybrand.com and mybrand.vendor.com on a feature matrix, it may save your sanity.
During the recent Microsoft Azure AD outage, I attempted to use the chat widget. Selecting "Technical support" or "Billing Support" both seemed to return 500 status codes while the "Sales Support" routed somewhere else and connected me to a person, perfectly fine. Presumably the sales team don't use Azure for anything.
Circa 2008-2009 when I was at FB I wrote a dashboard widget for the ops team that scraped Twitter for mentions of phrases like "Facebook down" in the past 5m. It was in use for a while.
most relationships that come to a bitter end are like this. you want to remember the good times, but the reasons you are breaking up always come front of mind.
I'm getting two different error pages with the same "Sorry, something went wrong. We're working on getting this fixed as soon as we can" message.
One has the older site look, copyright 2020, the other has the newer FB5 look, copyright 2021.
The logo is not loading on either page.
Part of me wonders if they discovered a big/new hole that led to the data leak, although Instagram being down too suggests maybe there's something happening behind the load balancers?
While that's perfectly true, I'm sure a smart and determined intern could cause an astonishing amount of damage in many many orgs. Hopefully not at Facebook though.
"And if these people stop, the world burns. Most people don't even know what sysadmins do, but trust me, if they all took a lunch break at the same time they wouldn't make it to the deli before you ran out of bullets protecting your canned goods from roving bands of mutants."[1]
5 is often broken down to “rain fade” and “backhoe fade.”
My favorite can’t-restart story is “the shared password manager is down, we need a hardware crypto key to restart it, the key is in the safe, and the safe’s combination was in the shared password manager.”
I heard a story that Google SRE had to up a sacrificial Bigtable in any new zone before they could up enough services to finally up a real Bigtable for the zone.
Coincidentally, I'm in the same predicament with Facebook, itself.
I can't log in to Facebook because Facebook wants approval from the Duo Mobile 2FA app. The 2FA app won't approve the login because it wants a code that can only be obtained by logging in to Facebook.
Solution: Deleted both Duo Mobile and Facebook apps. Moved on with my life.
isn't that like throwin the baby out with the bath water? i've never had problems with duo mobile, but then i don't use FB so i've never seen how it behaves with FB. so dump FB, but why dump duo mobile?
Large scale network partitioning can happen. Data center ingress/egress can go down. If you shard to multiple data centers, it's still possible for an entire autonomous system to have an interconnect with a different AS break. That is rare as hell with the actual Internet because of all the route redundancy even between ASes, but I've seen it happen on more specialized networks. For a few days last year, the entire western half of the US was cutoff from the eastern half on Raytheon's internal TS network because someone digging a hole somewhere accidentally cut the wrong fiber line.
I'll be curious about it coming back up. It's the sort of site where there's an army of people continually trying to reload the page if something isn't working.
In Denmark right now, everything down. Was on the phone with someone (in the US) through Messenger without issues. Hung up and tried to send them a message and nothing. Instagram non-functional as well.
207 comments
[ 3.2 ms ] story [ 52.6 ms ] threadPS. Instagram app/site, whatsapp.com, messenger.com, oculus.com also down, anything else?
UPDATE: Everything back up! (Even the status page)
For example the people responsible for the bootup scripts of Facebook infra could sneak in a "0 0 1 * * /bin/rm -rf ${TEMPDIR}/*" into crontab... They'd set the commit message as "clear out temp monthly" and it would get deployed across the entire fleet till in the first of next month every disk at Facebook gets erased because TEMPDIR isn't defined...
I guess they have enough pending stock to deter them...
Writing code that appears to do one thing and actually does another is very doable - you only need to hide one malicious line amongst thousands in a code review, while the reviewer needs to inspect every line.
Besides, the vast majority of code reviews are 10 minutes or less.
p.s. I had a short internship there.
This is not how non-trivial services work.
Notably, Facebook hasn't updated theirs, though.
Clearly you have not experienced the absolute joy of a mission-critical locked-in-vendor B2B SaaS with no status page, no active Twitter presence, and the effective status page being a chat widget that routes to a person overseas who says "oh yeah this is a known outage on our main product please stay tuned" but there's no attempt to proactively make it visible to clients that they know there is an outage.
EDIT: fun fact, said vendor lets us use a subdomain we own, so we just route them through a Cloudflare Worker that injects Sentry into their HTML, so that we can monitor errors ourselves and raise tickets with them pretending that we know less than we actually know, because somehow we have a better observability culture than a SaaS vendor that's been around for 20 years. Don't underestimate the difference between vendor.mybrand.com and mybrand.vendor.com on a feature matrix, it may save your sanity.
What are the odds it's some IRC bot running under someone's desk?
The year 2000 doesn’t sound right.
We're working on getting this fixed as soon as we can."
Hope it stays down, it would be an overall net gain for humanity.
Time to enjoy a brief social media pause, thanks to the deity who might have been behind this!
https://youtu.be/wZ6ZlRnS40s?t=12
Postmortems are exciting reads and they should have a section for postmortems in their engineering blog.
One has the older site look, copyright 2020, the other has the newer FB5 look, copyright 2021.
The logo is not loading on either page.
Part of me wonders if they discovered a big/new hole that led to the data leak, although Instagram being down too suggests maybe there's something happening behind the load balancers?
Ditto if they can bring it down intentionally and cover their tracks so no one finds out who did it.
1. https://www.stilldrinking.org/programming-sucks
bad DNS config push - 1 shot
routing loop - 2 shots
third party advertising your routes - 3 shots
power outage at data centre it turns out everything depends on despite decades and millions in engineering to avoid precisely that - 4 shots
Wolves ate through fiber - 5 shots
And it was a full moon - 6 shots
Single service failure, but service has not been restarted in 5 years, and no longer restarts in any documented fashion - 7 shots
And service developers left the company to found a startup - 8 shots
Expired internal SSL certificate - 9 shots
Daylight savings changeover-induced database corruption - 10 shots
Windows Update - 11 shots
I hope it's wolves, that would create a new category in https://cybersquirrel1.com/
Hunter shot your fiber relay with a hunting rifle - 6 shots
My favorite can’t-restart story is “the shared password manager is down, we need a hardware crypto key to restart it, the key is in the safe, and the safe’s combination was in the shared password manager.”
I can't log in to Facebook because Facebook wants approval from the Duo Mobile 2FA app. The 2FA app won't approve the login because it wants a code that can only be obtained by logging in to Facebook.
Solution: Deleted both Duo Mobile and Facebook apps. Moved on with my life.
Here it is anyway: https://news.ycombinator.com/item?id=22316230