Seems like a convenient excuse to walk back WFH. Surely the vast majority of Facebook employees would have been as helpless as Facebook's users to fix this outage.
What's preventing Facebook from continuing to offer WFH to many/most employees, with a minority of critical staff required to be on-site and compensated for it if necessary?
Take it with a pinch of salt as it's from the daily mail, but
"Some reports claimed company keycards were also knocked offline, meaning employees were unable to gain access to the offices or the server rooms - with some claiming employees were forced to, quite literally, break in."
Well that would work well for people working from an office.
Most places I've been have been badge in - badge out. You can still prop the door open, but first you have to find that hidden fire emergency override button.
Seem like a massive overreaction. Keep the sys admins on-site to manage your on prem gear but they should have been doing that anyway?! Surely when you're running your own servers for anything of any scale you need actual people nearby 24/7. Am I missing something?
You're not. The process that allowed this to occur was broken, and critical changes should only occur when you've confirmed someone is on site and available in the event you need a crash cart hooked up to a piece of gear (or perhaps automated rollbacks in the event of an unexpected state I suppose is the new hotness with software defined everything).
If you're making a change where you're unsure if someone can perform a physical recovery, you're flying without a chute. It works most of the time until it doesn't.
(infra/physical colocation/hosting/sysadmin in a previous life for ~20 years)
EDIT: @mprovost (HN is throttling me, so I have to reply to your comment here):
Absolutely. It's a combination of experience, context, and wisdom. I myself have made changes that I believed were trivial but, because I lacked context, data loss or an outage occurred. These are painful lessons to learn. A culture of psychological safety and commitment to continual process improvement are key, with shared responsibility between contributors and the org. You are going to fail, but you shouldn't get cold sweats when you do as long as you did your best to derisk.
The hard part is determining with 100% accuracy what is a "critical change". Many of these incidents start with someone making a change that "should be" safe. So you either have to make all your changes with a crash cart nearby, or trust in your ability to recover from a bad change remotely.
A computer on a movable cart that is correctly provisioned to get direct access to a server in a rack in a datacenter. Nowadays just an appropriate laptop on a desk with casters.
A crash cart used to be a literal cart in the datacenter with a monitor and keyboard to plug into a rack-mounted server to recover access. Not sure if there is something more modern now.
Still is that where I work. Doesn't help if you need serial console access, for that you'll need a laptop and a USB/serial dongle, or an actual old school serial terminal such as a VT100, VT220, etc.
In a previous life, I and a biz partner ran a web company on about 5 servers. We cabled the serial ports together in a daisy chain, used serial port consoles, and had software running on every system to access the ports. This way, if a system died, you just had to ssh into the system connected to it and you had access to the failed system's console. These were Supermicro servers, so the pre-boot screens and all that were also available. It came in very handy and saved us lots of trips to Hurricane Electric.
Remember that a huge part of this fall out was that FB staff couldn't talk to each other in a moment of crisis. Not just the people trying to manage the crisis, but the entire company faced a work stoppage in one way or another.
If you do the cost benefit of allowing people to work from home vs the cost of this one outage, I bet the outage is more expensive.
Surely they need easy comms fallback options that are independent of their own tech / infra. Even if they were on site this would still be pretty terrible to have everything so coupled right?
I monitored the Facebook amateur radio club’s repeater during the outage to see if they utilized it in any capacity like that. I didn’t hear it used that way.
Yes. Every time I've ever participated in an oncall rotation with 24/7 coverage, we absolutely did have one another's phone numbers, whether for work-issue phones or - as in my own case, since I never have wanted to bother with a second phone - personal ones. Sure, not everyone had everyone else's number, but among all of us we had enough coverage to reach anyone who might be needed to help respond to a sufficiently severe incident.
The point was to make sure that, even if all of our infra including third-party IP comms and chat providers was hard down, we'd still be able to talk to one another and figure out what to do about it.
For an organization on Facebook's scale to overlook something so fundamental implies lots of things about the collective blind spots of its engineering and operational culture. None of those things make me less glad I so assiduously avoid even using any of its products, to say nothing of relying on them - at least to the extent that I'm able to avoid it, given their apparent ability to break half the anglophone Internet more or less at whim.
Facebook issues a company iPhone with AT&T service to every employee. If they couldn’t communicate with each other it’s because the phone book is not set up well to handle this kind of event.
That’s a good point; however, I do wonder if an outage like this could trigger an emergency use exemption. There was a point when this was unfolding where it seemed at least plausible it was an attack on a robust communication platform.
Mostly I was hoping to hear some amateur enthusiasm gloating about invisible light resilience.
It is, and it would have been illegal for any Facebook employee with a ham radio license to use ham radio to contact another employee for any business reason during the outage. We don't have any evidence that that happened.
Where I live there is a General Motors amateur radio club. The club is not owned or run by GM, but GM sponsors it by allowing repeater equipment on top of their building, and presumably monetary and other donations over the decade. Many of the members are current and former GM employees but you don't have to be affiliated with GM in any way (or even have an amateur radio license) in order to join.
There are lots of amateur radio clubs associated with a company for sponsorship/community outreach reasons, I assume the Facebook one is the same. (And I have to assume because they don't seem to have a website that I could find.)
To reply to another comment in this thread, you can break nearly all FCC rules if you need help in a life-threatening situation. Facebook being down was NOT a life-threatening event as defined by the FCC, even if their services being unavailable indirectly lead to loss of life.
> ... communications directly related to the immediate safety of human life or the protection of property may be provided by amateur stations to broadcasters for dissemination to the public where no other means of communication is reasonably available before or at the time of the event.
Somehow Facebook have a SOC2 (wonder who their auditor is and how much they paid them?)
Part of a SOC2 audit is demonstrating that you have a good BCP, and part of a good BCP is demonstrating that you have fallback communication mechanisms available. So, either their BCP was out of date, or possibly they just ignored it during the moment of crisis.
> Surely they need easy comms fallback options that are independent of their own tech / infra.
And you know what's excellent for that ... literal word of mouth. I'm back in the office with 5 of my coworkers right now, and when we have issues we can just talk about them, zero technology needed.
Trut, that helps. However, even medium sized companies are often spread across multiple cities and timezones so it only helps to an extent. Having a single point of failure for literally everything seems to be a pretty large elephant in the room here.
Yup, it's not really about remote hands in datacenters (which of course have been staffed continuously), it's about losing 'run over to their desk' and 'shout across the office' as a viable communication channels in a crisis when literally all others have failed.
It means that for certain functions you need people on site, so someone can physically get to the server or router that is causing the problem. It doesn't mean you suddenly need everyone on site.
It may also mean that you need more decentralization. They put all their eggs in one basket, so they took down Instagram, Messenger, and WhatsApp as well as Facebook, as well as their internal network, their building security, everything. Seems that they need to look at redesign to create more resiliency even if some efficiency is lost, so they don't lose everything again at some point.
Is this what happens when product engineering runs the company? In all of my previous jobs, the production customer-facing systems have been separated from the internal day-to-day IT systems. Beyond potentially sharing a physical building/dc, IT and engineering could in no way interfere with each other.
If your company dogfoods their own products then this can happen.
I would expect the value of dogfooding to be massively larger than the cost of internal losses in cases like this. But it's good to remember to have a plan B.
I'm not sure I agree. It's one thing to treat IT as a customer for a product or service your company offers. I don't doubt that the business also uses FB messenger to communicate for example. But FB as a consumer product doesn't offer DNS or SAML or any of these other critical services, so why would IT share infrastructure for those? That's not dogfooding.
People need to talk to each other to quickly debug and understand what the problem even is.
To understand that the fix is a change to equipment in a data center, and on top of that what that change needs to be, how to make that change safely (though in this situation it's hard to imagine making things worse).
The claim that this was related to the recent outage doesn’t match all of the pre-outage articles that also point to the January 2022 date.
Unless someone has evidence that this is a recent change, I think this is a case of heavy editorialization by a journalist eager to capitalize on the combined anger toward Facebook and return-to-office plans.
Nothing in your article contradicts the OP -- a RTO for non-remote workers being delayed doesn't mean that all workers had to RTO. (But now it seems like FB may have changed that policy.)
The OP seems to only source its claims to the Daily Mail[0], which itself does not contain any new claims except for this:
> The need to request formal permission also applies to employees offered pay cuts in return for staying at home in cases where the local cost of living is lower than around the office where they are usually based.
To start, this claim is unsourced.
I'm having a hard time interpreting this claim, but it seems like _maybe_ it's saying that FB is requesting individuals who were already approved for remote work to RTO. But it also seems like they're just describing the existing process, which is that if you want to remain remote beyond January 2022, you must be explicitly approved for remote work. Anyway, I can't speak for everyone, but as a remote approved FB employee, I've received no such request to RTO.
This seems like a very large change as a response to the outage. What about employees that have already migrated to different locations? They are expected to move back within 3 months with the housing market like it is right now?
The subset of people that could fix the issue that caused the outage was probably .1% of the engineers at FB.
Why would you migrate to a different location unless you have a contract that state that you have the right to work from anywhere (or laying out the conditions, such as maximum time zone differences and such)?
Anything that is simply a current policy at a company basically has to be ignored. It's a nice to have but if you "can't live without that", go look for another job or adjust otherwise.
And even stuff that _is_ in your contract is not a full on guarantee either as contracts can be amended. Which you still have to sign but they will make it very hard for you not to sign those amendments. But at least it gives you some warning when that happens. I.e. you can refuse to sign and then you have time to look for another job while they figure out how to get you fired.
"If you owe the bank $100, that's your problem. If you owe the bank $100 million, that's the bank's problem."
Technically you're correct. If only a handful of employees migrated, they could probably be let go pretty easily. No skin off Facebook's back.
Now if hundreds or thousands of employees migrated, Facebook needs to make a decision whether enforcing the RTO or not will hurt them more. Regardless of technicalities, demanding a non-significant percentage of your workforce to uproot their lives is not an easy ask (or great look).
Or you could just say "I'm looking at my future, and I'm thinking I could grow more by joining a different team." And then just drop "working from home" as something that impacts your quality of life, and you're good. This is a sellers market.
Is your point "Never trust WFH contracts/policies because they will be amended!"?
The article may be misleading, but if employees who were told they could WFH forever now are told to come back that is a change of policy that WILL affect employees. I don't care if you are saying that no one should have moved, there will be people that left. Hell, I work for a much smaller company that didn't state WFH forever (WFH until we tell you to come back, no idea when that will be) and we have had hundreds of people relocate that have stated they will deal with that problem when they ask us to come back.
Pretty much, except strike the "WFH" from that sentence. I.e. never trust that a nice policy that company has today is going to be there tomorrow. They can change it overnight.
Of course you can still move away (or do whatever else depending on what other type of policy we're talking about) but you have to be prepared to deal with the consequences.
E.g. if you work for FAANG and you base your life off of a FAANG type salary but now you move to a flyover state, the option of saying "well if you want me in the office, I quite!" is suddenly way less attractive. Unless you were prepared for that. Personally I wouldn't be but I also don't want to work for FAANG in the first place.
You have a relatively low base salary but they wave a bonus of 30% in front of you? Great! Just don't go buy a house with that bonus money (or be ready to deal with the consequences). Some of my dad's colleagues did that in the 80s. They lost their houses when the bonus was reduced/not paid any longer.
Sure, you can have this attitude about how you operate.
Many of us believe there are expectations of companies, contracts whether written or social that should be followed. Even if Facebook reverts their position on this to "back in office" we have the right to complain and/or leave the company (while lamenting that this significantly affects people's lives).
We might have a slight misunderstanding. I'm not saying you do not have a right to complain about them doing that. You absolutely do.
I'm just saying that you shouldn't have too many expectations of companies. If they really meant it, they'd put it in writing. Employment contracts in many many places are written to favour the employer in all aspects.
Another example we could use are the notice periods. You will be hard pressed to find a North American company where the notice period in the contract is longer than 2 weeks, if even mentioned explicitly at all. Never mind states that have no minimums by law. Yet there seems to be a weird "social contract" that expects employees to give a longer notice period in many cases. If you want me to give you ample notice, give me a contract that requires the same of you.
A CTO of a startup I joined said "we are moving fast and break things". Sounded great! But after the first outage they were looking for someone to blame and quickly revoking production access)))
Yeah, I think a lot of managers and executives want the social credit of "we're totally open to experimentation and trying new things!" because that makes them sound cool. I don't think they're "lying" since that requires intent, but I think a lot of the time that's also not a 100% honest sentiment either. They are nominally open to experimentation...so long as you know the experiment is going to work.
Not going to say the company name (though you might be able to figure it out based on my post history, I ask that you don't paste it here if you figure it out), but I was working for a bigass megacorporation where management told me upon being hired "we don't play the blame game here, things are going to break". That sounded cool, and about three months later, I committed some "risky" code (which passed the PR review process evidently), and it broke production for about two hours. I then got VoIP call with that same manager who yelled at me telling me that I need to be more careful because this is unacceptable.
Presumably this manager was frustrated and just needed someone to blame, so fair enough, but it's not something I ever really forgave if I'm being honest.
Yeah, that's my point; it's easy to say you want to "move quick and break shit", people think you're cool as you say it, but it's not really honest. Managers (somewhat understandably) do not want uncertainty, and experimentation (almost by definition) leads to uncertainty.
When I was at Amazon in 2001 outages were done via everyone sitting at their desks on a POTS conference call. The engineers working the issue would be spread out around building at least, and the people who were actually on-call would generally be at home. Remote hands were often in a datacenter that was on the other side of continent.
The whole idea of piling everyone into a "war room" doesn't really scale anyway.
The bigger problem here was just no out of band comms.
Word-of-mouth I've heard is that the backup IRC server was off-site working fine. The problem was that, because it was only used in emergencies and wasn't hooked into FB's central authentication, a lot of employees simply didn't remember how to log in to it.
This headline is misleading at best, and outright false at worst.
It's technically correct, but the article doesn't actually say that the the end of the WFH forever is _because_ of the outage, just that they are going back.
Isn't this another example of "how does a company with 1000s of the most able employees not notice a massive risk to business continuity sitting there waiting to happen"?
DNS hosted on your own registrar, within your own data center for a whole host of domains. Somehow integrating your office security so that it also depends on that same DNS and running BGP updates without an emergency "undo" plan?
I hate to over-simplify but Single point of failure is the first thing you ever learn about Business Continuity. We shouldn't need to learn these lessons from a big mistake, that's what college is for.
Totally agree. A lot of time, money and effort can be spent to make everything redundant and it's one single point of failure you overlooked that will break the camel's back.
The point I think that we have to keep in mind here is that it seems that Facebook overlooked many many points of failure and was very sure of their own ability at doing everything and doing it perfectly all the time.
Anecdote from my previous life: Large corporation you would all recognize and which I won't name for obvious reasons. They ran their own data center(s) and it wasn't a side hustle to do that. One morning I come to the office and I have lots of messages about a massive outage. One of the data centers is offline. Like completely offline. I'm a bit fuzzy on some details because it's been a while but the main story went something like:
External power goes out for whatever reason (don't remember)
Batteries take over
Diesel generators start up
Batteries keep draining
Diesels are spinning
All Servers shut down
What happened to this massively redundant and well planned out data center? Nobody knew and again, fuzzy on the details but the report that came out a few days later was that somewhere in that system, was a non-redudant breaker/power line combo somewhere. And that breaker went flying for some reason.
This is why Netflix has a policy of routinely breaking things on purpose. If all your other practices are pretty decent, it's easy to build up false confidence based on all the disasters that you haven't actually prevented, just made unlikely.
This was not a DNS failure, nor was it caused by a single point of failure. Their entire backbone was taken offline due to a bad configuration change that went out to all of their routers.
Depends on how you define a single point. Single point doesn't mean 1 cause, it means at some point in the chain, there is a non-redundant component that when it fails the system stops. The fact FB could remove 100% of their capacity from the backbone is a SPoF if they don't have a redundant backbone completely separated from the configuration system used to control the primary.
I won't pretend to know the details of FB's infrastructure, and I'm sure there was a cascade of problems that made the outage harder to fix, but it sounds like years of separate teams building on top of other layers that were "guaranteed" to be online finally collapsed into a spectacular fireball.
How is a bad configuration change that took out their entire network NOT a single point of failure?
It's not like there were twenty different network engineers that typed the exact same command across their network. Ultimately it was one person who typed "enter" and submitted the bad change. And the problem wasn't even necessarily the change -- mistakes should be expected. The problem was the lack of a back-out strategy in case this happened.
Nobody expects a BGP issue. It's main weapon is surprise.
But, more seriously, nobody plans for BGP failures, BGP is a single system, with a distributed database across the internet, so there's no redundancy for it, and to add it up, it's nearly completely invisible, most people don't even know it's there.
I plan for a routing failure over my network, key points have a separate simpler OOB network on an independent internet handoff, with local hosts files and local username/passwords if tacacs is down.
You can say this about pretty much any outage or mistake ever. Every single security vulnerability or jailbreak too. How could the entire computing industry with 10000s of amazing people have missed Spectre/Meltdown? In 2018, Spectre affected basically every computer system ever - desktops & mobiles, AMD/Intel/ARM and it was basically open to exploitation since speculative execution which is decades. 1000s of employees is not as big as it sounds given how much FB does and hindsight is 20/20..
A test positivity rate below 2.5%, where contact tracing definitively works.
According to the April 16, 2020 3 Phase Reopening Plan for the United States, each state should follow, "14 days of declining cases[/positivity rate] per phase and hospital capacity that exists in case you have a rebound" were the criteria for each phase; The 8 worst states for Delta variant have declining positivity rates starting about 3 weeks ago.
The virus is never going away and will spread epidemically again in the future.
At some point the population has enough T-cells though that the infection fatality rate and infection hospitalization rate is that of influenza or colds, and even though it spreads people don't wind up in the hospital that much more than usual for cold and flu season.
And we're still probably going to have vaccine stragglers for years winding up finally getting their Herman Cain Awards at some level so it'll never be zero.
Before I even read this thread it was pretty obvious this meant they never planned on offering WFH forever.
If they wanted some on site employees to handle these kind of outages, they could just pay their top 1% to come to office or move within 15 minutes of the office or something.
That's my thought as well. These corporations get a ton of free publicity for being "the cool companies that are going to make remote permanent." Then all they have to do to more or less save face is wait for something bad to happen, show some vague graphs about how this could have been prevented if we weren't remote, and that they are forced to implement a RTO order.
Everyone I talk to in tech has a RTO date of January 2022. However they also made fully and partially remote an official offering (I say official because of all the tech companies I’ve worked for it was never out of the question, just arranged and not offered). That’s fine, but most of my coworkers are now either partial or fully remote (to the point where we have almost no existing NYC team that would come into the office). So what does this January 2022 RTO really mean for any company?
The firm I currently work for (in the 4 digits employee size) has pushed theirs back to Mar-2022.
Many recruiter emails I have received are pushing "100% remote" - for how long I can't say as I don't answer them, but it feels like RTO is becoming more in the minority.
I had a RTO of August 2021. Mandated vaccine in an exchange for no masks (among other things, such as getting to remain employed). Well they reneged on the no mask rule. I can't undo my vaccine, so I'm working from home until they can provide a safe work environment.
Good! Usual digital nomad spots are getting too crowded. I welcomed covid initially, it normalised remote work. But it drives cost of nice places too high.
Does this mean that people who were approved for remote work will be forced to come back into the office? Or that people who are supposed to come into the office Jan 2022 will continue to have to come into the office in Jan 2022 (i.e., no change).
This is probably somewhat insensitive but given everything recently, if this leads some workers from quitting facebook, it probably will be good for them (the workers that is), or at the very least, a silver lining is easy to see. I don't think or know if fb will fall or decay (or ever will) but after watching their company get so many beatings and if those people are lucky enough to have options, it's not hard to see people wanting to get out. And good for them.
I've been wondering how these huge companies with town-sized campuses, and all of that capital investment would respond. I guess it would take a really special kind of company to recognize that these campuses are a sunk cost and pivot to true remote.
127 comments
[ 0.31 ms ] story [ 303 ms ] threadWhat's preventing Facebook from continuing to offer WFH to many/most employees, with a minority of critical staff required to be on-site and compensated for it if necessary?
"Some reports claimed company keycards were also knocked offline, meaning employees were unable to gain access to the offices or the server rooms - with some claiming employees were forced to, quite literally, break in."
Well that would work well for people working from an office.
If you're making a change where you're unsure if someone can perform a physical recovery, you're flying without a chute. It works most of the time until it doesn't.
(infra/physical colocation/hosting/sysadmin in a previous life for ~20 years)
EDIT: @mprovost (HN is throttling me, so I have to reply to your comment here):
Absolutely. It's a combination of experience, context, and wisdom. I myself have made changes that I believed were trivial but, because I lacked context, data loss or an outage occurred. These are painful lessons to learn. A culture of psychological safety and commitment to continual process improvement are key, with shared responsibility between contributors and the org. You are going to fail, but you shouldn't get cold sweats when you do as long as you did your best to derisk.
https://en.wikipedia.org/wiki/Crash_cart
If you do the cost benefit of allowing people to work from home vs the cost of this one outage, I bet the outage is more expensive.
I mean if it happens naturally (say, meeting up after work for a drink) then sure but I don’t do it as a matter of course.
That said, I don't know what's so wrong with telephones.
Do you have all your coworkers' personal phone numbers?
My manager has my phone number, as does HR, and I imagine this is a standard situation at nearly every company.
The point was to make sure that, even if all of our infra including third-party IP comms and chat providers was hard down, we'd still be able to talk to one another and figure out what to do about it.
For an organization on Facebook's scale to overlook something so fundamental implies lots of things about the collective blind spots of its engineering and operational culture. None of those things make me less glad I so assiduously avoid even using any of its products, to say nothing of relying on them - at least to the extent that I'm able to avoid it, given their apparent ability to break half the anglophone Internet more or less at whim.
Mostly I was hoping to hear some amateur enthusiasm gloating about invisible light resilience.
Where I live there is a General Motors amateur radio club. The club is not owned or run by GM, but GM sponsors it by allowing repeater equipment on top of their building, and presumably monetary and other donations over the decade. Many of the members are current and former GM employees but you don't have to be affiliated with GM in any way (or even have an amateur radio license) in order to join.
There are lots of amateur radio clubs associated with a company for sponsorship/community outreach reasons, I assume the Facebook one is the same. (And I have to assume because they don't seem to have a website that I could find.)
To reply to another comment in this thread, you can break nearly all FCC rules if you need help in a life-threatening situation. Facebook being down was NOT a life-threatening event as defined by the FCC, even if their services being unavailable indirectly lead to loss of life.
https://www.ecfr.gov/cgi-bin/text-idx?SID=1a361a6eb3d1594e6a...
Part of a SOC2 audit is demonstrating that you have a good BCP, and part of a good BCP is demonstrating that you have fallback communication mechanisms available. So, either their BCP was out of date, or possibly they just ignored it during the moment of crisis.
And you know what's excellent for that ... literal word of mouth. I'm back in the office with 5 of my coworkers right now, and when we have issues we can just talk about them, zero technology needed.
It may also mean that you need more decentralization. They put all their eggs in one basket, so they took down Instagram, Messenger, and WhatsApp as well as Facebook, as well as their internal network, their building security, everything. Seems that they need to look at redesign to create more resiliency even if some efficiency is lost, so they don't lose everything again at some point.
I would expect the value of dogfooding to be massively larger than the cost of internal losses in cases like this. But it's good to remember to have a plan B.
To understand that the fix is a change to equipment in a data center, and on top of that what that change needs to be, how to make that change safely (though in this situation it's hard to imagine making things worse).
Everything else I’m reading says that January 2022 was always the planned return-to-office date ( https://www.cnbc.com/2021/08/12/facebook-delays-return-to-of... )
The claim that this was related to the recent outage doesn’t match all of the pre-outage articles that also point to the January 2022 date.
Unless someone has evidence that this is a recent change, I think this is a case of heavy editorialization by a journalist eager to capitalize on the combined anger toward Facebook and return-to-office plans.
https://www.cnbc.com/2021/08/12/facebook-delays-return-to-of...
https://www.cnbc.com/2021/06/09/facebook-will-let-all-employ...
> The need to request formal permission also applies to employees offered pay cuts in return for staying at home in cases where the local cost of living is lower than around the office where they are usually based.
To start, this claim is unsourced.
I'm having a hard time interpreting this claim, but it seems like _maybe_ it's saying that FB is requesting individuals who were already approved for remote work to RTO. But it also seems like they're just describing the existing process, which is that if you want to remain remote beyond January 2022, you must be explicitly approved for remote work. Anyway, I can't speak for everyone, but as a remote approved FB employee, I've received no such request to RTO.
[0] https://www.dailymail.co.uk/news/article-10064865/Facebook-g...
The same people who own interest in commercial real estate and companies like Starbucks.
No idea why they would be so concerned with pushing their agenda.
Then I'm more inclined to believe you. (Hence my initial wording "this policy may have changed.")
The subset of people that could fix the issue that caused the outage was probably .1% of the engineers at FB.
Was FB looking for an excuse to end those rules?
Anything that is simply a current policy at a company basically has to be ignored. It's a nice to have but if you "can't live without that", go look for another job or adjust otherwise.
And even stuff that _is_ in your contract is not a full on guarantee either as contracts can be amended. Which you still have to sign but they will make it very hard for you not to sign those amendments. But at least it gives you some warning when that happens. I.e. you can refuse to sign and then you have time to look for another job while they figure out how to get you fired.
Technically you're correct. If only a handful of employees migrated, they could probably be let go pretty easily. No skin off Facebook's back.
Now if hundreds or thousands of employees migrated, Facebook needs to make a decision whether enforcing the RTO or not will hurt them more. Regardless of technicalities, demanding a non-significant percentage of your workforce to uproot their lives is not an easy ask (or great look).
The article may be misleading, but if employees who were told they could WFH forever now are told to come back that is a change of policy that WILL affect employees. I don't care if you are saying that no one should have moved, there will be people that left. Hell, I work for a much smaller company that didn't state WFH forever (WFH until we tell you to come back, no idea when that will be) and we have had hundreds of people relocate that have stated they will deal with that problem when they ask us to come back.
Of course you can still move away (or do whatever else depending on what other type of policy we're talking about) but you have to be prepared to deal with the consequences.
E.g. if you work for FAANG and you base your life off of a FAANG type salary but now you move to a flyover state, the option of saying "well if you want me in the office, I quite!" is suddenly way less attractive. Unless you were prepared for that. Personally I wouldn't be but I also don't want to work for FAANG in the first place.
You have a relatively low base salary but they wave a bonus of 30% in front of you? Great! Just don't go buy a house with that bonus money (or be ready to deal with the consequences). Some of my dad's colleagues did that in the 80s. They lost their houses when the bonus was reduced/not paid any longer.
Many of us believe there are expectations of companies, contracts whether written or social that should be followed. Even if Facebook reverts their position on this to "back in office" we have the right to complain and/or leave the company (while lamenting that this significantly affects people's lives).
I'm just saying that you shouldn't have too many expectations of companies. If they really meant it, they'd put it in writing. Employment contracts in many many places are written to favour the employer in all aspects.
Another example we could use are the notice periods. You will be hard pressed to find a North American company where the notice period in the contract is longer than 2 weeks, if even mentioned explicitly at all. Never mind states that have no minimums by law. Yet there seems to be a weird "social contract" that expects employees to give a longer notice period in many cases. If you want me to give you ample notice, give me a contract that requires the same of you.
It's obvious that the first mentality was a bit reckless and only optimized growth at all costs. At some point you have to pay for that debt.
https://www.wired.com/2014/04/zuckerberg-f8-interview/
Not going to say the company name (though you might be able to figure it out based on my post history, I ask that you don't paste it here if you figure it out), but I was working for a bigass megacorporation where management told me upon being hired "we don't play the blame game here, things are going to break". That sounded cool, and about three months later, I committed some "risky" code (which passed the PR review process evidently), and it broke production for about two hours. I then got VoIP call with that same manager who yelled at me telling me that I need to be more careful because this is unacceptable.
Presumably this manager was frustrated and just needed someone to blame, so fair enough, but it's not something I ever really forgave if I'm being honest.
Of course announcing a policy change now is closing barn doors behind horses.
The whole idea of piling everyone into a "war room" doesn't really scale anyway.
The bigger problem here was just no out of band comms.
It's technically correct, but the article doesn't actually say that the the end of the WFH forever is _because_ of the outage, just that they are going back.
DNS hosted on your own registrar, within your own data center for a whole host of domains. Somehow integrating your office security so that it also depends on that same DNS and running BGP updates without an emergency "undo" plan?
I hate to over-simplify but Single point of failure is the first thing you ever learn about Business Continuity. We shouldn't need to learn these lessons from a big mistake, that's what college is for.
And even if someone thinks of the problem there's usually some kind of failsafe (an override key or angle grinder) in place.
The point I think that we have to keep in mind here is that it seems that Facebook overlooked many many points of failure and was very sure of their own ability at doing everything and doing it perfectly all the time.
Anecdote from my previous life: Large corporation you would all recognize and which I won't name for obvious reasons. They ran their own data center(s) and it wasn't a side hustle to do that. One morning I come to the office and I have lots of messages about a massive outage. One of the data centers is offline. Like completely offline. I'm a bit fuzzy on some details because it's been a while but the main story went something like:
What happened to this massively redundant and well planned out data center? Nobody knew and again, fuzzy on the details but the report that came out a few days later was that somewhere in that system, was a non-redudant breaker/power line combo somewhere. And that breaker went flying for some reason.https://netflix.github.io/chaosmonkey/
I won't pretend to know the details of FB's infrastructure, and I'm sure there was a cascade of problems that made the outage harder to fix, but it sounds like years of separate teams building on top of other layers that were "guaranteed" to be online finally collapsed into a spectacular fireball.
It's not like there were twenty different network engineers that typed the exact same command across their network. Ultimately it was one person who typed "enter" and submitted the bad change. And the problem wasn't even necessarily the change -- mistakes should be expected. The problem was the lack of a back-out strategy in case this happened.
But, more seriously, nobody plans for BGP failures, BGP is a single system, with a distributed database across the internet, so there's no redundancy for it, and to add it up, it's nearly completely invisible, most people don't even know it's there.
A test positivity rate below 2.5%, where contact tracing definitively works.
According to the April 16, 2020 3 Phase Reopening Plan for the United States, each state should follow, "14 days of declining cases[/positivity rate] per phase and hospital capacity that exists in case you have a rebound" were the criteria for each phase; The 8 worst states for Delta variant have declining positivity rates starting about 3 weeks ago.
The virus is never going away and will spread epidemically again in the future.
At some point the population has enough T-cells though that the infection fatality rate and infection hospitalization rate is that of influenza or colds, and even though it spreads people don't wind up in the hospital that much more than usual for cold and flu season.
And we're still probably going to have vaccine stragglers for years winding up finally getting their Herman Cain Awards at some level so it'll never be zero.
If they wanted some on site employees to handle these kind of outages, they could just pay their top 1% to come to office or move within 15 minutes of the office or something.
Many recruiter emails I have received are pushing "100% remote" - for how long I can't say as I don't answer them, but it feels like RTO is becoming more in the minority.
(context: https://mobile.twitter.com/sheeraf/status/144509915031650305... )
I’m no Facebook fan, but is there actual evidence of “Facebook pointing the finger at remote working”?
Does this mean that people who were approved for remote work will be forced to come back into the office? Or that people who are supposed to come into the office Jan 2022 will continue to have to come into the office in Jan 2022 (i.e., no change).