In a world where it can take weeks for other companies to publish a postmortem after an outage (if they ever do), I never ceases to amaze me how quickly CF manage to get something like this out.
I think it's a testament to their Ops/Incident response teams and internal processes, it builds confidence in their ability to respond quickly when something does go wrong. Incredible work!
I feel like others lose opportunities by not doing the same. By publishing early and publishing the details they: keep the company in the news with positive stuff (free ad), get an internal documentation of the incident (ignoring the customer oriented "we're sorry" part), effectively get a free recruitment post (you're reading this because you're in tech and we do cool stuff, wink), release some internal architecture info that people will reference in discussions later. At a certain size it feels stupid not to post them publicly. I wonder how much those posts are calculated and how much organic/culture related.
> I wonder how much those posts are calculated and how much organic/culture related.
Don't companies have a fiduciary duty to calculate things; the reason for doing something actually cannot just be that it's a nice thing to do? Not down to the word, but at least the general decision to be this way?
No they don't have such duty. In practice very little decision making is based on hard data in my experience. Real world being fuzzy and risk being hard to quantify do not help the situation.
I agree that this is a free ad/recruitment. However, it’s easy to see how more conservative businesses see this as a risk. They are highlighting their deficiencies, letting their big important clients know that human error can bring their network down.
Additionally, these post-mittens work for Cloudflare because they have a great reputation and good uptime. If this were happening daily or weekly, it would be a warning sign to customers.
It’s a strategy other companies could adopt, but to do it effectively requires changes all across the organization.
OTOH, I think most actual engineers would know that everywhere has deficiencies and can be brought down by human error, and I'd personally rather use a product where the people running it admit this rather than just claim that their genius engineers made it 100% foolproof and nothing could ever possibly go wrong
1. On the buy side, crappy big companies with procurement etc. may not have “actual engineers” deciding things. Maybe for something like Cloudflare that’s likely to sit within an actual technical team’s mandate
2. Not 100% foolproof but if a startup is selling its tool and has an uptime page and details of all its downtime and as a prospective customer you go and see that they have 3-4 hour downtimes once or twice a month, it should raise alarm bells.
Ehhhh… I think it’s good (for us) that they do this, but I don’t think it’s a free ad (contrary to popular belief, not all news is good news, and this is bad news) and any sort of conversion rate on recruitment is probably vanishingly small (which would normally be fine, but incidents like these may turn off some actual customers, which is where actual revenue comes from).
I think their calculation (to the extent you can call it that) is that in the interest of PR and damage control, it’s better to get a thorough postmortem out quickly to stem the bleeding and keep people like us from going “I can’t wait to hear what happened at Cloudflare” for a week. Now we know, the customers have an explanation, and this bad news cycle has a higher chance of ending quickly.
> but incidents like these may turn off some actual customers
Incidents - yes. But why would a post-mortem turn someone off? The incident happened regardless. Do you think anyone would be more likely turned off by reading how they solved it / plan to prevent it on the future than by silence?
>I feel like others lose opportunities by not doing the same
IMO it is a slippery slope to see this as opportunity too strongly. Sure, doing the right thing may be net beneficial to the business in the long run...but the $RIGHT_THING should be done first and foremost because it's the right thing.
To further add to your point, the CTO is the one who shared it here & the CEO is incredibly active on forums & social media everywhere with customers. Communication has always been one of their strengths.
I do wonder what would happen should happen if either of them left the company, I feel like there's a lot of trust on HN (and other places) that's heavily attached to them as individuals and their track record of good communication.
Good communicators generally foster that environment. And their customers appreciate it, so there is an external expectation now too. Everything ends some day, but I think this will be regarded as a valuable attribute for awhile.
Devil's advocate, you could get taken over or end up with a different board. I wouldn't like to see it but someone's got to compete with you or we'll have to send in the FTC! :)
I agree, I think the transparency builds trust and I encourage it where I can. The counter thought I had when reading this case though, is it almost feels too fast. What I mean by that is I hope there isn't an incentive to wrap up the internal investigation quickly and write the blog and send it, and go we're done.
Doing incident response (both outage and security), the tactical fixes for a specific problem are usually pretty easy. We can fix a bug, or change this specific plan to avoid the problem. The search for conditions that allowed the incident to occur can be alot more time consuming, and most organizations I've worked for are happy to make a couple tactical changes and move on.
What I mean by that is I hope there isn't an incentive to wrap up the internal investigation quickly and write the blog and send it, and go we're done.
There is not. From here there's an ongoing process with a formal post-mortem, all sorts of tickets tracking work to prevent further reoccurrence. This post is just the beginning internally.
I have to agree. The environment that leads to a fast blog post may also lead to this quote from the post:
> This was delayed as network engineers walked over each other's changes, reverting the previous reverts, causing the problem to re-appear sporadically.
They are running as fast as they can and this extended the incident. There is a “slow is smooth, smooth is fast” lesson in here. I’d rather have a team that takes a day to put up the blog post, but doesn’t unnecessarily extend downtime because they are sprinting.
It can be easy to arm-chair it afterwards, but unless things can be done in parallel (and systems should be designed so this can be done, things like "we're not sure what's wrong, we're bringing up a new cluster on the last known good version even as we try to repair this one") you have to make a choice, and sometimes it won't be optimal.
There's normal operating procedure and sign offs and automation etc. etc. and then there's "we've lost contact with these data centers and normal procedures don't work we need to break glass and use the secondary channels". In that situation you are in an emergency without normal visibility.
At a previous job, I worked with two guys who were excellent in a crisis. One of them used to run operations, the other was a crusty old programmer who’d been around for a while. I tried to learn as much as I could from both of them.
At around the same time, I was watching HBO’s The Wire. One scene had a high-profile shooting with a frantic police response; people were running everywhere trying to help. The lead commander on the scene gave this instruction to his sergeant: “Slow this thing down to a crawl. Give these bastards no chance to fuck up in a meaningful way.”
And then it hit me. That’s exactly how they ran the calls. I asked them about this, and they said absolutely – human nature, when things are broken badly, is biased towards action. You want to try to make a change, to reboot a system, to do something to hopefully make things better. But often if you were not careful, you risk losing information about the outage you are in. Best case scenario you luck into fixing the problem and don’t know how. Worst case? You’ve changed the state of an already broken system, and have done nothing but add more variables to unwind.
So now, every time I’m on an outage call, I try to do what Wes and Tim and Major Rawls would all do: I take control, pump the brakes, and make sure that we are capturing enough information about the current state that we don’t confuse ourselves further.
>take weeks for other companies to publish a postmortem
And with nowhere near the detail level of what was presented here. Typically lots of sweeping generalizations that don't tell you much about what happened, or give you any confidence they really know what happened or have the right fix in place.
To be fair though they sort of MUST do things like this to have our confidence - their whole business is about being FAST and AVAILABLE. Were not talking about Oracle here :-D
Well cloudflare’s entire value is in uptime and preventing outages. Showing they have a rapid response and strong fundamental technical understanding is much more critical in the “prevent downtime” business.
No provider is perfect, but it's because of stuff like this that I trust Cloudflare waaaaaaaaaaay more than the likes of Amazon. Transparency engenders trust, and eventually, love! Thank you, Cloudflare.
The sheer level of technical competence of your engineering team continues to astound me. (Yes, they made a mistake and didn't catch an error in the diff. But your response process went exactly as it should, and your postmortem is excellent.) I couldn't even begin to think about designing or implementing something of this complexity, much less being able to explain it to a layperson after a failure. It is really impressive, and I hope you will continue to do so into the future!
Most of the companies I've worked for unfortunately don't use your services, but I've always been a staunch advocate and converted a few. Maybe the higher-ups only see downtime and name recognition (i.e. you're not Amazon), but for what it's worth, us devs down the ladder definitely notice your transparency and communications, and it means the world. I've learned to structure my own postmortems after yours, and it's really aided in internal communications.
Thank you again. I can't wait for the day I get to work in a fully-Cloudflare stack :)
AWS is pretty decent if you're in an NDA contract (you have paid support). You can request RCAs for any incident you were impacted and they'll usually get them within a day.
Not as transparent as "post it on the internet" but at least better than the usual hand wavey bullshit
CloudFlare are a hosting provider and CDN, they aren't "push[ing] ... hard for recentralization of the web".
If it was AWS, Akamai, Google Cloud, or any of the other massive providers this comment wouldn't be made. I don't really understand the association between centralisation and CloudFlare, other than it being a Meme.
It's often mentioned about AWS, especially when us-east-1 fails. The others are not big enough to affect basically "the internet" when they go down, so don't get pointed out as centralisation issues as much.
And yeah, cf is trying to get as much traffic to go through them as possible and add edge services for more opportunities - that's literally their business. Also now r2 with object storage. They're already too big, harmful (as in actually putting people in danger) and untouchable in some ways.
What do you have when you have all DNS going through them, via DoH, and all web requests going through them, if not recentralization?
Sure, they want us to think they give us the freedom to host our web sites anywhere because they're "protected" by them, but that "protection" means we've agreed to recentralize.
It's pretty dismissive to describe something as a meme just because you don't understand it, and either you're pretending to not understand it, or you truly don't.
Look at it this way: If a single company goes down for an hour, and that company going down for an hour causes half the web traffic on the Internet to fail for that hour, what is that if not recentralization?
I understand that for their WAF, DDOS and threat detection products they need to have a very large amount of traffic going through them. They have been very aggressive with their free service to achieve that, to the benefit of all their customers (including the free ones). Some could see that as a push to at centralisation, I don't.
What I don't understand, or believe, is that they want to be the sole (as in centralised) network for the internet. I don't believe they as a company, or the people running it, want that. They obviously have ambition to be one of the largest networking/cloud providers, and are achieving that.
I don't intend either to dismiss your concerns (which are a legitimate thing to have, centralisation would be very bad), my suggestion with the meme comment is that there is at times a trend to "brigade" on large successful companies in a meme-like way. That isn't to suggest you were.
They want to be a monopoly. They want everyone to depend on them. They may not want recentralization in general, but they definitely want as much of the Internet to depend on them as possible.
It's very difficult if not impossible to create a staging environment that would well enough replicate production at this scale. What bog posts suggest as a remediation in the process:
"There are several opportunities in our automation suite that would mitigate some or all of the impact seen from this event. Primarily, we will be concentrating on automation improvements that enforce an improved stagger policy for rollouts of network configuration and provide an automated “commit-confirm” rollback. The former enhancement would have significantly lessened the overall impact, and the latter would have greatly reduced the Time-to-Resolve during the incident."
07:42: The last of the reverts has been completed. This was delayed as network engineers walked over each other's changes, reverting the previous reverts, causing the problem to re-appear sporadically.
Now I'm remembering the story of how, when a certain blue website fell off the Internet for a day ~a decade and a half ago (due to some slightly broken database migration logic), out-of-band access boiled down to who was still logged in (!): https://rachelbythebay.com/w/2019/01/20/quiet/
Amusingly when things went wrong again last year, it was BGP's fault (is this the hyperscale equivalent of "it's always DNS" or something?). Engineers (with adequate credentials) had to actually drive to the datacenter haha.
I would be very interested to hear more about how the break-glass process worked.
This was something I was surprised not to see directly addressed in terms of follow up steps. When discussing process changes, they mention additional testing, but nothing to address what seems to be a significant communication gap.
I'm sure they have a more detailed internal postmortem, and I imagine it'd go into that. This is a nice high-level overview. They probably don't want to bury that under details of their communication processes, much less go into exactly who did what when for wide consumption by an audience that may not be on board with blameless postmortem culture.
You're probably right, but if they're going to mention it as part of the problem, I would want to see it as part of the solution. However, I agree that they certainly shouldn't name names.
I think I experienced first-hand the moment those network engineers were reverting their own reverts, breaking the web again. For example, DoorDash.com had come back online, then went back to serving only HTTP 500 errors from Cloudflare, then came back online again. I raised it in the HN discussion and @jgrahamc responded minutes later.
I'd be super interested in understanding what this means concretely. For example, are we talking about reverting commits? If so, why were engineers reverting reverts?
also can happen when your deploy process has two flows for revert a forward movement revert (where new bits and head are committed fixing the items that needed to be reverted) and a "previous head" revert which just goes back one revision in the rcs (or tagged version).
Imagine the first eng team did a forward movement revert that corrected the issue and had a new head bits that gets deployed, where shortly after another eng fires off the second process type and tells the system to pull back to the last revision (which is now the bad revision as it was just replaced with fresher deploy bits).
Having two revert processes in the toolkit and maybe a few disperse teams working to revert the issue without tight communication leads to this issue.
I think this is more likely the basis issue vs a bad merge (I assume that the root cause was broadcasted wide and large to anyone making a merge)
Time and time again, this type of response proves that it's the right way handle a bad situation. Be humble, apologize, own your mistake, and give a transparent snapshot into what went wrong and how you're going to learn from the mistake.
Or you could go the opposite direction and risk turning something like this into a PR death spiral.
Exactly. I trust businesses/people that are transparent about their mistakes/failures much more than the ones that avoid them (except Apple which never accepts their mistakes, but I still trust their products, I think I'm affected by RDF).
At the end of the day, everybody makes mistakes and that's okay. Everybody else also know that everybody makes mistakes. So why not accept it?
I really don't get what's wrong with accepting mistakes, learning from them, and moving on.
> I really don't get what's wrong with accepting mistakes, learning from them, and moving on.
Some people really struggle with this (myself included) but I think it's one of the easiest "power ups" you can use in business and in life. The key is that you have to actually follow through on the "learning from them" clause.
I would agree with that. Apple Maps was worse than the hockey puck mouse or the trashcan macpro. trying to decide if it is worse than the butterfly keyboard, but I think the keyboard wins for the shear fact that it impacted me in a way that was uncorrectable where I could just use a different Maps app
I agree, but lately (as in the past month) I've been finding myself using apple maps more and more than google. When on a complicated highway interchange, the 3d view that Apple Maps gives for which exit to take is a life-saver
Recently I used Apple Maps much more than Google Maps.
In addition to trying to de-Googlify my life, there was also an occurance where Google Maps literally tried to kill me: at an intersection that connects into a highway it guided me to drive straight into the opposite direction to a highway, straight onto the coming cars at 140km/h. I've quit Google Maps right there and never used it again.
They didn’t apologize about the direction the pro macs were going a few years back but they certainly listened and made amends for it with the recent Pro line and MacBook Pro enhancements
Am I the only who really doesn't think this is a big deal? They had an outage, they fixed it very quickly. Life goes on. Talking about the outage as if it's reason for us to all ditch CF, then buy/ run our own hardware (which will be totally better), so hyperbolic.
It was a bit of a thing as people in Europe started their office work, and found out a lot of their internet services were down, and they were unable to access the things they needed. It's rather dangerous that we all depend on this one service being online.
There was a common pattern in use back in the day when I managed openbsd filewalls (can't remember if it was ipf or pf days). When changing firewall rules over ssh, you'd use a command line like:
$ apply new rules; sleep 10; apply original rules
If your ssh access was still working and various sites were still up during that 10sec you were probably good to go - or at least you hadn't shut yourself out.
Back when I was a briefly a network engineer at the start of my career, on cisco equipment we'd do 'reload in 5' before big changes - so it'd auto restart after 5 minutes unless cancelled.
I'm sure there were and are better ways of doing it, but it was simple enough and worked for us.
most ISP tier routers have an entire commit engine to load and apply configs.
junipers allows for instance, one to do the command commit confirmed, which will apply the configuration, and revert back to the previous version if one does not acknowledge this command within a predifined time.
this prevents permanent lockout out of a system.
correct, i saw that too. the outage returned 500/nginx. no version number either on footer. @jgrahamc thought that was strange too as few commenters last night were caught off guard trying to determine if it was their systems or cloudflare. supposedly its been forwarded along.
yes, there is definitely an nginx service in the path. We don't have any nginx in our infrastructure, but this was the response we had for our urls during the outage.
<html> <head><title>500 Internal Server Error</title></head> <body bgcolor="white"> <center><h1>500 Internal Server Error</h1></center> <hr><center>nginx</center> </body> </html>
I’m surprised they did not conclude roll outs should be executed over longer period with smaller batches. When a system is complicated as theirs with so much impact, the only sane strategy is slow rolling updates so that you can hit the brake when needed.
Actually, I think the flip side is even more interesting. If you want to give good, low latency service to 50% of the world you need a lot of data centers.
Well half of those cities were in Asia during business hours, so given that the majority of humans live in Asia it makes sense. CF data centers in Asia also seem to be less distributed than in the West (e.g. Vietnam traffic seems to go to Singapore) meanwhile CF has multiple centers distributed throughout the US.
This is a great concise explanation. Thank you for providing it so quickly
If you forgive my prying, was this an implementation issue with the maintenance plan (operator or tooling error), a fundamental issue with the soundness of the plan as it stood, or an unexpected outcome from how the validated and prepared changes interacted with the system?
I imagine that an outage of this scope wasn’t foreseen in the development of the maintenance & rollback plan of the work.
Yes, that was one of the issues they mentioned in the post. Not that they didn’t have a staging/testing environment but that it didn’t include the specific type of new architecture configuration, “MCP”, that ultimately failed.
One of their future changes is to include MCPs in their testing environments.
Yet another BGP caused outage. At some point we should collect all of them:
- Cloudflare 2022 (this one)
- Facebook 2021: https://news.ycombinator.com/item?id=28752131 - this one probably had the single biggest impact, since engineers got locked out of their systems, which made the fixing part look like a sci-fi movie
There are lots of other causes of incidents, like cut cables, failed router hardware, data centers losing power etc.
It just seems that most of these are local enough and the Internet resilient enough that they don't cause global issues. Maybe the exception would be AWS us-east-1 outages :-)
Maybe a testament to BGP's effectiveness that so many large-scale outages are due to misconfiguring BGP rather than the frequent cable cuts and hardware failures that BGP routes around.
These are the public facing BGP announcements that cause problems, but doesn't account for the ones on private LANs that also happen. Previous employers of mine have had significant internal network issues because internal BGP between sites started causing problems. I'm not sure there's anything better (I am not a network guy), but this list can't be exhaustive.
"Due to this withdrawal, Cloudflare engineers experienced added difficulty in reaching the affected locations to revert the problematic change. We have backup procedures for handling such an event and used them to take control of the affected locations."
But Cloudflare had sufficient backup connectivity to fix it. I'm curious how Cloudflare does that today-- the solution long ago was always a modem on an auxiliary port.
Not sure how common it is, but you can get serial OOBM devices accessible over cellular which would then give you access to your equipment.
I'm surprised more places don't implement a "click here to confirm changes or it'll be rolled back in 5 minutes" like all those monitor settings dialogues
Except that this wasn't an example of BGP being prone to error or fragile. This was, as the blog post specifically calls out, human error. They put two BGP announcement rules after the "deny everything not previously allowed" rule. It's the same as if someone did this to a set of ACLs on a firewall.
The main difference between BGP and all other tools is that if you mess up BGP, you've done a very visible thing because BGP underpins how we get to each other's networks. But it's not a sign of BGP being fragile, just very important.
Some tools are more prone to human error than others.
Another canonical example is C++. Some tools make it easy to blow your leg off. Some tools provide safety mechanisms to stop the saw from cutting off your finger.
That does seem like bad UX/"DevX" that that configuration of rules is "valid" syntactically and there weren't better equivalents of "linters"/"compilers" flagging that before it ever got sent out as an announcement. UX issues are a "proneness" to error/fragility. It sounds like there is room to build a "higher level language" (like a "Typescript : Javascript :: ? : BGP") for BGP announcements that is less prone to "accidentally bad programs". Not that I have immediate suggestions, just that my gut reaction from skimming these sorts of outage reports is that if it was a "language" I was writing in I can hear that I'd want a lot more (type) safety nets.
You say that like it hasn't been going on since the mid 1990's, when it got deployed.
I'm not blaming BGP, since it prevents far more outages than it causes, but BGP-based outages have been a thing since its beginning. And any other protocol would have outages too - BGP just happens to be the protocol being used.
I was on a severely understaffed edge team fronting several thousand engineers at a fortune 500 - every deploy felt like a spacex launch from my cubicle. I have a lot of reverence for the engineers who take on that kind of responsibility.
Something else that I think would be smart to implement is a reorder detection. Have the change approval specificy point out stuff that gets reordered, and require manual approval for each section that gets moved around.
I also think that having a script that walks through the file and points out any ovibious mistakes would be good to have as well.
Yeah, there's got to be some sweet spot between "formally verify all the things" and "i guess this diff looks okay, yolo!".
I'd say that if you're designing a system which has the potential to disconnect half your customers based on a misconfiguration, then you should spend at least an hour thinking about what sorts of misconfigurations are possible, and how you could prevent or mitigate them.
The cost-benefit analysis of "how likely is it such a mistake would get to production (and what would that cost us)?" vs "how much effort would it take to write and maintain a verifier that prevents this mistake?" should then be fairly easy to estimate with sufficient accuracy.
233 comments
[ 40.3 ms ] story [ 2399 ms ] threadI think it's a testament to their Ops/Incident response teams and internal processes, it builds confidence in their ability to respond quickly when something does go wrong. Incredible work!
Don't companies have a fiduciary duty to calculate things; the reason for doing something actually cannot just be that it's a nice thing to do? Not down to the word, but at least the general decision to be this way?
https://scholarship.law.cornell.edu/cgi/viewcontent.cgi?http...
Additionally, these post-mittens work for Cloudflare because they have a great reputation and good uptime. If this were happening daily or weekly, it would be a warning sign to customers.
It’s a strategy other companies could adopt, but to do it effectively requires changes all across the organization.
1. On the buy side, crappy big companies with procurement etc. may not have “actual engineers” deciding things. Maybe for something like Cloudflare that’s likely to sit within an actual technical team’s mandate
2. Not 100% foolproof but if a startup is selling its tool and has an uptime page and details of all its downtime and as a prospective customer you go and see that they have 3-4 hour downtimes once or twice a month, it should raise alarm bells.
I think their calculation (to the extent you can call it that) is that in the interest of PR and damage control, it’s better to get a thorough postmortem out quickly to stem the bleeding and keep people like us from going “I can’t wait to hear what happened at Cloudflare” for a week. Now we know, the customers have an explanation, and this bad news cycle has a higher chance of ending quickly.
Incidents - yes. But why would a post-mortem turn someone off? The incident happened regardless. Do you think anyone would be more likely turned off by reading how they solved it / plan to prevent it on the future than by silence?
IMO it is a slippery slope to see this as opportunity too strongly. Sure, doing the right thing may be net beneficial to the business in the long run...but the $RIGHT_THING should be done first and foremost because it's the right thing.
But once it happens things will change and, to be honest, likely for the worse.
edit> fix typo
Succession?
Doing incident response (both outage and security), the tactical fixes for a specific problem are usually pretty easy. We can fix a bug, or change this specific plan to avoid the problem. The search for conditions that allowed the incident to occur can be alot more time consuming, and most organizations I've worked for are happy to make a couple tactical changes and move on.
There is not. From here there's an ongoing process with a formal post-mortem, all sorts of tickets tracking work to prevent further reoccurrence. This post is just the beginning internally.
> This was delayed as network engineers walked over each other's changes, reverting the previous reverts, causing the problem to re-appear sporadically.
They are running as fast as they can and this extended the incident. There is a “slow is smooth, smooth is fast” lesson in here. I’d rather have a team that takes a day to put up the blog post, but doesn’t unnecessarily extend downtime because they are sprinting.
At around the same time, I was watching HBO’s The Wire. One scene had a high-profile shooting with a frantic police response; people were running everywhere trying to help. The lead commander on the scene gave this instruction to his sergeant: “Slow this thing down to a crawl. Give these bastards no chance to fuck up in a meaningful way.”
And then it hit me. That’s exactly how they ran the calls. I asked them about this, and they said absolutely – human nature, when things are broken badly, is biased towards action. You want to try to make a change, to reboot a system, to do something to hopefully make things better. But often if you were not careful, you risk losing information about the outage you are in. Best case scenario you luck into fixing the problem and don’t know how. Worst case? You’ve changed the state of an already broken system, and have done nothing but add more variables to unwind.
So now, every time I’m on an outage call, I try to do what Wes and Tim and Major Rawls would all do: I take control, pump the brakes, and make sure that we are capturing enough information about the current state that we don’t confuse ourselves further.
And with nowhere near the detail level of what was presented here. Typically lots of sweeping generalizations that don't tell you much about what happened, or give you any confidence they really know what happened or have the right fix in place.
The sheer level of technical competence of your engineering team continues to astound me. (Yes, they made a mistake and didn't catch an error in the diff. But your response process went exactly as it should, and your postmortem is excellent.) I couldn't even begin to think about designing or implementing something of this complexity, much less being able to explain it to a layperson after a failure. It is really impressive, and I hope you will continue to do so into the future!
Most of the companies I've worked for unfortunately don't use your services, but I've always been a staunch advocate and converted a few. Maybe the higher-ups only see downtime and name recognition (i.e. you're not Amazon), but for what it's worth, us devs down the ladder definitely notice your transparency and communications, and it means the world. I've learned to structure my own postmortems after yours, and it's really aided in internal communications.
Thank you again. I can't wait for the day I get to work in a fully-Cloudflare stack :)
Not as transparent as "post it on the internet" but at least better than the usual hand wavey bullshit
If it was AWS, Akamai, Google Cloud, or any of the other massive providers this comment wouldn't be made. I don't really understand the association between centralisation and CloudFlare, other than it being a Meme.
And yeah, cf is trying to get as much traffic to go through them as possible and add edge services for more opportunities - that's literally their business. Also now r2 with object storage. They're already too big, harmful (as in actually putting people in danger) and untouchable in some ways.
What do you have when you have all DNS going through them, via DoH, and all web requests going through them, if not recentralization?
Sure, they want us to think they give us the freedom to host our web sites anywhere because they're "protected" by them, but that "protection" means we've agreed to recentralize.
It's pretty dismissive to describe something as a meme just because you don't understand it, and either you're pretending to not understand it, or you truly don't.
Look at it this way: If a single company goes down for an hour, and that company going down for an hour causes half the web traffic on the Internet to fail for that hour, what is that if not recentralization?
What I don't understand, or believe, is that they want to be the sole (as in centralised) network for the internet. I don't believe they as a company, or the people running it, want that. They obviously have ambition to be one of the largest networking/cloud providers, and are achieving that.
I don't intend either to dismiss your concerns (which are a legitimate thing to have, centralisation would be very bad), my suggestion with the meme comment is that there is at times a trend to "brigade" on large successful companies in a meme-like way. That isn't to suggest you were.
It's pretty appalling that you are even being downvoted.
Ouch
Now I'm remembering the story of how, when a certain blue website fell off the Internet for a day ~a decade and a half ago (due to some slightly broken database migration logic), out-of-band access boiled down to who was still logged in (!): https://rachelbythebay.com/w/2019/01/20/quiet/
Amusingly when things went wrong again last year, it was BGP's fault (is this the hyperscale equivalent of "it's always DNS" or something?). Engineers (with adequate credentials) had to actually drive to the datacenter haha.
I would be very interested to hear more about how the break-glass process worked.
https://news.ycombinator.com/item?id=31821290
Imagine the first eng team did a forward movement revert that corrected the issue and had a new head bits that gets deployed, where shortly after another eng fires off the second process type and tells the system to pull back to the last revision (which is now the bad revision as it was just replaced with fresher deploy bits).
Having two revert processes in the toolkit and maybe a few disperse teams working to revert the issue without tight communication leads to this issue.
I think this is more likely the basis issue vs a bad merge (I assume that the root cause was broadcasted wide and large to anyone making a merge)
Or you could go the opposite direction and risk turning something like this into a PR death spiral.
At the end of the day, everybody makes mistakes and that's okay. Everybody else also know that everybody makes mistakes. So why not accept it?
I really don't get what's wrong with accepting mistakes, learning from them, and moving on.
Some people really struggle with this (myself included) but I think it's one of the easiest "power ups" you can use in business and in life. The key is that you have to actually follow through on the "learning from them" clause.
https://appleinsider.com/articles/12/09/28/apple-ceo-tim-coo...
This one line will forever cement exactly how bad Apple Maps' release was. Thanks Mike Judge!
In addition to trying to de-Googlify my life, there was also an occurance where Google Maps literally tried to kill me: at an intersection that connects into a highway it guided me to drive straight into the opposite direction to a highway, straight onto the coming cars at 140km/h. I've quit Google Maps right there and never used it again.
Apparently so terrible that Apple apologized, perhaps for the first (and last) time for something.
at time of writing no comment has done that except you.
For their part, they handled this very well, and are to be commended (quick fix, quick explanation of failure).
But you also can't help but see that they have a dangerous amount of control over such important systems.
It should revert as a failsafe if not confirmed within X minutes.
> Primarily, we will be concentrating on automation improvements ... and provide an automated “commit-confirm” rollback.
$ apply new rules; sleep 10; apply original rules
If your ssh access was still working and various sites were still up during that 10sec you were probably good to go - or at least you hadn't shut yourself out.
I'm sure there were and are better ways of doing it, but it was simple enough and worked for us.
junipers allows for instance, one to do the command commit confirmed, which will apply the configuration, and revert back to the previous version if one does not acknowledge this command within a predifined time. this prevents permanent lockout out of a system.
https://i.imgur.com/xHqvOzj.png
<html> <head><title>500 Internal Server Error</title></head> <body bgcolor="white"> <center><h1>500 Internal Server Error</h1></center> <hr><center>nginx</center> </body> </html>
Everybody at one time experiences the dreaded REJECT not being at the end of the rule stack but just too early.
Kudos to CF for such a good explanation of what caused the issue.
Even better if the tool was syntax aware so it could highlight the different types of rules in unique colors.
If you forgive my prying, was this an implementation issue with the maintenance plan (operator or tooling error), a fundamental issue with the soundness of the plan as it stood, or an unexpected outcome from how the validated and prepared changes interacted with the system?
I imagine that an outage of this scope wasn’t foreseen in the development of the maintenance & rollback plan of the work.
Sounds like a good idea!
One of their future changes is to include MCPs in their testing environments.
- Cloudflare 2022 (this one)
- Facebook 2021: https://news.ycombinator.com/item?id=28752131 - this one probably had the single biggest impact, since engineers got locked out of their systems, which made the fixing part look like a sci-fi movie
- (Indirectly caused by BGP: Cloudflare 2020: https://blog.cloudflare.com/cloudflare-outage-on-july-17-202...)
- Google Cloud 2020: https://www.theregister.com/2020/12/16/google_europe_outage/
- IBM Cloud 2020: https://www.bleepingcomputer.com/news/technology/ibm-cloud-g...
- Cloudflare 2019: https://news.ycombinator.com/item?id=20262214
- Amazon 2018: https://www.techtarget.com/searchsecurity/news/252439945/BGP...
- AWS: https://www.thousandeyes.com/blog/route-leak-causes-amazon-a... (2015)
- Youtube: https://www.infoworld.com/article/2648947/youtube-outage-und... (2008)
And then there are incidents caused by hijacking: https://en.wikipedia.org/wiki/BGP_hijacking#:~:text=end%20us...
Some more:
- Google 2016, configuration management bug/BGP: https://status.cloud.google.com/incident/compute/16007
- Valve 2015: https://www.thousandeyes.com/blog/steam-outage-monitor-data-...
- Cloudflare 2013: https://blog.cloudflare.com/todays-outage-post-mortem-82515/
It just seems that most of these are local enough and the Internet resilient enough that they don't cause global issues. Maybe the exception would be AWS us-east-1 outages :-)
Sounds like the same happened here:
"Due to this withdrawal, Cloudflare engineers experienced added difficulty in reaching the affected locations to revert the problematic change. We have backup procedures for handling such an event and used them to take control of the affected locations."
But Cloudflare had sufficient backup connectivity to fix it. I'm curious how Cloudflare does that today-- the solution long ago was always a modem on an auxiliary port.
Also lets face it - the utility of a trusted security guard/staff with an old fashioned physical key is pretty hard to screw up!
Now you can use mobile Internet (4G/5G)
I'm surprised more places don't implement a "click here to confirm changes or it'll be rolled back in 5 minutes" like all those monitor settings dialogues
BGP is just a tool, it would be something else to do the same purpose.
The main difference between BGP and all other tools is that if you mess up BGP, you've done a very visible thing because BGP underpins how we get to each other's networks. But it's not a sign of BGP being fragile, just very important.
Another canonical example is C++. Some tools make it easy to blow your leg off. Some tools provide safety mechanisms to stop the saw from cutting off your finger.
I'm not blaming BGP, since it prevents far more outages than it causes, but BGP-based outages have been a thing since its beginning. And any other protocol would have outages too - BGP just happens to be the protocol being used.
You broke half the internet: BGP You broke half of your company's ability to access the internet: DNS
I also think that having a script that walks through the file and points out any ovibious mistakes would be good to have as well.
I'd say that if you're designing a system which has the potential to disconnect half your customers based on a misconfiguration, then you should spend at least an hour thinking about what sorts of misconfigurations are possible, and how you could prevent or mitigate them.
The cost-benefit analysis of "how likely is it such a mistake would get to production (and what would that cost us)?" vs "how much effort would it take to write and maintain a verifier that prevents this mistake?" should then be fairly easy to estimate with sufficient accuracy.
The problem is, we couldn't tell all our client they should change this :(