As a general rule, Cloudflare hot paths (CDN, etc.) tend to still work during an API outage. Communication between the API servers and the hot paths are mostly async, and (usually) work fine independently.
You and all the others did not get it did you. The size of the outcry corresponds to the user base. If you fked something up that was really used at scale this thread would have exploded.
This has become my knee jerk reaction to these kinds of outages. It's always interesting when the more indepth engineering-focused post-mortem comes out
If you’re waiting for the inevitable “this is why we shouldn't centralize everything”; this is that comment.
As much as I like cloudflare and easy to use stuff, lets be a little bit mindful of the blast radius of highly centralized services which are even more complex to operate due to the scale of the deployment and problem scope.
luckily this time its just administrative interfaces and supplemental services that assist in development and that hopefully aren't used in production.
People will defend cloudflare here with the old saying “Operating at scale is hard” but simultaneously holding the opinion that operating services themselves is equally hard.
I guess the argument is that outsourcing is easier, but then you are very much out of control.
“the internet” in its practical entirety going down becomes more possible every passing day.
However the internet doesn't require people to "operate at scale". That's the beauty. It's a mash of different (relatively) small autonomous systems, with many hosts. If any host goes down, or any network goes down, the system ignores it, or reroutes it.
In practice, cloudflare or AWS down means half the internet doesn't work at all. The half that works - routing around problematic networks - isn't very helpful when the problematic networks are the useful ones.
> Depends what you want to get to. HN doesn't use any centralised services.
It's also one of the least reliable and performant public forums I've ever used. It goes down or slows down several times a week for me. Wish they'd just move it to the cloud instead of occasionally upgrading the bare metal...
Well technically they do, but honestly the number of companies that need to worry about redundant DNS is probably .001%. It does matter for the some people, but as someone from SRE once said "100% availability is the wrong target for basically everything".
> However the internet doesn't require people to "operate at scale".
The internet itself does not, yes, but the internet as a system, filled to the brim with bad actors ranging from 14 year old skiddies with Metasploit over hardened Russian ransomware groups to virtually all nations' secret services and militaries does.
As soon as you start a server with a globally routed IP address, it will get hammered from all kinds of scanners. Anything on port 80/443 will get bombarded with Wordpress, Drupal and log4j attempts, anything on port 22 will get hounded by people attempting credential stuffing, and anything on port 25/587 by spammers and anyone trying to exploit Exchange/Exim vulnerabilities. It's a nightmare to keep up with that crap alone, and you're one forgotten update away from getting your server 0wned by someone using Shodan and whatnot.
And then, once your site, whatever you operate - I've seen as an admin DDoS extortion attacks happen against small sites for dog walker services, pizza parlors or random bloggers as well as large car dealerships - finds itself some opponent for whatever reason, it gets nasty. Be it for political reasons (people don't like what the blogger writes), for the lulz (people don't like a Twitch streamer so they DDoS their sponsors), or for financial extortion, doesn't matter in a time where you can order such attacks for less than the price of your average McDonalds visit. There is no, utterly no, alternative to the giants (Cloudflare, AWS CloudFront, Akamai).
It's sad that it has come to that point, but unless we get the most obnoxious sources of such attacks - enemy nation states (China, Russia) and compromised domestic IoT devices/servers - off of the Internet, there is really no viable alternative.
Whats the impact if the entire internet is down for a couple of hours?
What if its more than a few hours because theres a natural monopoly and a dearth of capable people to run systems independently - thus no strong external incentive for rapid response outside of punitive SLAs?
How much soft power do you think such an entity would have on the world stage.
> Whats the impact if the entire internet is down for a couple of hours?
People get some off-screen time? Life goes on.
> What if its more than a few hours because theres a natural monopoly?[...snip...] How much soft power do you think such an entity would have on the world stage.
They're just commodity infra providers. The Facebook and Twitters of the world have way more power and influence. This isn't even a hypothetical. The big cloud providers have been around for what, a decade or so now? How many times has the world collapsed with every Cloudflare or AWS downtime? It's just not as big a deal as you paint it to be. These providers going down is like the digital equivalent of a highway crash -- a small inconvenience for a few hours, then forgotten about in a day or two.
> Whats the impact if the entire internet is down for a couple of hours?
Okay, so obviously a lot of "important" stuff happens via the Internet. You could come up with some financial impact if the Internet went down for some time.
Harder to measure: A lot of people would be inconvenienced or suffer. Some number of people would die indirectly, e.g. due to emergencies/lack of communication.
People are inconvenienced and suffer and die all of the time. It's part of life.
I'm not advocating for more death and suffering, but I am saying that, as another comment said, "life goes on".
---
Put another way, if the Internet disappeared for a week, month, year, what would happen? It'd suck and we'd adjust, but we (humanity as a whole) would be fine.
Most people blame Cloudflare and Amazon for centralizing everything.
I actually disagree; the centralization is a symptom of severe, systemic, unfixable flaws in the design of the internet itself. At least we didn't let AOL define internet.
3. Geolocation attacks (you shouldn't run a server from your home, when your IP Address could be used to find you; be it from a hack or an abuse of the legal process)
4. The US internet relying on a backbone mostly operated by 4 companies (meaning a direct connection is a massive perk)
For as much as we moan about Amazon and Cloudflare centralizing things; believe me, just taking out Level 3 / Lumen would be almost an act of war and would affect every ISP and cloud provider.
If you took out Lumen, Verizon Enterprise Solutions, and AT&T, just those 3 companies; that's 2.35 million kilometers of fiber going dark. For reference, Comcast has "only" 150,000 km.
I actually have a lot of respect for the big backbones. If Goog/MS/Amz owned them all, they'd find some way to wall it off from everyone else. If Comcast or Charter owned them all, our internet would cost about $1000 a month by now. All things considered, IMO they've been fairly good and fair stewards.
I'd appreciate if you'd elaborate on that. (Edit: I see you elaborated somewhat in another comment posted before I finished this comment)
I tend to think centralization is more about people's tendencies than it is internet design. As I see it, centralization on the internet happens as the emergent behavior of all of the elements in the system and the forces that exist. In other words, it's the natural result of incentives.
People use centralized services because
- prices (economies of scale)
- network effects ("I can't certain people without WhatsApp")
- social pressures ("People look at me weird when I say I don't have an
Instagram")
- visibility ("There are other CRMs than SalesForce?")
- making choices easy ("I don't want to choose between all these options. I'll pick the most familiar")
- safety ("Nobody got fired for choosing IBM")
- convenience ("It's a pain to self-host")
- security ("Google probably protects data better than I can")
True; but let's imagine everyone self-hosted. Perfectly decentralized. As I edited another comment to reflect, imagine if you took out just three companies.
With those three, you've knocked 2.35 million km, or 1.46 million miles, of fiber optic offline. That's over half of the US backbone in those three companies. Comcast "only" has 0.15 million miles for the sake of comparison. The internet would crash and burn immediately to speeds not seen since the dial-up era. There's nothing we can do about that. Cloudflare and AWS centralizing everything is actually, statistically, less risky than backbone attacks. If you succeeded in knocking down all of the big seven at once; you've got about 80-90% of the backbone offline plus all of the online cloud providers with direct connections to them; at which point the internet would cease to function in any practical sense.
I don't think this is more centralized, because it requires taking out multiple companies services. To take down three of these networking companies at once is something we've never actually seen, and that's the level needed to be comparable to a CloudFlare failure.
The current situation is preferable to the before times, where any website could be nuked from orbit by anyone with a mild grievance. Cloudflare has succeeded in effectively shutting down that attack vector, and that’s worth the risk of occasional brief but broad outages, at least to me.
There were mitigations for this but it was usually on “IT”s budget and IT was considered a cost center.
Really the best innovation in the last 10 years has been the penchant for putting things on development budgets instead which loosened the purse strings considerably.
I wish I had sufficient data to back this up but for context; I was part of a small infrastructure team running a little over 1% of all web traffic at one point in a pair of colos (with DDoS protection) for less than £5MM on infra costs.
Now I can easily be in the 6 digits with just a CRUD app
If you think internet infrastructure is centralized, boy are you going to be depressed about AI powered services.
Those are much more volatile (compute heavy and not enough compute to meet peaks), harder to setup/maintain and benefit greatly from sharing infrastructure for hardware utilization. Everything is moving in that direction.
Is it really that bad? Services go down for a few minutes, maybe a few hours? A thousand other companies are down at the same time.
Grab lunch, come back, and it'll have fixed itself. Nobody blames you because it's just another provider outage -- whether it's Cloudflare or AWS or Microsoft or Gmail doesn't really matter, it's a convenient outsourcing of not just network admin but also blame.
The alternative is, what, doing it all in-house with a bunch of people on pager duty? Few companies can afford that. Diversifying across multiple cloud providers? Sure, if multi-nines uptime is really critical for your business.
For most small-med businesses, it's more valuable to be able to rapidly build on top of existing, professionally-maintained infrastructure in order to deliver some unique business value on top of it, than to recreate that basic network infra themselves. If you're not a FAANG, going down for a few hours a year isn't really a big deal. Especially when you can just blame it on an overall "internet outage at some big provider" rather than some internal fuck-up.
I don't think the problem is a few minutes of down time but all services going down at the same time. You can't order food (Uber down), your email down, your TV down, maybe even the traffic control?
Are power outages even a thing? My PC at home (germany) is always on, so I'd notice, and I'm involved with energy at our data center (Frankfurt, Germany) and home and external data center had no outage for the last 20 years, at a minimum.
That actually happens with some frequency where I live, and they either use the old-school card imprint machines, or take down your number manually, or some terminals have an offline mode where it'll save the cards and then batch-run them later. It's also rare enough that it's not an issue at all in regular life.
Edit: Just to clarify, obviously this can't be both common and rare... what I meant is that different neighborhoods will frequently go down, but there's usually some neighborhood within a 10-min drive that still has power, or some food truck has a cellular connection off their iPad PoS that still works, or you happen to have cash in your wallet or whatever or know the guy and they'll just put it on your tab for next time, whatever. I live in a semi-rural community in Oregon and we have a lot of small businesses, a lot of wildfires, and not a whole lot of infrastructure. We just kinda make do when things happen.
A couple weeks ago, half the town lost power for the whole day, and the other half of the town just got some increased business and traffic. Life was back to normal the next day.
I think the solution here isn’t “don’t use Cloudflare” … IMO if you need high availability and can’t tolerate downtime, then deploy and split load between 2 or more providers (e.g. Cloudflare, Fastly) and 2 or more clouds.
The centralization around AWS and GCP and Azure is more concerning to me than the centralization around Cloudflare. It’s not obvious to me why lots of people using Cloudflare is any worse than lots of people using AWS.
If we go even deeper, the entire internet runs on a limited number of undersea cables connecting continents. That in particular has always seemed like the most vulnerable aspect of the internet, considering a targeted attack on < 100 locations could meaningfully cut off internet access for millions (billions?) of people for an extended period of time.
This. If downtime is that big of an issue, you should be on Cloudflare's business/enterprise plan which allows CNAME DNS, have redundant DNS providers with 30s TTLs that flip over to another provider. Better yet if you can 50/50 the traffic and 100/0 it when something goes bad. Downside is cost and DNSSEC has an issue with multiple keys. I think there was an RFC on that though I'm not sure it's well supported yet.
Usually each one will give you a CBAME, your DNS or Cedexic decides, sometimes per client. It's a pretty common setup although can complicate config changes and taking advantage of new features.
Downtime happens - no one is happy about it but it comes with the job and downtime will happen whether you use cloud or run on prem.
Cloudflare at least accurately reflects the incident on their status page.
Often dashboards will show everything green despite the service being down.
In these moments communication is critical and Cloudflare does it well especially as we all know when it comes to the post mortem. They are very transparent about these issues and incidents.
It's also nuts that this downtime according to creation of the post -> resolved in comments, was only 27 minutes...
I've literally been on standby to get an answer what happened with a huge cloud provider for 3 weeks now and today they got back with static assets that weren't reachable ( ico's ???)
They don't even know what happened since it's not logged in their logging, since it's using middleware and because of the sudden weekend downtime, they middleware wasn't triggered...
After a detailed description that we found that it was related to a app service plan, they came back with 404's on static assets...
Only now it got escalated..
Fuck the cloud, outside of cloudflare.
The only time we had a downtime with cloudflare, they gave a detailed description and it was related to our cloud provider and not Cloudflare . Then their was a infra update and it also didn't get logged... Because of that we thought it was the wrong party...
18 hours down, redeployment fixed it, nothing logged on their end. Luckily one service wasn't redeployed ( barely used), so we can prove there was an issue.
> The only time we had a downtime with cloudflare, they gave a detailed description and it was related to our cloud provider and not Cloudflare
Hah. Cloudflare's rise to fame was them proudly announcing how their servers were running but the users misconfigured their site, by "defacing" the users' non-resonse.
92 comments
[ 3.1 ms ] story [ 68.8 ms ] threadAs much as I like cloudflare and easy to use stuff, lets be a little bit mindful of the blast radius of highly centralized services which are even more complex to operate due to the scale of the deployment and problem scope.
luckily this time its just administrative interfaces and supplemental services that assist in development and that hopefully aren't used in production.
People will defend cloudflare here with the old saying “Operating at scale is hard” but simultaneously holding the opinion that operating services themselves is equally hard.
I guess the argument is that outsourcing is easier, but then you are very much out of control.
“the internet” in its practical entirety going down becomes more possible every passing day.
However the internet doesn't require people to "operate at scale". That's the beauty. It's a mash of different (relatively) small autonomous systems, with many hosts. If any host goes down, or any network goes down, the system ignores it, or reroutes it.
If someone chooses to host their service relying on a single unreliable point of failure that's on them. Plenty more hosts on the web.
It's also one of the least reliable and performant public forums I've ever used. It goes down or slows down several times a week for me. Wish they'd just move it to the cloud instead of occasionally upgrading the bare metal...
dig SOA news.ycombinator.com
; <<>> DiG 9.18.16-1-Debian <<>> SOA news.ycombinator.com ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 10818 ;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1
;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 512 ;; QUESTION SECTION: ;news.ycombinator.com. IN SOA
;; AUTHORITY SECTION: ycombinator.com. 900 IN SOA ns-225.awsdns-28.com. awsdns-hostmaster.amazon.com. 1 7200 900 1209600 86400
;; Query time: 39 msec ;; SERVER: 127.0.0.1#53(127.0.0.1) (UDP) ;; WHEN: Wed Oct 25 10:52
The internet itself does not, yes, but the internet as a system, filled to the brim with bad actors ranging from 14 year old skiddies with Metasploit over hardened Russian ransomware groups to virtually all nations' secret services and militaries does.
As soon as you start a server with a globally routed IP address, it will get hammered from all kinds of scanners. Anything on port 80/443 will get bombarded with Wordpress, Drupal and log4j attempts, anything on port 22 will get hounded by people attempting credential stuffing, and anything on port 25/587 by spammers and anyone trying to exploit Exchange/Exim vulnerabilities. It's a nightmare to keep up with that crap alone, and you're one forgotten update away from getting your server 0wned by someone using Shodan and whatnot.
And then, once your site, whatever you operate - I've seen as an admin DDoS extortion attacks happen against small sites for dog walker services, pizza parlors or random bloggers as well as large car dealerships - finds itself some opponent for whatever reason, it gets nasty. Be it for political reasons (people don't like what the blogger writes), for the lulz (people don't like a Twitch streamer so they DDoS their sponsors), or for financial extortion, doesn't matter in a time where you can order such attacks for less than the price of your average McDonalds visit. There is no, utterly no, alternative to the giants (Cloudflare, AWS CloudFront, Akamai).
It's sad that it has come to that point, but unless we get the most obnoxious sources of such attacks - enemy nation states (China, Russia) and compromised domestic IoT devices/servers - off of the Internet, there is really no viable alternative.
Yet my servers are fine without such services.
What if its more than a few hours because theres a natural monopoly and a dearth of capable people to run systems independently - thus no strong external incentive for rapid response outside of punitive SLAs?
How much soft power do you think such an entity would have on the world stage.
People get some off-screen time? Life goes on.
> What if its more than a few hours because theres a natural monopoly?[...snip...] How much soft power do you think such an entity would have on the world stage.
They're just commodity infra providers. The Facebook and Twitters of the world have way more power and influence. This isn't even a hypothetical. The big cloud providers have been around for what, a decade or so now? How many times has the world collapsed with every Cloudflare or AWS downtime? It's just not as big a deal as you paint it to be. These providers going down is like the digital equivalent of a highway crash -- a small inconvenience for a few hours, then forgotten about in a day or two.
Okay, so obviously a lot of "important" stuff happens via the Internet. You could come up with some financial impact if the Internet went down for some time.
Harder to measure: A lot of people would be inconvenienced or suffer. Some number of people would die indirectly, e.g. due to emergencies/lack of communication.
People are inconvenienced and suffer and die all of the time. It's part of life.
I'm not advocating for more death and suffering, but I am saying that, as another comment said, "life goes on".
---
Put another way, if the Internet disappeared for a week, month, year, what would happen? It'd suck and we'd adjust, but we (humanity as a whole) would be fine.
I actually disagree; the centralization is a symptom of severe, systemic, unfixable flaws in the design of the internet itself. At least we didn't let AOL define internet.
2. Port scanning attacks
3. Geolocation attacks (you shouldn't run a server from your home, when your IP Address could be used to find you; be it from a hack or an abuse of the legal process)
4. The US internet relying on a backbone mostly operated by 4 companies (meaning a direct connection is a massive perk)
For as much as we moan about Amazon and Cloudflare centralizing things; believe me, just taking out Level 3 / Lumen would be almost an act of war and would affect every ISP and cloud provider.
If you took out Lumen, Verizon Enterprise Solutions, and AT&T, just those 3 companies; that's 2.35 million kilometers of fiber going dark. For reference, Comcast has "only" 150,000 km.
I tend to think centralization is more about people's tendencies than it is internet design. As I see it, centralization on the internet happens as the emergent behavior of all of the elements in the system and the forces that exist. In other words, it's the natural result of incentives.
People use centralized services because
- prices (economies of scale)
- network effects ("I can't certain people without WhatsApp")
- social pressures ("People look at me weird when I say I don't have an Instagram")
- visibility ("There are other CRMs than SalesForce?")
- making choices easy ("I don't want to choose between all these options. I'll pick the most familiar")
- safety ("Nobody got fired for choosing IBM")
- convenience ("It's a pain to self-host")
- security ("Google probably protects data better than I can")
- etc
Level 3 Communications / Lumen; AT&T; Verizon Enterprise.
With those three, you've knocked 2.35 million km, or 1.46 million miles, of fiber optic offline. That's over half of the US backbone in those three companies. Comcast "only" has 0.15 million miles for the sake of comparison. The internet would crash and burn immediately to speeds not seen since the dial-up era. There's nothing we can do about that. Cloudflare and AWS centralizing everything is actually, statistically, less risky than backbone attacks. If you succeeded in knocking down all of the big seven at once; you've got about 80-90% of the backbone offline plus all of the online cloud providers with direct connections to them; at which point the internet would cease to function in any practical sense.
Really the best innovation in the last 10 years has been the penchant for putting things on development budgets instead which loosened the purse strings considerably.
I wish I had sufficient data to back this up but for context; I was part of a small infrastructure team running a little over 1% of all web traffic at one point in a pair of colos (with DDoS protection) for less than £5MM on infra costs.
Now I can easily be in the 6 digits with just a CRUD app
Those are much more volatile (compute heavy and not enough compute to meet peaks), harder to setup/maintain and benefit greatly from sharing infrastructure for hardware utilization. Everything is moving in that direction.
It's readily evitable, just choose not to write the generic, repetitive thing you already know is generic and repetitive.
Grab lunch, come back, and it'll have fixed itself. Nobody blames you because it's just another provider outage -- whether it's Cloudflare or AWS or Microsoft or Gmail doesn't really matter, it's a convenient outsourcing of not just network admin but also blame.
The alternative is, what, doing it all in-house with a bunch of people on pager duty? Few companies can afford that. Diversifying across multiple cloud providers? Sure, if multi-nines uptime is really critical for your business.
For most small-med businesses, it's more valuable to be able to rapidly build on top of existing, professionally-maintained infrastructure in order to deliver some unique business value on top of it, than to recreate that basic network infra themselves. If you're not a FAANG, going down for a few hours a year isn't really a big deal. Especially when you can just blame it on an overall "internet outage at some big provider" rather than some internal fuck-up.
(Edit: Not to be snarky, but really, so what if that happens? Same thing happens if there's a power outage in your neighborhood. Is it a big deal?)
Could not find data how this is distributed..
Sorry man, our Stripe terminals aren't working, we can't even accept cash... Um, come back later.
Edit: Just to clarify, obviously this can't be both common and rare... what I meant is that different neighborhoods will frequently go down, but there's usually some neighborhood within a 10-min drive that still has power, or some food truck has a cellular connection off their iPad PoS that still works, or you happen to have cash in your wallet or whatever or know the guy and they'll just put it on your tab for next time, whatever. I live in a semi-rural community in Oregon and we have a lot of small businesses, a lot of wildfires, and not a whole lot of infrastructure. We just kinda make do when things happen.
A couple weeks ago, half the town lost power for the whole day, and the other half of the town just got some increased business and traffic. Life was back to normal the next day.
The centralization around AWS and GCP and Azure is more concerning to me than the centralization around Cloudflare. It’s not obvious to me why lots of people using Cloudflare is any worse than lots of people using AWS.
If we go even deeper, the entire internet runs on a limited number of undersea cables connecting continents. That in particular has always seemed like the most vulnerable aspect of the internet, considering a targeted attack on < 100 locations could meaningfully cut off internet access for millions (billions?) of people for an extended period of time.
How does this work when most edge providers manage your DNS to point at the available servers they have?
Also, if you "HAVE" to be online, then you're the type of target that's at big risk of DDOS which complicates this even further.
"I can't be the only person who literally doesn't give a F?"
There is an option to use ngrok instead, but I have never seen such a setup for "local" development.
Cloudflare at least accurately reflects the incident on their status page.
Often dashboards will show everything green despite the service being down.
In these moments communication is critical and Cloudflare does it well especially as we all know when it comes to the post mortem. They are very transparent about these issues and incidents.
Looks like there were 2 outages[0], one in June and one in July of 2021.
[0] https://www.reuters.com/technology/websites-airlines-banks-t...
Your regular, run-of-the-mill outages aren't reported in Reuters... but are likely much more common than you'd believe.
It's also nuts that this downtime according to creation of the post -> resolved in comments, was only 27 minutes...
I've literally been on standby to get an answer what happened with a huge cloud provider for 3 weeks now and today they got back with static assets that weren't reachable ( ico's ???)
They don't even know what happened since it's not logged in their logging, since it's using middleware and because of the sudden weekend downtime, they middleware wasn't triggered...
After a detailed description that we found that it was related to a app service plan, they came back with 404's on static assets...
Only now it got escalated..
Fuck the cloud, outside of cloudflare.
The only time we had a downtime with cloudflare, they gave a detailed description and it was related to our cloud provider and not Cloudflare . Then their was a infra update and it also didn't get logged... Because of that we thought it was the wrong party...
18 hours down, redeployment fixed it, nothing logged on their end. Luckily one service wasn't redeployed ( barely used), so we can prove there was an issue.
Ho boy...
I'm wondering how many more weeks it will take...
I was affected by the tunnel outage. It was down for closer to 3 hours, not 27 minutes. (Their status page does accurately reflect this time)
I checked the author post. Normally it's pretty quick to post here as soon as something is "noticed".
Hah. Cloudflare's rise to fame was them proudly announcing how their servers were running but the users misconfigured their site, by "defacing" the users' non-resonse.
Cloudflare detects downtime of the underlying cloud/host.
You'll literally see a cloudflare error page and think it would be related to them, while it's actually the underlying provider.
https://hn.algolia.com/?q=cloudflare+down