Lol, talk about timing. I'm currently working on a TLS library and was pulling my hair out trying to figure out why tests against CF sites suddenly failed. Can't even ask my cohorts on Discord because they are behind CF, too!
No kidding! We had literally deployed a major page redesign and started watching our analytics drop off on it's way to zero. My heart is racing still. I wouldn't normally be happy for a cloudflare outage but in this case it's better than Google deciding to remove us from their index.
"StackOverflow devs have the most difficult job in the world. After all, when StackOverflow is down, they can't exactly look for help on StackOverflow".
Good DNS practice (at least when I did system admin 10 years ago) was ALWAYS having a secondary at some other location/network. Why do we just put some info in Cloudflare and call it good these days?
It's hard to use Cloudflare as a reverse proxy without using them as your delegated name servers (maybe you can use CNAMEs on paid plans?), and fancy dynamic nameservers make it hard to run secondary servers with zone transfers.
Discord is entirely down right now, both the website and the app itself. Amusingly, a lot of the sites that normally track outages are also down, which made me think it was my internet at first. Downdetector, monitortheinternet, etc.
Lots of other big sites that are down: Patreon, npmjs, DigitalOcean, Coinbase, Zendesk, Medium, GitLab (502), Fiverr, Upwork, Udemy
Edit: 15 min later, looks like things are starting to come back up
We're dealing with a deeper level problem here. Since a lot of the internet is relying on Cloudflare DNS at some part or another, even many backup solutions fail.
Since so much of DNS is centralised in so few services, such outages hit the core infrastructure of the internet.
A sudden disruption on a large number of services for everybody at once doesn't look like a DNS problem to me, with all the caching in DNS. It would fail progressively and inconsistently.
DNS absolutely was an issue. I changed DNS manually from Cloudflare's 1.0.0.1 (which is what DHCP was giving me) to 8.8.8.8 (Google) and most things I'm trying to reach work. There may be other failures going on as well, but 1.0.0.1 was completely unreachable.
> I don't use cloudflare DNS but google DNS and got the same problems thant everyone else
Cloudflare is also the authoritative DNS server for many services. If Cloudflare is down, then for those services Google's DNS has nowhere to get the authoritative answers from.
No, I changed the setting back and forth while it was down to confirm that the issue was that I could not reach 1.0.0.1. All the entries I tried from my host file were responsive (which is how I ruled out an issue with my local equipment initially and confirmed that it wasn't a complete failure upstream -- I could still reach my servers). Changing 1.0.0.1 to 8.8.8.8 allowed me to reach websites like Google and HN, and changing back to default DNS settings (which reset to 1.0.0.1, confirmed in Wireshark) resulted in the immediate return of failures. 1.0.0.1 was not responsive to anything else I tried.
Again, it may not have been the only issue -- and there are a number of possible reasons why 1.0.0.1 wasn't reachable -- but it certainly was an issue.
Only if you use their "infinitely scaling" services. eg. s3. If the attacker is hammering you with expensive queries and your database is on 1 ec2 server, you're still going to go down.
Don't think so. They used to use Cloudflare but stopped. To my knowledge, it's a single server without a database (using the filesystem as a database).
>Holy crap I am thinking either there is some magic or everything we are doing in the modern web are wrong.
Spin up an apache installation and see how many requests you can serve per second if you're just serving static files off of an SSD. It's a lot.
edit: I see that there are already a bunch of other comments to this effect. I think you're comment is really going to bring out the old timers, haha. From my perspective, the "modern web" is absolutely insane.
>I think you're comment is really going to bring out the old timers, haha.
That is great ! :D
>It's a lot.
Well yes, but HN isn't really static though. Fairly Dynamics with Huge number of users and comments. But still, I think I need to rethink lots of assumption in terms of speed, scale and complexity.
There is some caching somewhere as well, probably provides a bit more boost.
I've been at my work laptop (not logged in) and found something I wanted to reply to, so I pulled out my phone and did so. For a good 10 seconds afterwards, I could refresh my phone and see my comment, but refresh the laptop and not see it.
Huge numbers of users don't really mean that much. Bandwidth is the main cost, but that's kept low by having a simple design.
Serving the same content several times in a row requires very few resources - remember, reads far outnumber writes, so even dynamic comment pages will be served many times in between changes. 5.5 million page views a day is only 64 views a second, which isn't that hard to serve.
As for the writes, as long as significant serialization is avoided, it is a non-issue.
(The vast majority of websites could easily be designed to be as efficient.)
>* From my perspective, the "modern web" is absolutely insane. *
Agreed.
I was brought up as a computer systems engineer... So, not a scientist, but I always worked with the basic premise of keep it simple. I've worked on projects where we built all the fangled clustering and master/slave (sorry to the PC crowd, but that's what it was called) stuff but never once needed it in practice. Our stuff could easily handle saturated gigabit networks as the 2 core cpu only running at 40%. We had cpu spare and could always add more network cards before we needed to split the server. It was less maintenance, for sure.
It also had self healing so that some packets could be dropped if the client config allowed it, if the server decided it wanted to (but only ever did on the odd dodgey client connection)
That said, I was always impressed by the map-reduce of for search results (yes, I know they've moved on) which showed how massive systems can be fast too. It seemed that the rest of the world wanted to become like Google, and the complexity grew for the std software shop, when it didn't need to imho.
I jumped ship at that point and went embedded, which was a whole lot more fun for me.
There is only so much i can do - i'm not american and thats a change i can't make on my own aside from doing my best to be an ally when possible.
However i can open a few PRs and use some of my time to make that change. It's a minor inconvenience to me and if it makes even one black person feel heard and supported then yea, i'm gonna do it.
> From my perspective, the "modern web" is absolutely insane.
You know, it should be even better than it was in the past, because a lot of heavy lifting is now done on the client. If we properly optimized our stuff, we could potentially request tiny pieces of information from servers, as opposed to rendering the whole thing.
Kinda like native apps can do(if the backend protocols are not too bloated)
We were serving around that traffic off a single dual pentium 3 in 2002 quite happily off IIS/SQL Server/ASP. The amount of information presented has not grown either.
That little box had some top tier brand main corporate web sites on it too and was pumping out 30-40 requests a second peak. There was no CDN.
You were not serving that traffic, you were just serving your core functionality - no tracking, no analytics, no ads, no a/b, no dark mode, no social login, no responsiveness. Are most of those shitty? Sure, just let me know when you figure out how to pry a penny from your users for all your hard work.
With DO, these days, they don't run me out of bandwidth, but my instance falls over (depending on what I am doing - which ain't much), but with AWS, they auto-scale and I get a $5000 bill at the end of the month. I prefer the former.
Yeah, the bandwidth overage was a grand, give or take. It was a valuable lesson in a number of ways, and why for any personal things I wouldn't touch AWS with a shitty stick.
Unlike Reddit, logged in and logged out users largely see the same thing. I wouldn't imagine there is much logic involved in serving personalized pages, when they don't care who you are.
The username is embedded in the page, so you can't do full caching unfortunately. But the whole content could be easily cached and concatenated to header/footer.
That could be split as a setting in the user-specific header with the visibility part handled client-side. A bit more work, but it's not impossible if it's worth it.
That's only ~60 QPS, assume it is peaky and hits something more like 1000 QPS in peak minutes, but also assume most of the hits are the front page which contains so little information it would fit literally in the registers of a modern x86-64 CPU.
Even a heavyweight and badly written web server can hit 100 QPS per core, and cores are a dime a dozen these days, and storage devices that can hit a million ops per second don't cost anything anymore, either.
1M queries per day is ~10 queries per second. It's a useful conversion rate to keep in mind when you see anyone brag about millions of requests per day.
It is certainly doable , PoF was running lot of page views famously of a single IIS server for a long time .
HN is written in a lisp variant and most the stack is built in-house , it is not difficult to imagine efficiency improvements when many abstraction layers have been removed from your stack .
I don't remember PoF being famous for that, but they got a lot of bang for the buck on their serving costs.
What I do remember, is that it was a social data collection experiment for a couple of social scientists, that never originally expected that many people would actually find ways to find each other and hook up using it.
I miss their old findings reports about how weird humans are and what they lie about. Now, it's just plain boring with no insights released to the public.
For all my sibling comments, there is also context to be aware of. 5.5m page views daily can come is many shapes and sizes. Yes, modern web dev is a mess, but situation is very different from site to site. This should be taken as a nice anecdote, not as a benchmark.
It's amazing how little CPU power it takes to run a website when you're not trying to run every analytics engine in the world and a page load only asks for 5 files (3 of which will likely already be cached by the client) that total less than 1 MB.
That means an average of about 63 pages per second. Let's say that the total number of queries is tenfold and take a worst case scenario and round up to 1000 queries per second and then multiply by ten to get 10k queries per second, because why not.
I don't know what the server's specs are but I'm sure it must be quite beefy and have quite a few cores, so let's say that it runs about 10 billions instructions per second. That means a budget of about one million instructions per page load in this pessimistic estimate.
The original PlayStation's CPU ran at 33MHz and most games ran at 30fps, so about 1million cycles per fully rendered frame. The CPU was also MIPS and had 4KiB of cache, so it did a lot less with a single cycle than a modern server would. Meanwhile the HN servers has the same instruction budget to generate some HTML (most of which can be cached) and send it to the client.
A middle of the line modern desktop CPU can nowadays emulate the entire PlayStation console on a single core in real time, CPU, GPU and everything else, without even breaking a sweat.
>Holy crap I am thinking either there is some magic or everything we are doing in the modern web are wrong.
> Holy crap I am thinking either there is some magic or everything we are doing in the modern web are wrong.
It doesn't need to be crazy.
A static site on DigitalOcean's $5 / month plan using nginx will happily serve that type of traffic.
The author of https://gorails.com hosts his entire Rails video platform app on a $20 / month DO server. The average CPU load is 2% and half the memory on the server is free.
The idea that you need some globally redundant Kubernetes cluster with auto fail-over capabilities seems to be popular but in practice it's totally not necessary in so many cases. This outage is also an unfortunate reminder that you can have the fanciest infrastructure set up ever and you're still going down due to DNS.
> The idea that you need some globally redundant Kubernetes cluster with auto fail-over capabilities seems to be popular but in practice it's totally not necessary in so many cases
True, but this is why it shouldn't be bashed either. When you need it, you need it (cue very complex enterprise applications with SLA requirements).
> True, but this is why it shouldn't be bashed either. When you need it, you need it (cue very complex enterprise applications with SLA requirements).
To support this, look at how many people criticize Kubernetes as being too focused on what huge companies need instead of what their small company needs. Kubernetes still has its place, but some peoples expectations may be misplaced.
For a side project, or anything low traffic with low reliability requirements, a simple VPS or share hosting suffices. Wordpress and PHP are still massively popular despite React and Node.js existing. Someone who runs a site off of shared hosting with Wordpress can have a very different vision about what their business/sideproject/etc will accomplish compared to someone who writes a custom application with a "modern" stack.
That's why we have this bloated over-engineered multi-node applications: People just underestimate how fast modern computing is. To serve ~2^6 requests/sec is trivial.
I don't get this comment, what does page serving performance have to do with "the modern web"? It's not as if serving up a js payload would make it more difficult to host millions of requests on a single machine, html and js are both just text.
Makes sense based on what I've read about Arc, which HN is written in.
I've been working on something where the DB is also part of the application layer. The performance you can get on one machine is insane, since you spend minimal time on marshalling structures and moving things around.
This was a BGP/routing issue and has already been documented. Please don't spread misinformation and hysteria, especially on technical issues like this
CEO's are notorious liars. I just had a discussion with John McAfee who said that all good CEO's are feeding misinformation to the press as standard business practice.
The willingness that you believe a CEO who has every incentive to lie to the press, and the preposterous narrative that this was a simple "router issue" - as if half the USA internet system is dependent on a router not failing and there is not redundancy, is simply baffling.
I don't care how many downvotes I get. I'm not buying this flimsy narrative. And neither should you.
It was a misconfiguration we applied to a router in Atlanta during routine maintenance. That caused bad routes on our private backbone. As a result, traffic from any locations connected to the backbone got routed to Atlanta. It resulted in about 50% of traffic to our network to not resolve for about 20 minutes. Locations not connected to our backbone were not impacted. It was a human error. It was not an attack. It was not a failure or bug in the router. We're adding mitigations to our backbone network as we speak to ensure that a mistake like this can't have broad impacts in the future. We'll have a blog post with a full explanation up in the next hour or so — being written now by our CTO.
I host a dedicated server there (running https://www.circuitlab.com/) and when I traceroute/ping news.ycombinator.com, it's two hops (and 0.175 ms) away :)
Discord attempted to route me to: everydayconsumers.com/displaydirect2?tp1=b49ed5eb-cc44-427d-8d30-b279c92b00bb&kw=attorney&tg1=12570&tg2=216899.marlborotech.com_47.36.66.228&tg3=fbK3As-awso
It looks like you mistyped it and landed on a domain with spammy redirects. They have all kinds of weird URLs and there's not always any connection to anything you did other than go to the wrong domain.
I wonder if they are all using Cloudflare's free DNS stuff or if they're paying for business accounts?
My stuff is on Netlify (for the next week or so) and the rest is on a VPS bought from a local business who isn't reselling cloud resources. I'm kinda glad I moved all my stuff from cloudflare.
I did a pickup order a few weeks ago when of all things Tmobile SMS went down for 3+ hours. I couldn't go in the restaurant (covid) and I couldn't text them the parking # I was sitting at in a packed parking lot. I got a flood of about 50 texts a few hours later. Sat there for about an hour waiting for a $9 sandwich. I have no idea if they didn't get my order until late, or if they finally realized it was me or what. About 45 minutes in I decided to just give up on the day and take a nap, woke up to a door knock.
My iPhone actually popped up a message saying that my wifi didn't appear to have internet, which was strange and obviously false as I was actively using the internet on it and the laptop next to it, but now it makes sense that it must have been pinging something backed by cloudflare!
I can live without creepy instant messengers, but its shocking just how much everything else relies on one, central system. And furthermore, why is it always cloudflair?
idk about the UI the redesign with the forced collapses and extra clicks everywhere is annoying when handling a multitude sub domains plus their let's encrypt text entries I'm there mostly for the freebies.
Kudos to the people at Discord. Just a few minutes after I got disconnected they already tweeted about the issue. Some minutes later and they have a message in their desktop app confirming it's an issue with Cloudflare. All while Cloudflare's statuspage says there are 'minor outages'.
As a percentage of total traffic, a 'minor' outage for Cloudflare probably equates to a significant outage for a non-trivial amount of the internet.
It will also be especially noticeable to end-users, because sites using Cloudflare are typically high-traffic sites, and so a 'minor' issue that affects only a handful of sites is still going to be noticed by millions of people.
Freenode's IRC servers were down which was unexpected for me. I was expecting old-school communication networks to not have a dependency on Cloudflare.
Crazily, my local name resolution started failing, because I have these names servers: 192.168.0.99, 1.1.1.1 and 8.8.8.8. The first does the local resolution, but macOS wasn't consulting it because 1.1.1.1 was failing?? Crazy. When I removed 1.1.1.1 from the list, everything started working.
Thought something like this was going on. At first I thought it was my router and restarted everything - to no avail. Glad to see confirmation that it wasn't an issue on my end.
The point of the status page is so you can point to it for your five nines SLA and go "look? we were only down for one hour". As soon as the money relies on the metric, the metric will reflect the money.
Status pages are a marketing channel not a channel for developers most of the time. It most likely has to go through some layers before someone updates the status page.
I mean, I guess your entitled to look at it that way, but I don't think it's dishonest of them to distinguish between "nothing is working" and "some regions aren't working".
Despite their update, I like how they're saying only their recursive DNS had "degraded performance", while authoritative is "operational". The entire reason everything blew up was because their authoritative nameservers weren't responding.
Ahh I remember when AWS went down (think it was 2 years ago now?) or at least a data center in us-east? Majority of the internet went down and status page went down as well. Man good times.
That page still shows "Cloudflare System Status: All Systems Operational" for me, but it's definitely down for me. Along with 1.1.1.1, which is... bad.
Same. Even then, a bunch of sites are down. Maybe only ones behind Cloudflare? So far I've been trying to hit the various down detector sites and none of them will load. Google, Reddit, Hackernews are all fine.
Same. Had to go through the entire troubleshooting process --- is it my internet connection? my DNS resolver? firewall? ISP starting to filter DoT queries somehow?
Only last in my mental list was the possibility that Cloudflare would be down.
Hope they publish a detailed post-mortem. It's always fun to read (but certainly very painful for those directly involved in writing it).
495 comments
[ 3.1 ms ] story [ 255 ms ] threadLots of other big sites that are down: Patreon, npmjs, DigitalOcean, Coinbase, Zendesk, Medium, GitLab (502), Fiverr, Upwork, Udemy
Edit: 15 min later, looks like things are starting to come back up
The problem seems to have been resolved now, you might have made the change when they fixed it.
Cloudflare is also the authoritative DNS server for many services. If Cloudflare is down, then for those services Google's DNS has nowhere to get the authoritative answers from.
Again, it may not have been the only issue -- and there are a number of possible reasons why 1.0.0.1 wasn't reachable -- but it certainly was an issue.
Their IP belongs to AS21581; which is registered with a company called 'M5 Computer hosting' out of west-coast USA.
m5hosting.com
The last hop is Santa Barbera.
Definitely does not fall in the AWS ranges.
Holy crap I am thinking either there is some magic or everything we are doing in the modern web are wrong.
Edit: The number is from Dang [1]
>These days around 5.5M page views daily and something like 5M unique readers a month, depending on how you try to count them.
[1] https://news.ycombinator.com/item?id=23808787
Just need good stable code and server side caching.
Spin up an apache installation and see how many requests you can serve per second if you're just serving static files off of an SSD. It's a lot.
edit: I see that there are already a bunch of other comments to this effect. I think you're comment is really going to bring out the old timers, haha. From my perspective, the "modern web" is absolutely insane.
That is great ! :D
>It's a lot.
Well yes, but HN isn't really static though. Fairly Dynamics with Huge number of users and comments. But still, I think I need to rethink lots of assumption in terms of speed, scale and complexity.
I've been at my work laptop (not logged in) and found something I wanted to reply to, so I pulled out my phone and did so. For a good 10 seconds afterwards, I could refresh my phone and see my comment, but refresh the laptop and not see it.
Serving the same content several times in a row requires very few resources - remember, reads far outnumber writes, so even dynamic comment pages will be served many times in between changes. 5.5 million page views a day is only 64 views a second, which isn't that hard to serve.
As for the writes, as long as significant serialization is avoided, it is a non-issue.
(The vast majority of websites could easily be designed to be as efficient.)
Agreed.
I was brought up as a computer systems engineer... So, not a scientist, but I always worked with the basic premise of keep it simple. I've worked on projects where we built all the fangled clustering and master/slave (sorry to the PC crowd, but that's what it was called) stuff but never once needed it in practice. Our stuff could easily handle saturated gigabit networks as the 2 core cpu only running at 40%. We had cpu spare and could always add more network cards before we needed to split the server. It was less maintenance, for sure. It also had self healing so that some packets could be dropped if the client config allowed it, if the server decided it wanted to (but only ever did on the odd dodgey client connection)
That said, I was always impressed by the map-reduce of for search results (yes, I know they've moved on) which showed how massive systems can be fast too. It seemed that the rest of the world wanted to become like Google, and the complexity grew for the std software shop, when it didn't need to imho.
I jumped ship at that point and went embedded, which was a whole lot more fun for me.
Sincerely, old timer
That is what it _was_ called, it doesn’t have to be going forward.
The other way around surely doesn't work, and is just symbolic gestures without actual change.
However i can open a few PRs and use some of my time to make that change. It's a minor inconvenience to me and if it makes even one black person feel heard and supported then yea, i'm gonna do it.
You know, it should be even better than it was in the past, because a lot of heavy lifting is now done on the client. If we properly optimized our stuff, we could potentially request tiny pieces of information from servers, as opposed to rendering the whole thing.
Kinda like native apps can do(if the backend protocols are not too bloated)
We were serving around that traffic off a single dual pentium 3 in 2002 quite happily off IIS/SQL Server/ASP. The amount of information presented has not grown either.
That little box had some top tier brand main corporate web sites on it too and was pumping out 30-40 requests a second peak. There was no CDN.
Not a one trick marketoid pony.
90 to 99% of those are logged-out users, so fully cacheable.
Only a handful of dynamic requests each second remain.
It's ergonomic in a very lispy way but perfectly reasonably so from the POV of that aesthetic.
Even a heavyweight and badly written web server can hit 100 QPS per core, and cores are a dime a dozen these days, and storage devices that can hit a million ops per second don't cost anything anymore, either.
HN is written in a lisp variant and most the stack is built in-house , it is not difficult to imagine efficiency improvements when many abstraction layers have been removed from your stack .
What I do remember, is that it was a social data collection experiment for a couple of social scientists, that never originally expected that many people would actually find ways to find each other and hook up using it.
I miss their old findings reports about how weird humans are and what they lie about. Now, it's just plain boring with no insights released to the public.
Nick carver from SO also one mentioned that they could run SO if a single server , while it wasn’t fun it was doable and had happened some time .
I don't know what the server's specs are but I'm sure it must be quite beefy and have quite a few cores, so let's say that it runs about 10 billions instructions per second. That means a budget of about one million instructions per page load in this pessimistic estimate.
The original PlayStation's CPU ran at 33MHz and most games ran at 30fps, so about 1million cycles per fully rendered frame. The CPU was also MIPS and had 4KiB of cache, so it did a lot less with a single cycle than a modern server would. Meanwhile the HN servers has the same instruction budget to generate some HTML (most of which can be cached) and send it to the client.
A middle of the line modern desktop CPU can nowadays emulate the entire PlayStation console on a single core in real time, CPU, GPU and everything else, without even breaking a sweat.
>Holy crap I am thinking either there is some magic or everything we are doing in the modern web are wrong.
Magic, clearly.
It doesn't need to be crazy.
A static site on DigitalOcean's $5 / month plan using nginx will happily serve that type of traffic.
The author of https://gorails.com hosts his entire Rails video platform app on a $20 / month DO server. The average CPU load is 2% and half the memory on the server is free.
The idea that you need some globally redundant Kubernetes cluster with auto fail-over capabilities seems to be popular but in practice it's totally not necessary in so many cases. This outage is also an unfortunate reminder that you can have the fanciest infrastructure set up ever and you're still going down due to DNS.
True, but this is why it shouldn't be bashed either. When you need it, you need it (cue very complex enterprise applications with SLA requirements).
To support this, look at how many people criticize Kubernetes as being too focused on what huge companies need instead of what their small company needs. Kubernetes still has its place, but some peoples expectations may be misplaced.
For a side project, or anything low traffic with low reliability requirements, a simple VPS or share hosting suffices. Wordpress and PHP are still massively popular despite React and Node.js existing. Someone who runs a site off of shared hosting with Wordpress can have a very different vision about what their business/sideproject/etc will accomplish compared to someone who writes a custom application with a "modern" stack.
It's easily served by a simple server.
i used to host a wordpress site that has 5M pageviews a month on a $10 (and later $20) digitalocean instance.
that's wordpress and a shared vps. I imagine it could be a lot higher if I have dedicated server and use self-written software.
I've been working on something where the DB is also part of the application layer. The performance you can get on one machine is insane, since you spend minimal time on marshalling structures and moving things around.
They are still using Cloudflare. Unlike CF, M5 does not require SNI.
https://downdetector.com/
https://twitter.com/KEEMSTAR/status/1284240539437740033
Update: People are accusing me of using KEEMSTAR as a source and have somehow missed the link to downdetector.com, which keemstar is looking at.
https://twitter.com/eastdakota/status/1284253034596331520
You can see it here: https://www.youtube.com/watch?v=w_QWrzjKC9I&lc=UgyXzVtjVLbEs...
The willingness that you believe a CEO who has every incentive to lie to the press, and the preposterous narrative that this was a simple "router issue" - as if half the USA internet system is dependent on a router not failing and there is not redundancy, is simply baffling.
I don't care how many downvotes I get. I'm not buying this flimsy narrative. And neither should you.
This was a coordinated attack.
I host a dedicated server there (running https://www.circuitlab.com/) and when I traceroute/ping news.ycombinator.com, it's two hops (and 0.175 ms) away :)
I even checked to see if an AWS region was down once I realised it wasn't on my side (I thought it might have been my ISP's DNS servers or something).
The next move was to check Hacker News - thankfully it's not also hosted on Cloudflare, ha!
(Visit at your own risk.)
Hack?
If I type 'e' I get 'en.wikipedia.org'.
I was redirected.
That is why if you have this question, you should go to google.com
My guess is that there are more resources invested in making sure google.com stays up than for any other site on the internet.
My stuff is on Netlify (for the next week or so) and the rest is on a VPS bought from a local business who isn't reselling cloud resources. I'm kinda glad I moved all my stuff from cloudflare.
Ah, well. This too shall pass.
It will also be especially noticeable to end-users, because sites using Cloudflare are typically high-traffic sites, and so a 'minor' issue that affects only a handful of sites is still going to be noticed by millions of people.
[0] https://rachelbythebay.com/w/2019/07/15/giant/
I loathe Discord, and I can barely contain myself with schadenfreude at this news.
What's the point of a status page if it doesn't reflect the real status...
It's either the status page goes down with everything else or the status page is wrong. Great.
EDIT: Looks like it's accurate now, 20 minutes later.
[1]https://en.wikipedia.org/wiki/Goodhart%27s_law
Only last in my mental list was the possibility that Cloudflare would be down.
Hope they publish a detailed post-mortem. It's always fun to read (but certainly very painful for those directly involved in writing it).
I wonder if that includes the roots that Cloudflare operate.
Seems cloudflare took out a good chunk of the internet temporarily.
Doesn’t HN use cloudflare? Why did it survive? (I haven’t looked for about a year, but I seem to remember HN being proxied behind CF at one point.)
HN is so reliable that’s it’s almost never needed one. I’m extremely curious how HN survived this; almost positive they used cloudflare at one point.
I think the official status page is @hnstatus on Twitter, or something like that.
https://news.ycombinator.com/item?id=18188832
https://news.ycombinator.com/item?id=21799045
Looks like DNS issues, their nameservers aren't reachable.
DNS is completely broken.
"All systems operational" in nice soothing green.
No, not so much.