Someone should a website that collates the approx. 90M times this sentiment has been made on this website (a good chunk of me making it) just a reminder how nothing somehow changes on this front the moment it comes back up and people go back to relying on single SAAS's for everything.
I remember when some key part of AWS EC2--EBS in us-east-1 maybe?--was down for a few days straight. Honestly, the main thing it taught me was "if you are honest with your customers they will mostly just come back later and buy everything they didn't buy today".
To be fair this is not relying on a single SaaS for everything but many people relying on a single SaaS. I mean if you want to use a reverse proxy/CDN, you must rely on someone.
Our key customer facing services are a 99.995% uptime (and a total of 2 or fewer incidents per year of "any length"), which means once you start concatenating services with 99.995% SLAs you aren't there.
How that SLA measures a 2 second outage for some customers is a separate thing, and sort of shows how meaningless these things can be on the internet (if you lose service for 10% of your potential customers is that an outage? How about 90%? How do you know how many were lost).
Measuring outages doesn't seem so meaningless as long as you money seems inaccessible.
Their main site went down for about 20 hours a couple weeks ago because their hosting provider went down. They deployed an HTTPS only static site in its stead, so at first blush it looked like they deployed nothing. Great when you're trying to find contact information hosted on that site.
Their online banking site leveraged Cloudflare, so obviously they just rode that outage out with no notifications, etc.
What if for some reason a single /24 was unreachable from the site (say an errant route for 12.85.25.0/24 somehow got in the path). How would you even know that was a problem - how many customers are on that /24, how would I measure their failed attempts to connect?
I have a remote office in India on Tata. The other day it had access to much of the internet, but due to a fibre break in the Mederteranian it didn't have access to end points in Europe for a good 20 seconds.
However the other link on a different ISP remained working at that time.
Does that count as an outage? If I wasn't actively monitoring that link with a high resolution would I even know about it?
I'd argue you're starting from a few orders of magnitude more competency than the credit union was. Their non-banking site was hosted by some podunk company in Texas with no sense of redundancy anywhere. Their provider had a near total networking outage and the credit union had no plan to recover from that.
Insofar as proactively monitoring a single /24, you (probably) don't. I don't think it's (usually) a company's job to monitor their customer's ISPs. The failures that "my" credit union had were due to their own choice in infra (Armor, Cloudflare). When Sonic nuked my config on their DSLAM after some maintenance I raised an issue with Sonic not with whatever other companies became inaccessible as a result.
> Does that count as an outage?
My POV may very well differ from whatever contracts and SLAs you have in place, but yeah maybe. If you can't fail over to the alternative ISP then yes that's an outage. Of course a trans-atlantic fiber break would also likely be a lot more noticeable than fat fingering a route for a /24. And sure, I've been stuck at megacorp when the VPN started handing out addresses in a new subnet but our department's networking team hadn't caught up. That's why you listen to your customers instead of throwing out a "someone else screwed up there's nothing we can do" response.
Me personally I don't think that a 20 minute banking outage is a massive problem (I've long since moved my money elsewhere), even the 20 hour outage was relatively minor. It just speaks to the unwillingness of the credit union to be highly available. They knew of the Armor outage and didn't actually test the remediation. I assume they didn't know about the Cloudflare outage. Both worry me. What happens when they're faced with a total failure of their online banking system?
But it isn't an outage. My monitoring point in Singapore could reach both ends, they just couldn't talk to each over, due to a routing issue on a third party network over the internet.
On my own network which I control I accept that if a circuit breaks I'll have a 1, maybe 2 second outage while traffic reroutes. For some of my services that's would be a problem, for others it's not. If facebook loads 2 seconds later, nobody cares. If the winning penalty in the world cup final blacks out, that's a big problem.
I'm new to this whole thing. Can u point me on how I don't depend 100% on CF, if its DNS is down? Is there such a service? (kinda like load balancing, but with DNS?)
DNS is easy. You can (and must) have multiple nameservers for your domain. Just use different companies (and different regions) and if one goes down the others will still resolve.
I wonder if it really was though. I’d think that these centralised services go down less than the self hosted stuff previously. Is it better to have more overall uptime but downtime means everything stops, or random downtimes of individual sites that adds up to more downtime.
I mean if large websites like Notion or Medium had used IPFS instead, there would be no central point of failure, and web pages would still be available from distributed hosts
Yes and no. Obviously not great when everything goes down, but I find a strange sense of solace and calm when I know there’s a lot of people in the same boat and there’s little I can do but wait.
Yeah same lol. I saw that my SaaS services were not working and I got stressed thinking why all my EC2 instances went down at the same time. I checked down detector for EC2 and it reported that Cloudflare is down. I breathed a sigh of relief thinking that the (almost) whole of internet is down - nothing that I can do here.
It's quite ironic that the Internet was designated to withstand nuclear attack, yet with how much everyone started using "cloud" a stupid configuration mistake in an important company can put it on its knees.
We should really rethink that constant reliance on single point of failure.
This one seems be due to a hug of death rather than isitdownrightnow.com being behind Cloudflare, probably from too many people checking on all the other sites that are down.
I say this because: (1) it eventually loaded for me after I tried a few times and gave it time to load, and (2) its certificate doesn't report to be from Cloudflare but other sites I've checked that are down do
> Their status page is a joke, likely crippled to reduce legal liability, but at this point it's just an outright misrepresentation.
It's just Atlassian Statuspage, which is a manually-updated incident response system. Unlike AWS, Cloudflare actually makes an effort to update it fairly quickly, but it can still be slow-to-update when something is immediately wrong.
> Their status page is a joke, likely crippled to reduce legal liability, but at this point it's just an outright misrepresentation.
It's fairly standard practice these days for status pages to be manually updated. The difficulty with having them be automatically updated is that for it to be useful that system needs to have a greater reliability than the thing it's monitoring. The signal to noise ratio is otherwise a bit ridiculous.
Reddit Status [1] isn't perfect, but it's miles better than a static page saying everything is operational while everything is in-fact inaccessible. That it took 30 minutes for the page saying everything is fine to be updated with a warning that there is an issue (while almost all of the services and regions remained marked as operational) only makes the ineffectiveness of that page more blatant.
It goes without saying that the monitoring system must be separate from what it's monitoring and must be more reliable. Compared to running a CDN for half of the internet, automated monitoring is table-stakes.
Maybe they are more decentralized than what we are giving them credit. I'm having different error messages (nginx, dns, 404) on different websites. Not sure if it's a full breakdown of their systems or a coordinated attack.
Several sites I was trying to access all went down at the same time. Came to Hacker News to see what was up - not disappointed!
*Including America's Cardroom, perhaps the biggest "offshore" US poker site. I can promise you that there are a lot of people who were playing in tournaments that are very unhappy right now. New York here.
Yes, I started checking router and wondering if anyone had managed to install some sort of exploit on it as I was getting that 500 nginx page from half a dozen Web sites.
Explains why the online training course I was part way through stopped working! Amusing that the quickest diagnosis came from skimming the headlines here :)
CLoudflare just offer great services. Its straight up fact that even their free model is extremely generous. There is no big conspiracy to 'take over the internet' but when the product is good, the product is good.
Not a 'big conspiracy'. It's the business model, isn't it? Or isn't CF going for the biggest marketshare and maximizing profits on that, like all the others?
It's certainly any business' model to grow as big as possible, but it's a hard business model to implement so competition is hard to find. I just can't blame CF for that imo.
CloudFlare should be run by the CIA or something - asthonishing MITM opportunities. The only clear sign the CIA is not deeply involved is that CloudFlare is far too competent.
It blows my mind how most of the otherwise savvy readers of HN completely gloss over the fact that Cloudflare unwraps TLS on most their internet traffic.
I trust that the current leadership might not do something evil, but they are publicly traded. At some point a group of investors are going to figure out that merging Cloudflare with an advertising network would create a level of user targeting that Google and Facebook could never dream of.
Governments in Europe and elsewhere are already working on legislation to mitigate e2e encryption by law. Regulating things like cloudflare as they have already done with ISPs to hand over data is not even much of an imagination leap. For example in the UK all time:srcip:destip:user data must be kept for 1 year for every residential ISP and provided to government departments (not even law enforcement) through a standard system
444 comments
[ 2.4 ms ] story [ 64.3 ms ] threadHow that SLA measures a 2 second outage for some customers is a separate thing, and sort of shows how meaningless these things can be on the internet (if you lose service for 10% of your potential customers is that an outage? How about 90%? How do you know how many were lost).
Their main site went down for about 20 hours a couple weeks ago because their hosting provider went down. They deployed an HTTPS only static site in its stead, so at first blush it looked like they deployed nothing. Great when you're trying to find contact information hosted on that site.
Their online banking site leveraged Cloudflare, so obviously they just rode that outage out with no notifications, etc.
What if for some reason a single /24 was unreachable from the site (say an errant route for 12.85.25.0/24 somehow got in the path). How would you even know that was a problem - how many customers are on that /24, how would I measure their failed attempts to connect?
I have a remote office in India on Tata. The other day it had access to much of the internet, but due to a fibre break in the Mederteranian it didn't have access to end points in Europe for a good 20 seconds.
However the other link on a different ISP remained working at that time.
Does that count as an outage? If I wasn't actively monitoring that link with a high resolution would I even know about it?
Insofar as proactively monitoring a single /24, you (probably) don't. I don't think it's (usually) a company's job to monitor their customer's ISPs. The failures that "my" credit union had were due to their own choice in infra (Armor, Cloudflare). When Sonic nuked my config on their DSLAM after some maintenance I raised an issue with Sonic not with whatever other companies became inaccessible as a result.
> Does that count as an outage?
My POV may very well differ from whatever contracts and SLAs you have in place, but yeah maybe. If you can't fail over to the alternative ISP then yes that's an outage. Of course a trans-atlantic fiber break would also likely be a lot more noticeable than fat fingering a route for a /24. And sure, I've been stuck at megacorp when the VPN started handing out addresses in a new subnet but our department's networking team hadn't caught up. That's why you listen to your customers instead of throwing out a "someone else screwed up there's nothing we can do" response.
Me personally I don't think that a 20 minute banking outage is a massive problem (I've long since moved my money elsewhere), even the 20 hour outage was relatively minor. It just speaks to the unwillingness of the credit union to be highly available. They knew of the Armor outage and didn't actually test the remediation. I assume they didn't know about the Cloudflare outage. Both worry me. What happens when they're faced with a total failure of their online banking system?
On my own network which I control I accept that if a circuit breaks I'll have a 1, maybe 2 second outage while traffic reroutes. For some of my services that's would be a problem, for others it's not. If facebook loads 2 seconds later, nobody cares. If the winning penalty in the world cup final blacks out, that's a big problem.
The OP says "not Cloudflare"; so probably Akamai, Fastly, CloudFront?
Sure, there are alternatives and not everything uses it, but if it's enough to greatly affect a large proportion of internet users, it's a problem.
Just like if google mail went away for ever. There are plenty of other email providers, right ?
Fun times.
We should really rethink that constant reliance on single point of failure.
I say this because: (1) it eventually loaded for me after I tried a few times and gave it time to load, and (2) its certificate doesn't report to be from Cloudflare but other sites I've checked that are down do
Downdetector: ಠ_ಠ Well, this is awkward...
5XX Server Error Web server is returning an unknown error
There is an unknown connection issue between Cloudflare and the origin web server. As a result, the web page can not be displayed.
[0] https://www.cloudflarestatus.com/
(Also it doesn't help that uptime monitoring systems are usually stupid and love triggering on false positives)
Their status page is a joke, likely crippled to reduce legal liability, but at this point it's just an outright misrepresentation.
It's just Atlassian Statuspage, which is a manually-updated incident response system. Unlike AWS, Cloudflare actually makes an effort to update it fairly quickly, but it can still be slow-to-update when something is immediately wrong.
For their status page to be broken down into individual services and regions, I get the impression of some kind of automated monitoring.
The only service I saw get marked non-operational was their API, while their site and dashboard were not available at all yet marked as operational.
It's fairly standard practice these days for status pages to be manually updated. The difficulty with having them be automatically updated is that for it to be useful that system needs to have a greater reliability than the thing it's monitoring. The signal to noise ratio is otherwise a bit ridiculous.
It goes without saying that the monitoring system must be separate from what it's monitoring and must be more reliable. Compared to running a CDN for half of the internet, automated monitoring is table-stakes.
[1] https://www.redditstatus.com/
Edit: at least it showed there was a problem within <10 minutes, unlike other status pages that sometimes are green the entire time.
Cloudflare.com now up, and websites are coming up, argo tunnels still down
*Including America's Cardroom, perhaps the biggest "offshore" US poker site. I can promise you that there are a lot of people who were playing in tournaments that are very unhappy right now. New York here.
In UK, good old bbc.co.uk still working fine...
Turns out no one will notice, as all of Shopify is down anyway ¯\_(ツ)_/¯
Of course they offer a great product, that's how you create a monopoly.
CloudFlare should be run by the CIA or something - asthonishing MITM opportunities. The only clear sign the CIA is not deeply involved is that CloudFlare is far too competent.
I trust that the current leadership might not do something evil, but they are publicly traded. At some point a group of investors are going to figure out that merging Cloudflare with an advertising network would create a level of user targeting that Google and Facebook could never dream of.
Subdomains aren't working though (e.g., https://sketch.nono.ma).
Update: It seems everything is resolving properly now.