328 comments

[ 1.6 ms ] story [ 276 ms ] thread
So it seems... Even https://cloudflare.com/ itself is down
So cloudflare.com runs on Cloudflare?
Turtles all the way down.

But in all frankness, if Cloudflare's own site did not run off of Cloudflare's infrastructure, why would anybody trust them with their websites?

Certainly what I would use if I was building cloudflare.com...
For the site itself that's totally fine, and the status page is separate as it should be, too.
Fair. Some service recently had their status page hosted on the same server that went down, that was funny.
Discord's status page is also down haha
I'm quite surprised Discord's status page is behind Cloudflare. I thought they were using statuspage.io.
They are, but that subdomain is proxied through Cloudflare. If they'd set it to just DNS then it would have still worked.
I guess it's like diversifying one's assets - you put your status page elsewhere than on your own infrastructure.
This is fine-ish...

In case of network issues affecting their proxy, being able to change your configuration to allow direct traffic would be really nice.

(comment deleted)
Yep, I'm seeing 502's all over the web as of a few minutes ago.
Great, now we can't even disable HTTP proxy to allow traffic directly to AWS :/
Point your DNS back to your actual host? Might be the only short term solution, though DNS propagation times kinda rule that out :/
That's a no-go if you transferred your domains to Cloudflare Registrar.
Something something all your eggs in one basket :)
Seems like even other registrars might rely on Cloudflare (e.g. Namecheap) so now people have to continuously ensure there’s no cross-pollination between their infra providers...
I think the only option here would be to change our name servers at registrar level to point to AWS and recreate all DNS records there, but then you have to deal with name server propagation.
More importantly, their admin dashboard is down. It's impossible to bypass their "orange cloud" proxies and send traffic directly to our hosting. That they can't flip a switch and have their nameservers send dash.cloudflare.com to a separate piece of redundant infrastructure is mind-boggling.
Agreed. First thing I tried to do was login Cloudflare to disable proxying. But can't.
change the nameserver at the registrar to someone else
By the time that change propagates, cloudflare will be backup
Yes, but then next time you will be able to control your DNS.
Though, only if you're using a short TTL. I'm not arguing against your position in general though.
We were able to flip this switch on our services through their API, as we have our Cloudflare config in Terraform.

The API wasn't working perfectly, but with some retries we were able to change the config for our domains.

EDIT: the static pages load but trying to log in just times out. I think the static cache is back up but the rest of it is still down.

Seems to be back up and running for me.

Wow... our site went down so I went to cloudflare.com -> 502 bad gateway
Either downforeveryoneorjustme.com is itself served by Cloudflare too, or has been hugged to death.
It returns 502 Bad Gateway cloudflare so I assume it's not the HN hug...
Yep, we are 100% on CF workers :)

Here is a quick image of the peak downtime on downforeveryoneorjustme.com:

https://ibb.co/PZ9BMRc

Thanks for the fantastic work on the service. And thanks for sharing that stats!
Single point of failure - we all shouldn’t trust CF alone anymore...
(comment deleted)
Any CDN is a single point of failure and limits your availability to as low as three nines. Although anycast-based CDNs like Cloudflare are much less reliable than DNS-based CDNs, those can do orders of magnitude better.
Why do you think anycast based CDNs are worse than DNS based ones?
They rely on a single network infrastructure as opposed to many independent networks with independent edge nodes where isolating faults is rather trivial in comparison.
I'm getting '502 Bad Gateway' on all Cloudflare sites here in London too.
Also getting this in the UK. Not completely down, bits and pieces of medium comes through but very slowly and incomplete. I’m also unable to access npm.
Holy shit. This impacts nearly everything.
Yes - seeing lots of our sites offline as of 14:45 GMT.
(comment deleted)
Several sites behind CloudFlare are returning 502 errors for me as well (France).
(comment deleted)
omg I was not the only one.
1.1.1.1 DNS is down, and seeing a lot of 502s on cloudflare sites.
That's a biggie...
I just changed my dns-over-https script to use quad9:

    # sudo /usr/local/bin/dnsproxy -l 127.0.0.153 -p 53 -u https://cloudflare-dns.com/dns-query -b 1.1.1.1:53
    sudo /usr/local/bin/dnsproxy -l 127.0.0.153 -p 53 -u https://9.9.9.9/dns-query
The bootstrap 1.1.1.1:53 was working but the DoH dns-query url was not. Hence my visit here :)

Edit: Now fixed, it seems. Quick work, if it stays up!

digitalocean.com is down as well (seems to affect cloudflare-gated websites)
Looks like it, one of my sites is cached through Cloudflare, I'm getting a 502 from them. I'm seeing other random sites down as well.
So what should the ideal redundancy plan be here? If you can't log into the CDN provider and they are down do you you just have a second one ready (and paid for) and then log into your registrar and be ready to switch to that secondary CDN provider in this scenario? Or is there some sort of load balancing / routing solution between CDN's that I don't know about / understand?
If you use Cloudflare nameservers, you have to change to new nameservers, wait for that to propogate, and then wait for clients cached records TTLs to expire. So it will be a major disruption no matter what you do.
What about the API? If that is up, you could script it.
If you're using them for TLS certs then it's an even bigger problem unless you have them provisioned elsewhere.
Unless you need EV you can just pull some wildcards from Lets Encrpt (as long as you don't use pubkey pinning). No need to automate as it's just a one off.
Can you use your own nameservers that delegate to cloudflare or akami 50% each, then adjust? Is there a service more suited for this than r53?
(comment deleted)
Noticed that draw.io were down. 1.1.1.1 and all other Cloudflare backed sites as well... I'm off for today.
Cleartext DNS 1.1.1.1 is still up though, thankfully.
What will we do without our favourite "Enterprise MiTM" SaaS provider?
Cloudflare DNS 1.1.1.1 down too
Good thing I switched to OpenDNS literally two days ago.

edit. not that it matters, I'm just scrolling 9gag.

And so it begins......