I'm having issues reaching IP addresses unrelated to Cloudflare. Based on some traceroutes, it seems AS174 (Cogent) and AS3356 (Level 3) are experiencing major outages.
Is there any one place that would be a good first place to go to check on outages like this?
It would be really cool and useful to have an "public Internet health monitoring center"... this could be a foundation that gets some financing from industry that maintains a global internet health monitoring infrastructure and a central site at which all the major players announce outages. It would be pretty cheap and have a high return on investment for everybody involved.
Indeed, if we're to have a public Internet health meter, it must be distributed and hosted/served from "outside" somehow, to be resilient to all or parts of the network being down.
This is an excellent idea and simple but moderately expensive for anyone to set up.
Just have a site fetch resources from every single hosting provider everywhere. A 1x1 image would be enough, but 1K/100K/1M sized files might also be useful (they could also be crafted images)
The first step would be making the HTML page itself redundant. Strict round robin DNS might work well for that.
But yeah, moderately expensive - and... thinking about it... it'll honestly come in handy once every ten years? :/
Reddit, HN, etc. are inaccessible to me over my Spectrum fiber connection, but working on AT&T 4G. It’s not DNS, so a tier 1 ISP routing issue seems to be the most likely cause.
> Fastly is observing increased errors and latency across multiple regions due to a common IP transit provider experiencing a widespread event. Fastly is actively working on re-routing traffic in affected regions.
It wasn't a total outage for the site I was trying to reach. It took about 20 minutes to make an order, but after multiple retries (errors were reported as a 522 with the problem being somewhere between Manchester, UK and the host), it did go through.
Fyi I'm not having any problems right now with hetzner.com nor hetzner.de - my own dedicated server hosted at Hetzner datacenter in Germany seems to be reachable/working as well.
I was doing development work which uses a server I've got hosted on digital ocean. I started getting intermittent responses which I thought weird as I hadn't changed anything on the server. I spent a good ten minutes trying to debug the issue before searching for something on duckduckgo, which also didn't respond. Cloudfare shouldn't be involved at all with my little site, so I don't think it's limited to just them.
Cogent and Cox are also having problems, but we are seeing a lot more successful traffic on Cogent than CenturyLink. It appears that CL is also not withdrawing stale routes. It seems CLs issues are causing issues on/with everything connected to it.
Yup, definitely noticed earlier outages to both EU sites and also to HN. Looked far upstream because many sites/lots of things worked fine. Good to see it's at least largely fixed
Same here. I actually opened a support ticket with them because I was worried my ISP had started blocking their IP addresses for some unknown reason. Luckily it seems to clear up, and in the ticket they mentioned routing traffic away from the problematic infrastructure. Seems to have worked for now for my things.
M5 Hosting here, where this site is hosted. We just shut down 2 sessions with Level3/CenturyLink because the sessions were flapping and we were not getting complete full route table from either session. There are definitely other issues going on on the Internet right now.
There is a major internet outage going on. I am using Scaleway they are also affected. According to Twitter, Vodafone, CityLink and many more are also affected.
I spent too much time losing precious time when github/npm/cloudflare are going down, until I figure out it was them.
So currently working on a project[1] to monitor all the 3rd party stack you use for your services. Hit me up if you want, access I'll give free access for a year+ to some folks to get feedbacks.
There is at least one big tool that does exactly the same you wrote. It is called StatusGator https://statusgator.com
There are at least 3 much smaller ones.
Have you tried any of them?
If yes, what's your point of difference?
And how do you plan to market it? As I see the plans are cheap, means your LTV is low.
Odd, I'm trying to reach a host in Germany (AS34432) from Sweden but get rerouted Stockholm-Hamburg-Amsterdam-London-Paris-London-Atlanta-São Paulo after which the packets disappear down a black hole. All routing problems occur within Cogentco.
What seems to have happened is that Centurylinks internal routing has collapsed in some way. But they're still announcing all routes and they don't stop announcing routes when other ISPs tag their routes not to be exported by Centurylink.
So as other providers shut down their links to Centurylink to save themselves the outgoing packets towards centurylink travel to some part of the world where links are not shut down yet.
I just experienced HN down for several minutes before it loaded and I saw this story at the top.
I'm doing something with the HN API as I type this, so for a moment I was trying to decide if I'd been IP blocked, even though the API is hosted by Firebase.
I haven't noticed any obvious issues elsewhere yet.
(Just got a delay while trying to submit this comment.)
393 comments
[ 3.0 ms ] story [ 335 ms ] threadIt would be really cool and useful to have an "public Internet health monitoring center"... this could be a foundation that gets some financing from industry that maintains a global internet health monitoring infrastructure and a central site at which all the major players announce outages. It would be pretty cheap and have a high return on investment for everybody involved.
https://puck.nether.net/mailman/listinfo/outages
Public archives:
https://puck.nether.net/pipermail/outages/
Latest issue reported:
https://puck.nether.net/pipermail/outages/2020-August/013187... "Level3 (globally?) impacted (IPv4 only)"
Just have a site fetch resources from every single hosting provider everywhere. A 1x1 image would be enough, but 1K/100K/1M sized files might also be useful (they could also be crafted images)
The first step would be making the HTML page itself redundant. Strict round robin DNS might work well for that.
But yeah, moderately expensive - and... thinking about it... it'll honestly come in handy once every ten years? :/
> Fastly is observing increased errors and latency across multiple regions due to a common IP transit provider experiencing a widespread event. Fastly is actively working on re-routing traffic in affected regions.
However, they report that they've identified the issue and are fixing it.
[0]: https://status.fastly.com/
Connecting from Switzerland.
Fastly, HN, Reddit too.
Only Google domains are loading here.
Their console isn't responding at all and all my servers are unreachable. Their status console reports all normal though.
I spent too much time losing precious time when github/npm/cloudflare are going down, until I figure out it was them.
So currently working on a project[1] to monitor all the 3rd party stack you use for your services. Hit me up if you want, access I'll give free access for a year+ to some folks to get feedbacks.
[1] https://monitory.io
Edit: it’s up again!
Just want to let you know about the spelling error ”Save titme” :)
*3rd party services or possibly 3rd parties' services
Now wondering if it impacted conversion rate?
> Know when services you depend on goes down
"Services go down", not "goes".
Congratulations on your startup!
There is at least one big tool that does exactly the same you wrote. It is called StatusGator https://statusgator.com There are at least 3 much smaller ones.
Have you tried any of them? If yes, what's your point of difference?
And how do you plan to market it? As I see the plans are cheap, means your LTV is low.
So as other providers shut down their links to Centurylink to save themselves the outgoing packets towards centurylink travel to some part of the world where links are not shut down yet.
I'm doing something with the HN API as I type this, so for a moment I was trying to decide if I'd been IP blocked, even though the API is hosted by Firebase.
I haven't noticed any obvious issues elsewhere yet.
(Just got a delay while trying to submit this comment.)
I can't even access the private WoW server I play.
Incidentally, uBlock Origin seems to be completely broken. It doesn't have any local blacklists to work when their ?servers? are unavailable?