GitHub should monitor their status page traffic for spikes, which probably mean something is wrong somewhere, even if they themselves haven't noticed yet.
There was that hilarious multi-hour AWS failure a while back where the status page was updated via one of their internal services... and that service went down as part of the outage.
No it doesn't. The amount of false alarm alerts you can get with internet based monitoring is more than 0. You could have a BGP route break things for one ISP your monitoring happens to use. You could have a failover event happening where it takes 30 seconds for everything to converge. I have multiple monitors on my app at 1 minute intervals from different vendors and ALWAYS a user will email us within 5 seconds of an issue. It's not realistic for a company to have automatic status updates trigger things without a person manually reviewing them because too many things can go wrong on the automatic status update to cause panic.
Who would panic? If nobody notices it's out because it's not, then nobody is going to be checking the status page. And if they do see the status page showing red while it's up, it's not like they're going to be unhappy about their SLA being met.
Maybe you want human confirmation on historic figures, but the live thing might as well be live.
Most paid status monitoring services cover BGP route problems and ISP issues by only flagging an event if it's detected from X geographically diverse endpoints.
For the 30 seconds where you wait for failover to complete: that is a 30 second outage. It's not necessarily profitable to admit to it, but showing it as a 30 second outage would be accurate
TCP default is more than 30 seconds. The internet itself has about a 99.9% uptime. If one company showed every 30 second blip on their outage page all their competitors would have that screenshot on the first page of their pitch deck even if they also had the same issue. 2-5 minutes is reasonable for a public service to announce an outage.
Forgot about that centurylink BGP infinite loop route bug they had where it took down their whole system nationwide. A lot of monitoring services showed red even though it was one ISP that was done.
Not really, things fail in unexpected ways. Automated anomaly detection is notoriously error prone, leading to a lot of false positive and false negatives, in the trivial case of monitoring a single timeseries. For a system the size of GitHub, you need to monitor a whole host of things and if it's quasi impossible to do one timeseries well, there's basically no hope of doing automated many timeseries anomaly detection with a signal-to-noise ratio that's better than "humans looking at the thing and realizing it's not going well".
There's stuff like this that can't be automated well. The automated result is far worse than the human-based alternative.
Pretty much. They want the burden of proof for SLAs to fall on the customer, not on themselves. If a customer has to prove that an outage specifically affected them, they are much less likely to have a successful case against the failure to meet their SLA.
(Not directed at GitHub specifically, but at bogus status pages.)
Two technical reasons capstoned by driving business motivation:
-False positives
-Short outages that last a minute or three
Ultimately, SLA's and uptime guarantees. That way, a business can't automatically tally every minute of publicly admitted downtime against the 99.99999% uptime guarantee, and the onus to prove a breach of contract is on the customer
Status pages are updated by humans and the humans need to (1) realize there's a problem and (2) understand the magnitude of the problem and (3) put that on the status page.
It's not fake, it's just a human process. And automating this would be error prone just the same.
Very good points. Meanwhile I have clients asking me why they can't have a status page to which I reply: you can, but ultimately to be completely fail proof it will be a human updating it slowly. To which they reply: but GitHub or X does it...
And Jeli.io for this! With the Statuspage integration, you can set the status, impact, write a message for customers, and select impacted components all without leaving Slack. Statuspage gets updated with a click of a button.
I wouldn't necessarily call them fake, but the issue often has to be big enough for most companies to admit to it. AWS often has smaller outages that they will never acknowledge.
https://downdetector.com/status/github/ is a far more reliable source - it's just powered by user reports and often will show issues long before the status page ever receives an update.
Keep in mind that downdetector can be brigaded and/or show knock-on problems instead of root causes. e.g. A couple weeks ago there were fairly major spikes across a rather huge variety of services on there, but it turned out that it was actually Comcast that was having trouble, rather than any of the “down” services.
At this point I don't trust self-owned status pages at all - those crowd-sourced ones where users report issues are much faster to respond to outages that may never even go reported by status pages.
Your requests made it farther than mine - mine get to charter in nyc and die there
6 lag-26.nycmny837aw-bcr00.netops.charter.com (24.30.201.130) 158.033 ms
lag-16.nycmny837aw-bcr00.netops.charter.com (66.109.6.74) 29.575 ms
lag-416.nycmny837aw-bcr00.netops.charter.com (66.109.6.10) 30.077 ms
7 lag-1.pr2.nyc20.netops.charter.com (66.109.9.5) 81.351 ms 37.879 ms 27.877 ms
8 * * *
They do have other peering -- that IP from my ISP in Jakarta routes onto Hurricane Electric in Singapore and then to github. From Sao Paulo I go to Atlanta, USA, then to Paris and Frankfurt on twelve99/Telia
193 comments
[ 3.5 ms ] story [ 220 ms ] threadedit: The outage is now acknowledged on the status page https://www.githubstatus.com/
edit: EU folks appear to have things working so it looks like a regional network fault
Lunch time.
Edit: Front page still loads and I am logged in. Everything is as normal. Status page shows everything is down. Lol.
Maybe you want human confirmation on historic figures, but the live thing might as well be live.
For the 30 seconds where you wait for failover to complete: that is a 30 second outage. It's not necessarily profitable to admit to it, but showing it as a 30 second outage would be accurate
There's stuff like this that can't be automated well. The automated result is far worse than the human-based alternative.
(Not directed at GitHub specifically, but at bogus status pages.)
That's so disappointing.
-False positives -Short outages that last a minute or three
Ultimately, SLA's and uptime guarantees. That way, a business can't automatically tally every minute of publicly admitted downtime against the 99.99999% uptime guarantee, and the onus to prove a breach of contract is on the customer
It's not fake, it's just a human process. And automating this would be error prone just the same.
Very infuriating, that.
Unless it is cached
Edit: I could even login
https://githubstatus.com shows all green, but it's not the case...
Edit: deleted traceroute
lb-140-82-121-3-fra.github.com
which from the distance and name I would assume is a frankfurt based load balancer. I get there from BT -> Zayo
I can reach that IP from Washington too, but github returns 140.82.114.3 and 140.82.114.4 from DNS at 1.1.1.1 on a Level3 handoff in Washington
Spot checks around the place show the first returned IP as pingable across the world
Bangkok, Dhaka, Jakarta - 20.205.243.166
Seoul - 20.200.245.247
Nairobi - 20.87.225.212
Kabul, Dakar, Amman, Amman, Cairo - 140.82.121.3
Moscow, Riga, Istanbul - 140.82.121.4
Miami - 140.82.114.3
Sorry to bother!