Looking at downdetector.com right now and YouTubeTV, HostGator, DirecTV, YouTube, Paramount+, crunchyroll, slink, Philo and Roku all have big spikes in the past few minutes. Wonder how/why Github is related to all these streaming services.
My guess is that there is zero correlation between all those products you mention, and Github. DownDetector is interesting, but I find it to have more noise than signal.
There is a noise floor but I’ve found it to be a pretty reliable early indicator of problems when the question was “is it just me or…” and far superior to any company’s status page.
I've seen two different error pages intermittently. One is a 500 page with a cartoon character holding a sign that says "500" and the other is a page titled "Unicorn!" with the message "We had issues producing the response to your request" and a link to a status page that says everything is fine.
It happened to me and I was also confused. I saw it wasn't on the status page, and I thought, GitHub must be like Amazon, unwilling to admit fault on the status page.
Partial outages of cloud services feel more and more common lately. AWS had degraded service a few weeks (months?) ago. Am I mis-remembering how robust everything except Reddit and Twitter used to be? It feels as if the internet caught a nasty cold at some point and just can't kick it.
Also, it seems way more common to find devs you work with to be over-idealized (focusing on “code beautification” or some other vanity metric vs. accomplishing a task). There’s nothing wrong with beautiful code, but I’ve seen the complexity it can add to an otherwise simple project.
And there’s nothing wrong with that, but, I’ve seen some devs these days spend waaay too much time on “making it simpler” or whatever their ideal is, at the cost of error handling or you know, solving an actual problem. Personally, I don’t care what the code looks like as long as it’s resilient in the face of unexpected errors (ex: a user input of 0 to a division function is not “exceptional,” you should have anticipated that, while a service input of the same may be a breach of contract between the services and IS exceptional) and actually solves a problem.
I’ve been noticing the same thing too. I wonder if the state of the job market over the past couple of years has anything to do with it. More people leaving and taking their knowledge with them, meaning issues are identified/fixed at a slower pace. I know I’ve seen it happen at the company I work at, but I wonder if big tech similar issues.
A colleague got an AWS error saying there weren’t enough resources available to provision an EC2 instance and to try a different region or AZ. Not sure how common that is.
Data centers are finite and I assume Amazon has some metrics on when to stop allowing new instances (even if they have capacity) so they can continue to serve existing customers.
I've seen that regularly with spot instances and new instance types. I assume spot instance availability is correlated to on demand capacity (why would AWS discount instances at spot prices of they were able to sell at the on demand rate).
Azure has been having capacity problems all year and some new instance types aren't available to customers or you need special usage requests approved
I've heard of it happening to colleagues twice in the last three years (though it has doubtless happened more frequently than that) - I think that last time was because they were requesting some ancient instance type that AWS is probably phasing out.
Depends on the region, availability zone, instance type, and time of day. There are always resources if you are flexible. I’ve only ever seen that message when dealing with spot instances or cutting edge hardware that was in the process of rolling out.
I’ve noticed a number of changes over the past couple of years to make it easier to manage mixed pools of compute resources - both spot and on-demand - a good way for AWS to spread load across different instance classes.
A real-time map of cyber-attacks is actually nonsense.
You might visualize a small slice of DDoS attacks and similar routine automatically detectable botnet stuff, but nothing actually serious or interesting.
The trend towards cloud architecture has meant consolidating a lot of products onto just a handful of cloud solutions.
You trade a very bubbly flux of outages across countless bespoke deployments, hosts, and topologies for something with failures that are more rare but also more widespread when they occur.
Personally, I think what you’re noticing is that outages and failures are much more high profile than they used to be because it’s bigger news when these big bubbles burst than it was in that noisier flux where small outages were occurring all the time.
I think as dependence on web services increase there's also higher expectations about availability. You have giant services like Twitter, Facebook, and Google that are almost never down and start to expect that of other companies
So, funny second hand story. Google’s accountants were using third party software hosted on GAE. They bitched to the third part about how the downtime of the software was unacceptable. The third party pointed out that each instance mentioned was a documented GAE outage.
After some internal discussions, the accountants apologized.
Github.com can be reached using some AWS IP addresses, as well as some Microsoft Corporation IP addresses, and who knows what others. But one can also still use original GitHub, Inc. IP addresses on AS36459. I only use the AS36459 addresses for github.com.
Here are some Github, Inc. (AS36459) IP addresses for github.com
I have a RPi but I run NetBSD on it, not PiHole. AFAIK PiHole is just a modified dnsmasq and I dislike dnsmasq. I always used djbdns, never dnsmasq. Today, I use a custom root.zone loaded in nsd and a loopback bound forward proxy that stores the IP addresses for all the websites I visit in RAM. nsd listening on a loopback address just substitutes for a firewall rule directing traffic to the proxy. I use the HOSTS file for FTP server addresses.
These github.com addresses I listed are static for years at a time. As such, there is nothing to "update".
The errors seems to be rather intermittent. Metrist saw a grouping of them at about 4am pacific today, and then again 3pm. That said, in the last hour, less than 1% of Metrist functional tests failed.
More like it doesn't work on the internet with a general audience, because some people will need you to say something in a funny voice to know not to believe it.
It also doesn't work very well as "making a point" sarcastically, because that largely depends on pretending github only does git, and there's very little reason to do that.
The point being made is that we lose the decentralization benefits of Git if we tie ourselves to the proprietary functionality of centralized platforms, as the current event demonstrates. Obviously Git being decentralized doesn’t help much when GitHub is down. But why is that so? Because of GitHub’s proprietary centralized features. The comment was intended as food for thought that maybe we as an industry shouldn’t tie us so much to those features then.
Wouldn’t it be wonderful if when GitHub is down we could just move to a different GitHub instance? Isn’t it sad that we can’t, although it would be very feasible technologically?
But we're not losing the benefits of Git. We still have those. We're 'losing' the benefits of decentralized build servers, but we never had those to begin with.
This is a problem to work on, but it's not a Git problem.
Again? Looks like it is always down at least twice a month. [0] Doesn't affect you, if you are self-hosting like GNOME, KDE, Linux, wireguard, Redox OS, Wine etc who are not going all in on GitHub services.
GitHub is still unreliable even after two years of calling it [1] with little to no improvement.
86 comments
[ 3.8 ms ] story [ 156 ms ] thread"No server is currently available to service your request"
and the images are all broken.
https://i.imgur.com/rjw9ekL.png
Edit: northeast USA
Edit: GitHub making me look a fool by just updating it now (they didn't for the earlier outage...)
Oh, I mean, let’s redefine downtime so we can have nice numbers and everybody gets congratulated. /s
When you measure something for “consequences” you tend to destroy the value of the measurement.
Resilient code is easy to read/grasp code. It should pass a 3 am test too.
Azure has been having capacity problems all year and some new instance types aren't available to customers or you need special usage requests approved
I’ve noticed a number of changes over the past couple of years to make it easier to manage mixed pools of compute resources - both spot and on-demand - a good way for AWS to spread load across different instance classes.
Or build your own: https://github.com/hrbrmstr/pewpew :)
You might visualize a small slice of DDoS attacks and similar routine automatically detectable botnet stuff, but nothing actually serious or interesting.
You trade a very bubbly flux of outages across countless bespoke deployments, hosts, and topologies for something with failures that are more rare but also more widespread when they occur.
Personally, I think what you’re noticing is that outages and failures are much more high profile than they used to be because it’s bigger news when these big bubbles burst than it was in that noisier flux where small outages were occurring all the time.
After some internal discussions, the accountants apologized.
Google isn’t exactly a shining star.
Here are some Github, Inc. (AS36459) IP addresses for github.com
These github.com addresses I listed are static for years at a time. As such, there is nothing to "update".
Wouldn’t it be wonderful if when GitHub is down we could just move to a different GitHub instance? Isn’t it sad that we can’t, although it would be very feasible technologically?
This is a problem to work on, but it's not a Git problem.
Ain't it cloud scale? Have you tried moving it to serverless?
At this point, it seems that GitHub is even more unreliable than a self-hosted solution.
[0] https://news.ycombinator.com/item?id=22867803
GitHub is still unreliable even after two years of calling it [1] with little to no improvement.
[0] https://news.ycombinator.com/item?id=31921324
[1] https://news.ycombinator.com/item?id=22867803