Also seeing unicorns, mostly for "No server is currently available to service your request.". If I don't get a unicorn, the page takes a long old time to load.
Same here haha. StackOverflow informed me that it means GitHub is down, and indeed it was.
I have to wonder if Git could somehow report this better. I guess it depends on exactly how GitHub is down, but "fatal error in commit_refs" made me worry that my local repo was somehow hosed.
It reports what it has, if it manages to connect but fetching the metadata or whatever fails, that’s what it’s going to report.
If it can’t even connect it’ll tell you that, but I would assume on github the client will always manage to connect unless their entire network is down.
Seems better done than a lot of status pages, runs on separate infra, updated, has a way to subscribe, etc.
However, saying "degraded performance" when you know it's "down for everyone" is an industry phrasing thing that's irritating. AWS also has "elevated response times" when everyone is seeing 5xx errors, or infinite response times.
> However, saying "degraded performance" when you know it's "down for everyone" is an industry phrasing thing that's irritating. AWS also has "elevated response times" when everyone is seeing 5xx errors, or infinite response times.
Another popular one is "elevated API error rates" when the error rate is 1.
having been on the other side a number of times at a site with huge amounts of traffic, very often things can be down for a huge percentage but our logs will still show thousands of requests succeeding a minute. so it might be working well but slowly for some while not at all for many
It goes down every month like I said before [0]. The last time this happened was 2 days ago [1] then weeks ago [2] and it is evident that it is falling apart in front of us.
First, it was the RSA key leak in [1][3], then the site's key expired causing down time again [2] and now this.
I don't think anyone can tell me with a straight face that GitHub was any more reliable or better when Microsoft acquired it. It is now worse off.
Nothing has changed except for more outages and downtime.
I don't have the data to comment on whether it was better before or after MS acquisition, but would suggest this isn't the best sample size to base any conclusions on.
Because if you don't spend multiple days per year managing your own "backup" to GitHub, you might be left unable to push to GitHub for a few hours per year?
btw, I hope none of your CI system relies on build steps that might include pulling code from GitHub or downloading packages from GitHub Package registry. Often when GitHub is down, my CI system on GitLab is broken too.
I like to reduce the single point of failure count in my projects as much as I can, with common sense. My backup system is running a few cli commands instead of git push, so it's a no brainer. YMMV.
Because I use Github only for git hosting and actions (building & deploying containers). In this case, code is already in my computer, and I can build and ssh to server and pull&run the new container.
That assumes that none of your Docker images have a build step that includes interacting with GitHub, or if some do, that you have every affected layer already cached on your local computer.
I understand the point you're making here, but I feel it is being made in an effort to prove hakanderyal technically wrong rather than to evaluate the practicality of single points of failure, which is what they're trying to promote. I this this conversation would be much more helpful and insightful if it was kept on that evaluation track rather than trying to have a final word.
I don't think "GitHub is down" threads are known for the quality of their conversation, but sure.
I also agree with reducing (increasing?) single points of failure. I'm not trying to be pedantic, but rather observing that in practice, it's not nearly as easy as spinning up a backup Git server (which is already hard enough).
Maintaining two classes of build infrastructure throughout all your dependencies is probably not a worthwhile problem to solve, unless you want to control for the improbable risk that GitHub will be down for weeks at a time. You'd be much better off ensuring that you are able to perform rollbacks without needing to pull from the external world, because this way the worst case scenario is you run a stale version for the time that GitHub is down, in the off chance you pushed a broken version right before the outage.
True, but there are other ways to go about it. Rather than trying to challenge someone's setup, which (I assume, apologies if incorrectly) you aren't familiar with, you can start by asking them about the setup and how they keep it independent from GH. Let them expose either the success of the endeavor or its shortcomings. Such an approach is a constructive one, whereas trying to challenge someone is an antagonistic approach. We're all in this together, so let's keep our discussions constructive, and focus on learning from each other rather than try to tear each other down.
There is the GitHub Status Page [0], which doesn't display aggregate stats, but you could scrape it and do the analysis.
I suspect what you're getting at is that the downtime might evaluate to multiple days over the course of the year. Maybe that's true, idk. I'd be curious, but you'd probably want to do the analysis separately for different services (e.g. Actions vs. Package Registry vs. Git outages all have different effects on build infrastructure downtime).
Depends on the context. I need to push to a repo and QA needs to test the app, but since I can't push, they can't. No other practical way (other than setting up another origin on Bitbucket or GitLab, or emailing/zipping sending etc).
I also use it for connectivity check. Most other sites have so much ads and cookies it creates long lags with adblock in some older systems I use. So I point to HN and get a really fast response that the web is working
Step #3 is unfortunately often pointless given that often either:
1. the status page is unavailable too
2. the status page reports the service as green/available even though it's red/down (maybe it's still accessible to the service pinging its health status, maybe it's "accessible" but not actually functional, maybe the engineers were too busy fixing the problem to click the button to update the status page, maybe not updating the status lets them pretend they're within SLAs or KPIs)
I suppose a static html page saying everything is fine hosted on the same infrastructure as your service would be an accurate indicator a good chunk of the time.
Most places don't have an automated status page because of the issues with automation showing outages when they don't exist. Someone manually goes and clicks something on the status page.
This doesn't seem like a good reason. I can't imagine anyone checks githubstatus.com before accessing github.com. People check it after they have issues.
I assume it's more due to contracts, marketing, and denying responsibility.
Except for ISPs. Whenever a major website goes down, people blame their ISPs. When their own provider goes down, they don't remember which one they use and blame every other one in their region as well.
Is the implication that someone affiliated with Twitter took down GitHub because of something someone hosted there? Does that honestly seem plausible to you?
I'm not saying "that happened", I'm asking about the possibility. Does it seem plausible that someone sympathetic to Twitter launched an attack on GitHub in retaliation? Absolutely.
Honestly, while it's certainly possible it doesn't seem plausible to me. I have a hard time seeing someone with the extreme emotional connection needed to lash out because of the leak also having the technical sophistication required to bring GitHub down.
Why GitHub is down so often? Why is it not possible to keep it up 100% of time (without counting physical failure)?. I haven't seen any down time for my system (it has hundreds of thousands of users online) in months since I have completed the setup.
I Think he was referring to Google Search in general. I've never witness since Google went live in 1998 any Google Search downtime. Probably happened but I can not remember it.
Sure, but they can have several copies of the index per datacenter, retry your query multiple times posibly even in a diffrent datacenter. New code and even updated indexes can be tried and then fall back to yesterday's version.
I am certain the are normally writes going on; they do run Analytics on their homepage. However they get to defer, retry and play lots of eventually consistent tricks, worst case just swallow the exceptions. The fact they can make the service _seem_ to the end user as fully working whilst being unable to write is a major factor in achieving their world beating reliability.
Great question: Google's homepage revenue is directly 1:1 matches to its uptime. Its user retention is also loosely tied to its uptime, as the value is mostly a replaceable commodity (is Bing worse? sure, but it has results). This leads to the organization investing huge amounts of time and money in ensuring its uptime. I can recall a single outage in the past several years.
On the other hand, GitHub's revenue is mostly monthly/annual licensing and their have great stickiness as it's not trivial to migrate to an equivalent service provider (excluding minor projects who only use a couple of features). They can increase profits through feature development and cost saving, a lot more than through uptime. Is there a limit to this? Of course.
Google loses money when search is down because they can not serve ads. Does Github actually lose money when they are down? I think that because everyone is on subscription they do not lose money by the second, rather instead they lose reputation and long term they could lose customers, but Github's income isn't quite as sensitive to downtime as Googles in general, thus less investment in DevOps in comparison.
I think you are nitpicking. My point is that companies (including Microsoft!) are capable of running large scale infra with much higher uptimes than GitHub. They want to put themselves at the center of our workflows (e.g. GitHub Actions) yet they are not delivering uptimes that are commensurate with that. What is their excuse?
Yeah, I agree with you, bit nitpicky. I also agree that they shouldn't have an excuse, besides confessing their engineering standards are not up to the level of their ambition, which is why I never make anything in my infrastructure depend on anything GitHub, everything that I use GitHub for, I have alternatives setup for the inevitable ill-timed downtime I know will happen.
ummm, I guess scale is similar. I am just a single person vs an organization. My google knowledge vs industry expert with years of experience.
My point was not about similar scale though. How hard is it to keep a system up? AWS is a whole universe compared to GitHub, yet it doesn't go down as often as GitHub.
Only GitHub truly knows. But everyone here knows that since Microsoft acquired it, it has degraded to the point where it goes down every month.
It is so frequent and unreliable, you just might as well self-host at this point. You would likely have better up time than GitHub over the past three years since this prediction. [0]
I feel like HN could just have a traffic monitor that adds a little icon to the main page. Like "I dunno what's up, but there's a LOT of y'all here right now, so something probably is".
Suppose you start your day by downloading some code from GitHub to work on. This morning you would be stuck. Do you save code on your hard drive or company servers or GitLab to handle this risk?
"GitHub is down" probably wouldn't even make the top 10 stupidest reasons that I couldn't work. Is the idea here that you're downloading brand new code? I would imagine most people would have a stale from Friday copy of the repo locally.
168 comments
[ 4.0 ms ] story [ 97.9 ms ] threadI have to wonder if Git could somehow report this better. I guess it depends on exactly how GitHub is down, but "fatal error in commit_refs" made me worry that my local repo was somehow hosed.
If it can’t even connect it’ll tell you that, but I would assume on github the client will always manage to connect unless their entire network is down.
However, saying "degraded performance" when you know it's "down for everyone" is an industry phrasing thing that's irritating. AWS also has "elevated response times" when everyone is seeing 5xx errors, or infinite response times.
Another popular one is "elevated API error rates" when the error rate is 1.
https://research.kudelskisecurity.com/2023/03/06/polynonce-a...
First, it was the RSA key leak in [1][3], then the site's key expired causing down time again [2] and now this.
I don't think anyone can tell me with a straight face that GitHub was any more reliable or better when Microsoft acquired it. It is now worse off.
Nothing has changed except for more outages and downtime.
So so reliable. /s
[0] https://news.ycombinator.com/item?id=35004629
[1] https://news.ycombinator.com/item?id=35295216
[2] https://news.ycombinator.com/item?id=35003741
[3] https://github.blog/2023-03-23-we-updated-our-rsa-ssh-host-k...
It's apparant that it wasn't without issues prior to acquisition (e.g. a quick search for GitHub issues prior to 2018 gives this: https://techcrunch.com/2017/07/31/github-goes-down-and-takes...) - reporting issues in 2017, 2015, and 2012.
I don't have the data to comment on whether it was better before or after MS acquisition, but would suggest this isn't the best sample size to base any conclusions on.
Have backups for critical systems, people. In my case, it's building docker containers locally and luckily deploying to one server via ssh.
btw, I hope none of your CI system relies on build steps that might include pulling code from GitHub or downloading packages from GitHub Package registry. Often when GitHub is down, my CI system on GitLab is broken too.
"docker buildx build ... -push" "ssh ...@..." "docker pull ..." "docker compose up ..."
I also agree with reducing (increasing?) single points of failure. I'm not trying to be pedantic, but rather observing that in practice, it's not nearly as easy as spinning up a backup Git server (which is already hard enough).
Maintaining two classes of build infrastructure throughout all your dependencies is probably not a worthwhile problem to solve, unless you want to control for the improbable risk that GitHub will be down for weeks at a time. You'd be much better off ensuring that you are able to perform rollbacks without needing to pull from the external world, because this way the worst case scenario is you run a stale version for the time that GitHub is down, in the off chance you pushed a broken version right before the outage.
I suspect what you're getting at is that the downtime might evaluate to multiple days over the course of the year. Maybe that's true, idk. I'd be curious, but you'd probably want to do the analysis separately for different services (e.g. Actions vs. Package Registry vs. Git outages all have different effects on build infrastructure downtime).
[0] https://www.githubstatus.com/history
1. check my internet connection
2. check HN
3. check official statuspage
that said, HN is also down a lot lately, but that the kind of outage that makes more more productive actually!
1. the status page is unavailable too
2. the status page reports the service as green/available even though it's red/down (maybe it's still accessible to the service pinging its health status, maybe it's "accessible" but not actually functional, maybe the engineers were too busy fixing the problem to click the button to update the status page, maybe not updating the status lets them pretend they're within SLAs or KPIs)
I assume it's more due to contracts, marketing, and denying responsibility.
How do you do that?
HN though comes with all the commentary.
https://www.githubstatus.com/
AWS is basically never down.
WhatsApp is basically never down.
Time for GitHub to grow up?
You don't know this. Google results are not the same for all users. How do you know there isn't R/W going on, particularly when signed-in to Google?
(Unless you work at Google on search, in which case I stand corrected!)
On the other hand, GitHub's revenue is mostly monthly/annual licensing and their have great stickiness as it's not trivial to migrate to an equivalent service provider (excluding minor projects who only use a couple of features). They can increase profits through feature development and cost saving, a lot more than through uptime. Is there a limit to this? Of course.
The complexity difference between the Google search "app" (not counting the vast indexing infrastructure) and GitHub is also vastly different.
> AWS is basically never down.
Lol what? Have you used AWS?
> WhatsApp is basically never down.
Makes sense, Whatsapp always had a huge focus on reliable infrastructure, since day 0. Pays off I guess :)
My point was not about similar scale though. How hard is it to keep a system up? AWS is a whole universe compared to GitHub, yet it doesn't go down as often as GitHub.
It is so frequent and unreliable, you just might as well self-host at this point. You would likely have better up time than GitHub over the past three years since this prediction. [0]
[0] https://news.ycombinator.com/item?id=22867803