Loading github.com is returning a 500 for me currently, so seems like more than just issues/pull requests. Also seeing actions fail with 500s on assorted steps.
Similar issues for me. I can load github.com and my profile, but visiting a repository (or trying to git pull a repo with the https origin) returns a 500.
They're having so many outages that they've moved from making statements after each event to aggregating them all and write about all of them on a month-by-month basis.
I don't remember when it started getting worse, but if I do a guess, I'd say around 2018 sometime, which I'd also guess is when they started pushing more changes too.
Yeah, I was getting 500s for about three minutes before they posted the status update. I guess it's good that they at least update the status page in a timely fashion, but the third day in a row of downtime is not exactly good service.
Have to give it to them for how useful their status page is. Other products we use play all kinds of word games to downplay issues so they don't have to show them on the status page, which is extremely annoying.
I can confirm this as well. I started seeing 500 errors intermittently when trying to view pages, so I checked status page and saw everything was green. Status page started showing the incident within about 3 minutes of when I started seeing issues. Clearly that's all based on happenstance of when I was landing on GitHub's website, but I have found that of all the status page's by large companies, GitHub's is almost always showing an incident as soon as I start noticing issues myself.
I'd say it's reasonable to list it as separate outages on the status page as it's really a representation of "is github available and working as expected". Even if it is the same issue, when they manage to mitigate it (or it goes away) I'd want to see that everything is now available from a user perspective.
That said, they're getting to the point where they really need to make some larger post about this. It seems reasonable to assume it is all from one root cause.
Yes, we do this using https://gitea.io/en-us/ on a private server (firewall, backups and a replica) for most projects. Github is only used when it's required by a stakeholder.
Gitea here. Should disclose that we ended up contributing to the project and developing an understanding of its code in order to integrate into our org, so ymmv. (but at least you can see the source).
In the past I worked at a company which used the commercial solution from JFrog, I don't remember ever having problems with git availability as a user.
Are you interested in spinning up an entire CI environment? or something where anyone can push a branch to a file based mirror?
There are aspects of permissioning that the cloud git repo providers have that become more challenging to implement as a home grown solution and unless you have the resources to maintain it, it also becomes interesting.
On one hand, you can do `git clone --mirror` and you'll have a copy of the repo and put that on a file share... though there's no permissions or automatic syncs for it (or CI). If you want those, then you get into some development (and maintenance) of the git hooks.
Going to things like a local hosted gitlab instance means that you need to have a local docker hosted environment running, and someone to maintain that, and the storage for it, and all of the other fun that comes with administering a complex 3rd party application on prem. When things are going good, it's an hour or two a week... when something breaks its several hours with calls to support (you're using a paid / licensed version to get support... right?) from someone who has a sysadmin skill set rather than a developer skillset. And don't forget about DR.
How??? You do the merge, which either creates a new commit for the change, or appends the commits to your existing tree. Then you push that to the remote. If the push fails, you can just push again, it's not lost. And if the merge failed, you didn't have any merge commit to begin with.
There used to be a pretty consistent bug that if an on-site PR merge failed but you clicked "Retry", that it just did a basic non-squash full-merge discarding all your commit message work, often requiring a revert to tidy things up. It could be similar to that.
Sales handing out Office365 discounts and trying to convince people that AWS and GCP is going to steal their data, judging by companies I worked for that used Azure.
Wasnt it true? Thas Amazon abused their AWS position and stole their competitors data, so thats why Germany's retail businesses are building their own Clouds
I worked on the volume licensing part of Microsoft years ago and deployments were stressful. They'd start at friday late like 8pm or so and go until 8am in the morning. Everyone was on a long call the entire time. I hated it.
Not only Copilot, seems like some Microsoft services like Bing AI and Bing Image Creator have some issues today as well with 4xx / 5xx, and incorrect region authorization (had to switch the account region from a European country to US to make it work again on mobile)
I think you may have missed the joke - they were implying Github was using Copilot internally, causing the outages, due to poor output. Not that Copilot itself was unavailable (although that may be true, also)
Today seems worse than yesterday. I'm getting wildly inconsistent results when viewing repositories after a push. Hard to tell if my push actually went through, and it's not triggering actions.
They blamed the march and april outages on some database query that was changed due to an infrastructure change they rolled out. I'm guessing their infrastructure change caused some other race condition issue that they are only seeing after major production failure due to not load testing enough in their staging environment https://github.blog/2023-05-03-github-availability-report-ap...
From an SRE, one of their DB clusters failed. They use Vitess which is great, but it can be prone to hotspots and doesn't auto-shard. Heavy usage (esp. from large customers, rogue jobs) can take down the cluster. When it goes down, it's a PITA to resolve.
Ah, unbalanced shards via wrong sharding keys was an issue at one point, IIRC. I remember talking with an SRE there when something bad happened at GitHub last year, and I know that this time the current DB cluster failed.
To be clear, I _was_ mapping previous incidents with this year's incident — no competitor or hard feelings involved. I really like Vitess, fwiw. And the only thing I really love is FoundationDB :)
Side note: "Autosharding" is largely a myth that unproven databases are touting. Sharding is complex and requires planning and control. Databases that start shuffling data round without oversight produce nasty surprises. Trying to be too magic is normally always a mistake with databases.
> Codespaces is experiencing degraded performance. We are continuing to investigate.
Imagine not only not being able to push your code, but also not even being able to _write your code_ at all. And so many orgs rely on Actions to even be able to deploy. Geez. I personally believe that the cloud sucks.
I'm considering host a gitea instance backup all of my repos.
I have an important fix that need to be deployed right now but there is no way to deploy it in a normal way with our CI which one was setup with Github Action. Fortunately I have a instruction to bypass CI and build the source by myself.
But again, Github defeat me because our release workflows are depend on GitOps which are effected by Github issue. Ahhhhhhhhhh I have to build the docker image, push it to ECR then update a YAML template to make EKS apply the new changes
It's 9PM in my timezone and I'm waiting for my patches are up. A frustrating incident
Gitea's ability to create a local repository as mirror of a remote repository is great for this. You can stay on Github and have your code regularly mirrored locally.
Or use Gitlab. I remember doing a mirror syncing from Github with one of my gitlab repository. The main reason I used to do it is Gitlab offer free built-in CI at that time
In my case one of most important thing is the GitOps workflow. It's single source of truth so it's also single point of failure ;(
I have this setup running on a Synology NAS at home. I'm currently syncing all of my starred github repos to local storage using a short bash script that runs once a week. Once a repo is in gitea, it pulls any new updates from github every 6 hours or so. It's mostly for archival purposes, just in case something majorly bad happens to github.
142 comments
[ 3.3 ms ] story [ 243 ms ] threadI'm not joking: https://github.blog/tag/github-availability-report/
But strange that it keeps happening almost every day now.
That said, they're getting to the point where they really need to make some larger post about this. It seems reasonable to assume it is all from one root cause.
In the past I worked at a company which used the commercial solution from JFrog, I don't remember ever having problems with git availability as a user.
There are aspects of permissioning that the cloud git repo providers have that become more challenging to implement as a home grown solution and unless you have the resources to maintain it, it also becomes interesting.
On one hand, you can do `git clone --mirror` and you'll have a copy of the repo and put that on a file share... though there's no permissions or automatic syncs for it (or CI). If you want those, then you get into some development (and maintenance) of the git hooks.
Going to things like a local hosted gitlab instance means that you need to have a local docker hosted environment running, and someone to maintain that, and the storage for it, and all of the other fun that comes with administering a complex 3rd party application on prem. When things are going good, it's an hour or two a week... when something breaks its several hours with calls to support (you're using a paid / licensed version to get support... right?) from someone who has a sysadmin skill set rather than a developer skillset. And don't forget about DR.
This is the error I was seeing:
https://www.wsj.com/articles/amazon-scooped-up-data-from-its...
The gist is that yes it’s true. They’d come out with their own Amazon Basics branded stuff and push it to the top.
I suspect that it's related to high load.
To be clear, I _was_ mapping previous incidents with this year's incident — no competitor or hard feelings involved. I really like Vitess, fwiw. And the only thing I really love is FoundationDB :)
Side note: "Autosharding" is largely a myth that unproven databases are touting. Sharding is complex and requires planning and control. Databases that start shuffling data round without oversight produce nasty surprises. Trying to be too magic is normally always a mistake with databases.
It was the DB, and it was rogue usage on May 10, so I'm standing by my original comment
Of course, if this continues happening it may be time for the OSS community seriously consider migrating en masse.
[Edit to fix grammar - thanks for the corrections!]
As you note, en masse is common — enough so that you even don't need to italicize it (although you can).
Imagine not only not being able to push your code, but also not even being able to _write your code_ at all. And so many orgs rely on Actions to even be able to deploy. Geez. I personally believe that the cloud sucks.
It's sill weird to me how many and how much companies relly on Github infra.
I have an important fix that need to be deployed right now but there is no way to deploy it in a normal way with our CI which one was setup with Github Action. Fortunately I have a instruction to bypass CI and build the source by myself.
But again, Github defeat me because our release workflows are depend on GitOps which are effected by Github issue. Ahhhhhhhhhh I have to build the docker image, push it to ECR then update a YAML template to make EKS apply the new changes
It's 9PM in my timezone and I'm waiting for my patches are up. A frustrating incident
I'm interested in building the same but if yours already works well then I see no point in duplicating important work.
Here's a gist to the script: https://gist.github.com/sklarsa/845152721ee9292eb01f70756b89...
As for gitea, I'm just hosting it using Docker and orchestrating using a simple docker-compose file that maps the gitea data directory to a Volume on the synology: https://gist.github.com/sklarsa/0dd6d6094dac6bf6e7bf61df9ca5...
It's all hosted on my private network at home.