ah this is what I do too, I understand there are things like woodpecker but wish there was just some community docker image that handles this all for you.
I have to second this, GitLab CI is pretty good. I'll never be on board with infrastructure people's obsession with putting code in yaml but outside of that, the model makes much more sense to me.
A GitLab runner is a machine/VM which can get notified of a GitLab CI job, then run the job's container image (downloading it from GitLab's container registry if necessary), then run the code in the yaml file. You get to control your build environment.
It's so weird that GitHub Actions's model is "have one absolutely gigantic container image which contains everything any build could ever need, and if something's missing, the documented answer is to apt install it every run".
Spent 5 minutes debugging why my builds were failing... then recalled github is not google when it comes to availability... checked status.github.com ... and voila.. there it is.
>> Anything I want to (have to*) be able to deploy on a moments notice, for hotfixes and what not, was moved away from Github like a year ago, because of the horrible uptime. Did others not get the memo yet?
This misses the point in a bad way. You shouldn't have a single deploy funnel to begin with. You should be able to do deploys from multiple places.
This misses the business reality in a bad way. A typical outage with something like Github is not a big deal for most businesses. Sure, some nerds get annoyed, but that's about it. The extra effort in maintaining multiple deploy approaches for a non-trivial system simply doesn't make business sense in most cases.
It's free and you can get it up and running pretty much instantaneously with a coding agent. That's it.
They really should be partitioning their infrastructure so that their paying customers are not in the blast radius of outages from the legions of repos running their 700 test vibecoded regression suite every time they change a config file.
It's crazy but there's really still so much out there where hosting on GitHub is practically a requirement, with no other solutions on the market. Off the top of my head, stuff I encountered in the last month:
3. Only GitHub allows you to use free minutes on managed Windows and macOS runners, which are essential for building and signing native Windows and macOS/iOS software. GitLab's are in beta and require a paid plan, Buildkite requires you to pay at least $30/user without any bundled macOS minutes and no hosted Windows runners available. And for small shops, keeping the macOS and Windows runners clean is such a stupid time sink, with the small number of minutes involved it's much more preferable to just start with free managed minutes and plan to pay for managed runner minutes later.
In those cases, it's more often than not enough to just have a read-only mirror on GitHub, that gets automatically replicated when your true upstream gets updated. Might mean a bit more machinery, but I'll take that over letting some flaky Microsoft service stop me from delivering what I promise my users.
We had a similar experience recently with Google: a few team members were trying to diagnose an issue with our systems on Sep 1, for quite a bit longer than 5 minutes. Turns out it was a major Gcloud outage for about 4 hours: https://status.cloud.google.com/incidents/J5ia5t9p3g9Q5Wi7r8...
Google may be more reliable overall, but Gcloud at least tends to be more critical to a business that uses it, so it kind of balances out. I'd much rather have a Github outage than a Gcloud one.
One of the PRs I eventually managed to open 8 mins ago had a page flash up a generic error page containing a badge error of: "Unexpected token '<', "<!DOCTYPE "... is not valid JSON"
Always thought this rule is hilarious because the fact that such comparisons are so overwhelmingly common that they needed to address it with this bitchy little rule (adding one link to an example per word is a classic tell that an internet moderator is spending their highly compensated moderation time Extremely Mad lol).
However, rather than understand the obvious explanation for it, that reddit and HN's karma systems encourage performative comment behavior with people trying to be epic for the peanut gallery, they make the boneheaded decision to just assume it's an "illusion". It's like Dunning-Krueger but for their theory of mind and general emotional intelligence, with the cause being obvious enough that I don't feel I need to name it.
I suspect it'll only become more common as Reddit enshittifies further and further. The platform is basically unusable for anyone, thus its suitability as a 'containment zone' for the redditor phenotype goes down with it.
I hope BlueSky makes a Lemmy-style Reddit alternative, just something to satiate people's social media-ified forum fix.
GitHub is never going to improve and at this point. You might as well say that the AI chatbots, Tay and Copilot are maintaining the platform and are running it into the ground. No CEO of GitHub to go to as well.
It is time to self host instead of using something as broken as GitHub as I predicted 6 years ago.
Embarrassing that a multi-billion dollar corporation that other billion dollar corporations depend on seems to be fine with breaking their clients workflows. Was github always this shoddy or has something changed to cause this many outages?
They used to run their own data centers, and while the unicorn (their error page back in the day) was visible sometimes, it wasn't nearly as bad as things are today.
And of course, Microsoft is saying that none of this downtime is at all related to them moving everything to Azure, and also at the same time they'll fix all this downtime by finishing moving everything to Azure.
> has something changed to cause this many outages?
AI coding. Which can be interpreted one of two ways:
* The generous way, which is that Github is so overloaded with massive volumes of AI codebases and AI-driven automation that they're hitting a scale they never anticipated; or
* The not-so-generous way, which is that Github itself was one of the first companies to push everyone to AI code as much as possible, which has lead to an eventual breakdown of the stability of the system, as the people responsible for it no longer understand how it actually works, leading to production outages once every few days.
Which let's not forget that they themselves push, both gh themselves and their parent ms.
Let's also not forget that rate-limiting is a trivial thing and small but non-zero charge is also a trivial thing (and would have the convenient benefit of more than paying for itself).
They actively want the free tiers which make the traffic flood/abuse possible because they want the community/market capture and the data.
None of this is unavoidable burden sat on them from the outside like a solar flare.
They push the use of AI tools including their own, and choose not to do any of the trivial things that any of us in any form of IT knows they could do to rate-limit which wouldn't hurt any normal legit user, even ones that want to use AIs.
They beg people to run wild on their service and then fall over every other day from it, and just keep on begging people to run wild on their service. It's been literally years now. Not some new development.
137 comments
[ 1.5 ms ] story [ 53.1 ms ] threadMaybe 30 minutes to get it all setup and then I moved on with my life.
(about to be hit with rotten vegetables by the purists...)
A self-hosted enterprise GitLab instance in Azure, on the other hand, has had outages and performance degradation quite a few times.
A GitLab runner is a machine/VM which can get notified of a GitLab CI job, then run the job's container image (downloading it from GitLab's container registry if necessary), then run the code in the yaml file. You get to control your build environment.
It's so weird that GitHub Actions's model is "have one absolutely gigantic container image which contains everything any build could ever need, and if something's missing, the documented answer is to apt install it every run".
A few hours ago GH pages just would not publish at all.. and the status page showed as fine, I guess it's now cascaded into a proper event.
https://github.com/about
This misses the point in a bad way. You shouldn't have a single deploy funnel to begin with. You should be able to do deploys from multiple places.
This misses the business reality in a bad way. A typical outage with something like Github is not a big deal for most businesses. Sure, some nerds get annoyed, but that's about it. The extra effort in maintaining multiple deploy approaches for a non-trivial system simply doesn't make business sense in most cases.
It's also something that's easy to do if you start from the beginning treating your CI/CD system as something that just runs scripts out of your repo.
They really should be partitioning their infrastructure so that their paying customers are not in the blast radius of outages from the legions of repos running their 700 test vibecoded regression suite every time they change a config file.
1. Homebrew really, really wants you to host your taps on GitHub. The docs are very GitHub-focused: https://docs.brew.sh/How-to-Create-and-Maintain-a-Tap as are downstream projects like dist: https://axodotdev.github.io/cargo-dist/book/installers/homeb...
2. Terraform and OpenTofu public registries support only GitHub, see e.g. https://developer.hashicorp.com/terraform/registry/modules/p... and https://opentofu.org/docs/language/modules/develop/publish/#...
3. Only GitHub allows you to use free minutes on managed Windows and macOS runners, which are essential for building and signing native Windows and macOS/iOS software. GitLab's are in beta and require a paid plan, Buildkite requires you to pay at least $30/user without any bundled macOS minutes and no hosted Windows runners available. And for small shops, keeping the macOS and Windows runners clean is such a stupid time sink, with the small number of minutes involved it's much more preferable to just start with free managed minutes and plan to pay for managed runner minutes later.
Perhaps we're not so dysfunctional that we regularly need to "deploy on a moment's notice." That's a business smell.
Google may be more reliable overall, but Gcloud at least tends to be more critical to a business that uses it, so it kind of balances out. I'd much rather have a Github outage than a Gcloud one.
[0] https://news.ycombinator.com/newsguidelines.html
However, rather than understand the obvious explanation for it, that reddit and HN's karma systems encourage performative comment behavior with people trying to be epic for the peanut gallery, they make the boneheaded decision to just assume it's an "illusion". It's like Dunning-Krueger but for their theory of mind and general emotional intelligence, with the cause being obvious enough that I don't feel I need to name it.
I hope BlueSky makes a Lemmy-style Reddit alternative, just something to satiate people's social media-ified forum fix.
GitHub is never going to improve and at this point. You might as well say that the AI chatbots, Tay and Copilot are maintaining the platform and are running it into the ground. No CEO of GitHub to go to as well.
It is time to self host instead of using something as broken as GitHub as I predicted 6 years ago.
I noticed because SSO was down. They don't even report on that as far as I can tell.
And of course, Microsoft is saying that none of this downtime is at all related to them moving everything to Azure, and also at the same time they'll fix all this downtime by finishing moving everything to Azure.
I wouldn't hold my breath here.
AI coding. Which can be interpreted one of two ways:
* The generous way, which is that Github is so overloaded with massive volumes of AI codebases and AI-driven automation that they're hitting a scale they never anticipated; or
* The not-so-generous way, which is that Github itself was one of the first companies to push everyone to AI code as much as possible, which has lead to an eventual breakdown of the stability of the system, as the people responsible for it no longer understand how it actually works, leading to production outages once every few days.
The data speaks clearly: https://damrnelson.github.io/github-historical-uptime/
1: The status site says that pull requests are working,
2: A PR I'm working on right now is missing commits,
3: The contact support page errors out.
Let's also not forget that rate-limiting is a trivial thing and small but non-zero charge is also a trivial thing (and would have the convenient benefit of more than paying for itself).
They actively want the free tiers which make the traffic flood/abuse possible because they want the community/market capture and the data.
None of this is unavoidable burden sat on them from the outside like a solar flare.
They push the use of AI tools including their own, and choose not to do any of the trivial things that any of us in any form of IT knows they could do to rate-limit which wouldn't hurt any normal legit user, even ones that want to use AIs.
They beg people to run wild on their service and then fall over every other day from it, and just keep on begging people to run wild on their service. It's been literally years now. Not some new development.