I was just debating if I was going to side step actions altogether for a new build and deployment flow.
I'm at a point where I'm going to set two permanent branches on the remote, one pertaining to each environment. The targets will poll for the latest commit at their configured branch and perform a rebuild internally if a change is detected. Assuming git core ops are still alive, it is possible to keep this running. It's also easy to migrate to a different vendor.
How long have you been writing software? This is pretty much how we did it pre-EC2 except it was SVN + S/CP for the tip of the changes. There is a ton of work around this to build on!
I have a noninformed thought on this... I think a lot of these issues are actually because they're not on modern Cloud yet.
I think GitHub was rolling everything themselves when Microsoft bought them. Ever since, MS has been working to strategically shift to Azure but that's a difficult and careful process that takes time.
In the meantime, the AI boom hit and usage went through the roof, crippling aging non-scaled infrastructure that hasn't had a chance to be modernized yet.
A lot of issues might very well be "we broke something while migrating", but I think a lot are also simply capacity issues on legacy.
For someone the size of GitHub, the cloud is not infinite or infinitely scalable, and running their own datacenters is fine and completely normal and not at all "aging" or "crippled". Like the meme says, the cloud is just someone else's computer running in someone else's datacenter.
The traffic increases they cite as the reason are absolutely the reason. Those numbers are insane.
Not really hard to understand code is a arbitrary commodity and anyone can grab a cloud model and push code and hook into ci, then have no idea theyre triggering rebuilds everytime they rename their todo apps title.
Basically yes, that's modern day cloud native large scale scalable software. Many smaller software components working together, and sometimes you get a perfect storm of issues or a single point of failure (like DNS) that fails.
Anything people are building with agentic coding could be called a "house of cards."
It should not surprise people when things start collapsing when there have been little falsehoods and random bits of senseless codes written by agents all over the place.
I moved my team from GitHub Actions to RWX and I can't even express how much better it has been. It’s literally 10 times faster, qualitatively different.
Yeah, only yesterday I let claude build and deploy me a end to end project that will flicker my lifx light-bulb when commits to any open source project lands. In the morning I saw it stopped around 11 pm PST yesterday. I was wondering why and I was wondering, how easy it is do a complete working project like this in minutes and what is stopping people to do 100s of such project per day. Github wasn't built for this scenario.
I have not understood, and still do not understand, why everyone locks themselves into github's CI/CD pipeline. It's not hard to set up checks that run on a generic docker setup. I have a home server that I use for all my builds and checks, with a github actions config to run manually if I can't VPN into my home network.
Besides being more reliable, it's just faster to run on a machine you control.
If you're a small/FOSS project, it's not worth the time overhead to set up and maintain and troubleshoot.
If you're a large project, companies generally get more value paying for SaaS and letting devs do dev, rather than troubleshooting Jenkins for the billionth time.
The real value self-host brings is somewhere in the middle, where you need paid-level features with 0 fiat (and can pay in time instead).
Or rather rate limit CI or something. Free repos that don't use large scale LLM activity to trigger a humongous amount of workload onto the github servers don't have to be caught in the crossfire.
Why does everything work perfectly for a week after an outage is fixed? Usage is still 30x and even higher because everyone catches up from the downtime.
Sounds more like a bloated, vibe coded stack that has to be unclogged every 2 weeks.
People want to dunk on GitHub, but honestly going from 50M of <thing> per day to 2B <thing> per day will break pretty much any system anyone has ever built here. Even if it was over the span of ~8 months, many things will fall over and then as you patch those the next domino will fall.
I feel bad for the team, but will be great growth opportunities for all.
> going from 50M of <thing> per day to 2B <thing> per day will break pretty much any system anyone has ever built here
This is likely a pricing failure. Systems breaking due to over-usage is like going to the store and finding that they're sold out. You're much less likely to sell out if you charge more. Meanwhile, GitHub not only didn't raise prices, they cut them at the beginning of the year: https://github.com/resources/insights/2026-pricing-changes-f...
Bear in mind as well that the Pro tier includes 3,000 minutes/month and the Enterprise tier ($21/seat/month, and no published minimum seat floors) includes 50,000 minutes/month. $21/50,000 minutes/month for a solopreneur is practically giving away the farm.
GitHub needs to drastically increase pricing, at least for Actions; their current pricing is practically a WTO dumping violation.
I don't see the connection. Pricing may effect their long term prospects, but I don't see any evidence these issues are being caused by a shortage of metered / allocated resources. They should have access to as much Azure compute as necessary.
> They should have access to as much Azure compute as necessary.
"Should" doesn't imply "does". GitHub is competing with all of the other Microsoft products that need Azure compute, in addition to Microsoft customers. If a customer is willing to buy the compute at a higher margin than the internal chargeback rate, it's definitely not a given that GitHub is getting that compute over the customer. It's hunger games for compute everywhere right now.
Sure, that is possible, but there's still no evidence that compute scarcity is the issue here. It seems a stretch to assume it must be the pricing model when there are a multitude of challenges presented by this kind of ramp.
Problem is while we have a visceral feel for what the "value" of GitHub is to human users for human usage, I'm not sure we have a feel for what the value of the agentic equivalent is.
IMO it's probably very high on paper if you think through what kind of work is unlocked at agent speeds, provided you have a very capable work-execution fabric.
But since we're anchored to human pricing, whatever they propose will feel expensive.
They wanted direct access to the repositories for other purposes than serving the customer. Such as training data for copilot, and who knows what else.
I don't recall this being an issue for Facebook, YouTube, Google, Instagram... All of which have had much better up-time than GitHub, without the excuses. I don't even like those services or companies, but I have to give the engineering teams props for being professional.
I like GitHub (maybe liked?), but their up-time isn't acceptable and I don't get the "poor little Microsoft/8B acquisition" story either.
Allowing that growth was a policy decision, and a bad one at that. They could have acted differently and they deserve all the complaints for not doing so.
When it became apparent that this growth was on the trajectory, instead of doing something to limit it and thus preserving the reliability of the platform, GitHub (or maybe Microsoft) decided to embrace that growth and allow their own platform to fail. They didn’t need to do that. And they shouldn’t have done that. They deserve all the criticisms coming at them for doing this.
You say “they” as there is a single coherent mind in this case. In the large orgs there are a lot of little independent minds doing their little things, barely producing coherent result through shared incentives.
In this case most simple explanation is inaction. Yeah, it was coming, but nobody was exactly in position or cared enough to do something about it.
When your business model is to provide valuable thing for free to corner out competition out of business, don’t be surprised that your service gets to be used asymmetrically.
> going from 50M of <thing> per day to 2B <thing> per day will break pretty much any system
Okay, and?
GitHub chose this. They explicitly market it as a pro-AI platform. They literally invited this giant self-DDoS! GitHub can trivially stop the deluge any time they want, simply by introducing sensible rate limits.
This isn't some poor early startup getting Slashdotted. What we are seeing is upper management intentionally sacrificing existing paying customers, just to avoid having to be the one to suggest slowing down the AI hypetrain.
GitHub knows that it has a mostly-captive audience. Screwing over your day is a sacrifice they are willing to make to keep Line Going Up.
GitHub has had garbage uptime since long before the advent of LLM’s. It may be easy to point at the scale and say “no one can do it”, which may be true, but it’s also possible that uptime has never really been a priority at GitHub
I disabled CI on my part and need to make a separate "ci run" comment to start the CI. I found it increasingly annoying to have CI run "on every commit / push" because I only really need it to be green on the prod-release, not on every commit.
For a small team I setup Forgejo with self hosted CI on Framework desktop. Connect via Tailscale. Runs amazingly well, feels good to have independence from this
My company moved from Azure DevOps to GitHub because of the AI features. ADO was reliable and easy to use, and GitHub has been neither for us. Now Microsoft is adding AI to ADO too, so the main reason for the move is going away :) ADO is much better in my opinion
83 comments
[ 0.23 ms ] story [ 31.3 ms ] threadI'm at a point where I'm going to set two permanent branches on the remote, one pertaining to each environment. The targets will poll for the latest commit at their configured branch and perform a rebuild internally if a change is detected. Assuming git core ops are still alive, it is possible to keep this running. It's also easy to migrate to a different vendor.
It's on Microsoft Azure.
I think GitHub was rolling everything themselves when Microsoft bought them. Ever since, MS has been working to strategically shift to Azure but that's a difficult and careful process that takes time.
In the meantime, the AI boom hit and usage went through the roof, crippling aging non-scaled infrastructure that hasn't had a chance to be modernized yet.
A lot of issues might very well be "we broke something while migrating", but I think a lot are also simply capacity issues on legacy.
For someone the size of GitHub, the cloud is not infinite or infinitely scalable, and running their own datacenters is fine and completely normal and not at all "aging" or "crippled". Like the meme says, the cloud is just someone else's computer running in someone else's datacenter.
The traffic increases they cite as the reason are absolutely the reason. Those numbers are insane.
It should not surprise people when things start collapsing when there have been little falsehoods and random bits of senseless codes written by agents all over the place.
Besides being more reliable, it's just faster to run on a machine you control.
If you're a large project, companies generally get more value paying for SaaS and letting devs do dev, rather than troubleshooting Jenkins for the billionth time.
The real value self-host brings is somewhere in the middle, where you need paid-level features with 0 fiat (and can pay in time instead).
The entire US economy is tottering due to a lack of anti-trust enforcement.
Even The Economist, an oligarch-friendly magazine, has been hitting the alarm bell on this for over a decade: "America needs a giant dose of competition" https://www.economist.com/briefing/2016/03/26/too-much-of-a-...
Sounds more like a bloated, vibe coded stack that has to be unclogged every 2 weeks.
There are no more excuses to self host. You can now ask Claude to do it for you, given that “coding is solved”.
Or even vibe code your own GitHub at this point.
https://social.coop/@netopwibby/117418961623523555
I feel bad for the team, but will be great growth opportunities for all.
This is likely a pricing failure. Systems breaking due to over-usage is like going to the store and finding that they're sold out. You're much less likely to sell out if you charge more. Meanwhile, GitHub not only didn't raise prices, they cut them at the beginning of the year: https://github.com/resources/insights/2026-pricing-changes-f...
And this is beside them cutting the Pro tier from $7/seat/month to $4/seat/month back in 2020 and not touching the price since: https://docs.github.com/en/get-started/learning-about-github...
Bear in mind as well that the Pro tier includes 3,000 minutes/month and the Enterprise tier ($21/seat/month, and no published minimum seat floors) includes 50,000 minutes/month. $21/50,000 minutes/month for a solopreneur is practically giving away the farm.
GitHub needs to drastically increase pricing, at least for Actions; their current pricing is practically a WTO dumping violation.
slopreneurs are the issue
"Should" doesn't imply "does". GitHub is competing with all of the other Microsoft products that need Azure compute, in addition to Microsoft customers. If a customer is willing to buy the compute at a higher margin than the internal chargeback rate, it's definitely not a given that GitHub is getting that compute over the customer. It's hunger games for compute everywhere right now.
"Microsoft turns to Amazon for help with GitHub's AI-driven capacity issues"
Problem is while we have a visceral feel for what the "value" of GitHub is to human users for human usage, I'm not sure we have a feel for what the value of the agentic equivalent is.
IMO it's probably very high on paper if you think through what kind of work is unlocked at agent speeds, provided you have a very capable work-execution fabric.
But since we're anchored to human pricing, whatever they propose will feel expensive.
They wanted direct access to the repositories for other purposes than serving the customer. Such as training data for copilot, and who knows what else.
Pricing low helped ensure access to that data.
The number of PRs sits even lower at 130M per month, or ~50 per second.
I like GitHub (maybe liked?), but their up-time isn't acceptable and I don't get the "poor little Microsoft/8B acquisition" story either.
When it became apparent that this growth was on the trajectory, instead of doing something to limit it and thus preserving the reliability of the platform, GitHub (or maybe Microsoft) decided to embrace that growth and allow their own platform to fail. They didn’t need to do that. And they shouldn’t have done that. They deserve all the criticisms coming at them for doing this.
In this case most simple explanation is inaction. Yeah, it was coming, but nobody was exactly in position or cared enough to do something about it.
Okay, and?
GitHub chose this. They explicitly market it as a pro-AI platform. They literally invited this giant self-DDoS! GitHub can trivially stop the deluge any time they want, simply by introducing sensible rate limits.
This isn't some poor early startup getting Slashdotted. What we are seeing is upper management intentionally sacrificing existing paying customers, just to avoid having to be the one to suggest slowing down the AI hypetrain.
GitHub knows that it has a mostly-captive audience. Screwing over your day is a sacrifice they are willing to make to keep Line Going Up.