I am going to try out Cursor Origin, GitHub is just too much of a bottleneck these days. The only thing that is not readily available is integration with claude code, but with the new claude code mods feature I think it's something I could vibe code quickly.
No, it's a separate forge. You can choose to mirror repos between Github and Origin, but you can also have repos on just Origin and w/ no dependency on Github.
At my company, we migrated basically all CI/CD to Github Actions. Also our Gitops processes. I have warned that putting all the eggs in one basket and they key to the fridge in the same basket was a bad idea. People didn't care. So I don't care anymore.
Github seems to have degraded measurably after they got acquired. As is tradition.
> Github seems to have degraded measurably after they got acquired. As is tradition.
I'm not disagreeing with you that it has degraded, however it was acquired almost 8 years ago at this point, and the software world has been turned on it's head in the last... 3?
Software generally degrades post-acquisition, I'm not denying that. Github has definitely degraded. I'm not sure I would agree that the core reason in this case is "the acquisition" and not "AI making software and free-CI-load demand 3-5 orders-of-magnitude higher than it was 3 years ago"
The latest problems correlate pretty closely when they tried to migrate the entire platform onto azure in an insane timeline, something like ~1 year (for a project of that size, that's borderline insane).
It has been reported new user volume has nearly doubled, and commit volume 10x+ - that could certainly have an impact, but "Azure" and "scale" in my career are words that have never really gone together. Azure Cloud is the only 'hard no, will not interview' of any cloud provider that's listed on a job posting, and I've worked with just about all of them by now I think. Nothing even comes close to such a miserable experience.
The thought of migrating something at the scale of github to azure gives me the cold sweats.
... The reason they did that is because the AI induced load. Their data centers and entire REGION couldn't support the increase in demand so they had to plan an accelerated migration to Azure. They forecasted COMPLETE regional capacity exhaustion.
The problem isn't Azure. Azure is the solution. They have written about this a lot. It's easy to research. You could even just ask ChatGPT to explain it.
Agreed. It's not at all clear that GitHub as an independent company would be doing much better at dealing with this new world than GitHub within Microsoft is doing.
Even taking into consideration the fact that Google never wants to talk to you for any reason, it has been my professional experience that Azure is -by far- the worst of the Big Three "cloud" providers.
Playing devil's advocate: from the graph alone, it could just be that the status page became more accurate, so same amount of down time, but reported more now.
On the other hand, I don't remember things being this bad before the acquisition.
> Github seems to have degraded measurably after they got acquired. As is tradition.
Their explosive growth caused by LLMs seems like it must be a much better indicator than the acquisition. I bet Microsoft are losing a shit ton of money on GitHub right now and seriously questioning the decision to buy them.
No, their decision to focus on a move to azure is the indicator, as the reliability issues started shortly after that, but before their boom in agentic commits
Their move to Azure was prompted by the explosion in load from AI. They've written about this extensively at this point.
You're basically claiming they are lying and the sudden development and execution of an Azure migration plan last year was for other reasons and they just blamed it on AI.
I feel sorry for the teams at this point. These kind of incidents at this scare and frequency are bound to create an acrimonious environment where everyone would be looking for scapegoats on the pretext of finding root causes.
Its infeasible to have a test environment with 1:1 mapping to production for everything in any big enough company. Most of the times people practically aim only for certain critical components to ensure coverage.
It's Microsoft's touch. Anything they touch gets fucked up. It's because of their shitty infrastructure aka Azure. There was an employee here who actually gave insight into how bad the situation was a while back. Just reading that should make you re-evaluate anything owned by Microsoft.
> which creates very different access patterns from how humans use GitHub.
it can
> They could, hypothetically, filter bot traffic;
Nobody seems to be able to do this reliably for a whole huge list of reasons, including: the LLMs themselves can simply mimic human access patterns, timing, and behavior.
I wonder if this will ever end (meaning they've solved scaling for AI), and if not, is that due to the problem being unsolvable at this scale or bad decisions on GitHub's part? For example, they always bring up AI as a cause, but it's also apparent the move to Azure didn't help.
I’m going to hope they can figure things out and improve their uptime. It’s so engrained in my teams workflow at this point, tearing it out and switching to anything else would be extremely disruptive.
I did work for a group using Atlassian bitbucket once, and that had its own problems at the time. I’m still pretty sure even a degraded GitHub beats it. That is to say I don’t know what alternatives for GitHub even are
It would not surprise me if they are going to shift to Plan B sometime soon, because this isn't working. They can't keep provisioning hardware resources in perpetuity to keep up with volume that the AI is pummeling them at. I think most other repo providers have either experienced downtime and/or have locked themselves up to prevent the onslaught.
It is impressive they've been able to keep going, but this is a regular denial-of-service attack at this point, nonstop and relentless.
I know there's a lot of GitHub annoyed people out there but after investigating a lot of the instances of outages it does seems like A) They're abusing ElasticSearch in a way is was never meant to be used B) They've seen an unprecedented increase in scale and now have to provide agent slop a lot of free action minutes until they figure out how to separate commercial support from free support C) Who knows how many other systems were never meant to see this scale there
Still grateful for a lot of the free services and the semblance of community that GitHub still offers. It does seem like it's still the yellowpages of tech.
Their infra is under constant DDoS + they provide a lot of free stuff for open source projects. There's also a constant churn of dependencies, patches, abandoned libraries, etc. We should give them some appreciation, dealing with this stuff is hard.
I think you are giving them way too much leeway. For the service they provide there’s no technical reason it can’t be infinitely scalable. For example their “pull request” page constantly goes down. Serving that kind of workload at high burst load is a fully solved problem.
If they are genuinely compute constrained on CI they should obviously reduce the free offerings rather than take down all their paying customers constantly.
You are basically saying that you, from the outside, have solved the problem perfectly and the whole team of engineers at GitHub just don't know about this one simple solution you are proposing.
Have you considered that you don't have the full context of the problem and it's not "just" a few small tweaks?
My guess is that the problem is internal politics and incentives. Yes I think it is a fully solved problem for a company worth trillions to have an infinitely scalable "pull requests" page.
For example, Microsoft is known to pay significantly less than their competitors. I am guessing they just don't have the systems internally to say "This problem is worth literally billions to us in revenue per year, so we will pay millions to IC's to get it fixed and clear internal roadblocks".
It's not "just" small tweaks it is likely that people who are entrenched need to be moved and they need people with the right incentives in place to get it done.
Even without out the free stuff, no one in this thread seems to have experience managing a self-hosted git environment in even a medium sized company.
It is also constant outages, and then it's your job to diagnose and fix it- and the technical work of that is the easiest part.
It's broken because a team figured out a new way to abuse the company's git infra to get around some process or limitation, and now you have to argue with their manager and then their director about why their team needs to stop using your VCS as a docker registry or whatever.
There is a screamingly obvious answer here and I don't understand why they aren't taking it -- raise prices, make the free tier less generous. Either slow down the growth or up the revenue pipe so you can throw more compute at it.
You can free tier but still sensible rate limits and QoS policy. You can have separate region or cluster for your paying customers. Finally you can spend time re-architecting GitHub Actions or just aquehire one of many competitors that do better co/cd.
Where are the sharks, though? I just see minnows. GitLab has a lot of outages/brown-outs on a much smaller load and Gitea is so confident in their ability to handle load that they host their own development on GitHub.
Trying to slow down the growth would be an admission that everyone having their AI agents pushing 500 commits per hour autonomously may not be an objectively good thing for everyone involved. They won't do that.
github has basically become a joke at this point. everyone engineering org that i know of is at least asking themselves if there's a migration path off there if they're not in the process of building one. Every slack has a custom emoji of github on fire. Github is so joever
Can we split Github, paid/pro users, my orgs are payings north of 25k per year to Github, can we get better service please, even AWS assigns a proper support performance once you hit few tens of thousands in spend. This is just insane, that all of github starts failing at once, I would like to have better experience for when I am working compared to when I want to spam commits with ai to slop something.
They have this with the ghe.com hosting option, it cuts you off from the GitHub.com instance so you can’t share projects there though, if that matters (also $$$).
I'm wondering if they can just build an entirely new platform, onboard new users to it, and then slowly migrate old ones. Even if they don't migrate, the load would stabilize. What are the cons here?
I am surprised that this is still newsworthy. It seems like _something_ is down at Github nearly every day, and their status history backs up that assessment: https://www.githubstatus.com/history
If you are doing CI/CD and want to be able to build and deploy on demand, you pretty much have to treat Github as a mirror and not the source of truth for your applications. If you have an outage, need to deploy a fix, and Github is down, you're on their recovery schedule, not your own.
81 comments
[ 8.7 ms ] story [ 96.3 ms ] threadGithub seems to have degraded measurably after they got acquired. As is tradition.
I'm not disagreeing with you that it has degraded, however it was acquired almost 8 years ago at this point, and the software world has been turned on it's head in the last... 3?
Software generally degrades post-acquisition, I'm not denying that. Github has definitely degraded. I'm not sure I would agree that the core reason in this case is "the acquisition" and not "AI making software and free-CI-load demand 3-5 orders-of-magnitude higher than it was 3 years ago"
The thought of migrating something at the scale of github to azure gives me the cold sweats.
The problem isn't Azure. Azure is the solution. They have written about this a lot. It's easy to research. You could even just ask ChatGPT to explain it.
Even taking into consideration the fact that Google never wants to talk to you for any reason, it has been my professional experience that Azure is -by far- the worst of the Big Three "cloud" providers.
No
Would data change your mind?
https://damrnelson.github.io/github-historical-uptime/
On the other hand, I don't remember things being this bad before the acquisition.
Their explosive growth caused by LLMs seems like it must be a much better indicator than the acquisition. I bet Microsoft are losing a shit ton of money on GitHub right now and seriously questioning the decision to buy them.
You're basically claiming they are lying and the sudden development and execution of an Azure migration plan last year was for other reasons and they just blamed it on AI.
We have a term for that; conspiracy theory.
https://isolveproblems.substack.com/p/how-microsoft-vaporize...
it can
> They could, hypothetically, filter bot traffic;
Nobody seems to be able to do this reliably for a whole huge list of reasons, including: the LLMs themselves can simply mimic human access patterns, timing, and behavior.
I will never trust github at any other company ever again.
I did work for a group using Atlassian bitbucket once, and that had its own problems at the time. I’m still pretty sure even a degraded GitHub beats it. That is to say I don’t know what alternatives for GitHub even are
Outages only come from our infra.
Migrating means migrating your ci workflows, but a decent llm can do that easily.
I'm assuming it's something to do with the USA users getting their first work in?
It is impressive they've been able to keep going, but this is a regular denial-of-service attack at this point, nonstop and relentless.
For example, GitHub can reduce/eliminate free GitHub Action minutes.
Or just cap concurrent jobs for free users from 20 to 2 or 1… lots of levers, their inaction is sad to see.
There’s no CEO of GitHub to contact and Microsoft does not care as Copilot, and Tay continue to wreck the platform as the incidents continue.
No better time to self-host, since as this point your self hosted setup would be far most stable than GitHub as predicted 6 years ago. [1]
[0] https://news.ycombinator.com/item?id=50057012
[1] https://news.ycombinator.com/item?id=22868406
Still grateful for a lot of the free services and the semblance of community that GitHub still offers. It does seem like it's still the yellowpages of tech.
If they are genuinely compute constrained on CI they should obviously reduce the free offerings rather than take down all their paying customers constantly.
Have you considered that you don't have the full context of the problem and it's not "just" a few small tweaks?
For example, Microsoft is known to pay significantly less than their competitors. I am guessing they just don't have the systems internally to say "This problem is worth literally billions to us in revenue per year, so we will pay millions to IC's to get it fixed and clear internal roadblocks".
It's not "just" small tweaks it is likely that people who are entrenched need to be moved and they need people with the right incentives in place to get it done.
It is also constant outages, and then it's your job to diagnose and fix it- and the technical work of that is the easiest part.
It's broken because a team figured out a new way to abuse the company's git infra to get around some process or limitation, and now you have to argue with their manager and then their director about why their team needs to stop using your VCS as a docker registry or whatever.
Raising prices risks churn, reducing free tier risks lowering customer acquisition rate.
You can free tier but still sensible rate limits and QoS policy. You can have separate region or cluster for your paying customers. Finally you can spend time re-architecting GitHub Actions or just aquehire one of many competitors that do better co/cd.
Not remotely compared to github, and you can run GitLab OSS internally anyway.
If you are doing CI/CD and want to be able to build and deploy on demand, you pretty much have to treat Github as a mirror and not the source of truth for your applications. If you have an outage, need to deploy a fix, and Github is down, you're on their recovery schedule, not your own.