HN Explorer

Tell HN: GitHub Is Experiencing Degraded Performance

93 points by SpyCoder77 ↗ HN
Just got this message: "No server is currently available to service your request. Sorry about that. Please try refreshing and contact us if the problem persists."

Edit: at the time of posting there was not an incident on githubstatus.com. Now there is. https://www.githubstatus.com/incidents/zkxwbgr0cnmx

Original Title: "Tell HN: GitHub Is Overloaded"

143 comments

[ 0.18 ms ] story [ 84.4 ms ] thread
ozarkerD
Is github down? Does the day end in y?
SigmundurM
Same here. Just as I was going to push out a release
MallocVoidstar
https://www.githubstatus.com/

It's all green which means it's working. Ignore the unicorn error on every page.

edit: After 5-10 minutes they got around to opening an incident and (apparently manually?) marking the API as having 'degraded performance'.

biennvops
Seems to affect web as well. Currently working on a PR and now it says "Merge status cannot be loaded"
pavo-etc
> Update - We are seeing an approximate 20% error rate across numerous experiences including Pull Requests, Issues, and others. Investigations are currently under way and we will be posting updates as they become available

brutal

leumon
Not sure if related, but claude seems to be down for me too.
hn58622tsf
Been doing this wrong for years, thanks
b800h
Well I was dumped out of Github Copilot as this issue started, and haven't been able to get back in. Seems like a slightly strange thing to be connected, but feasible.
porridgeraisin
Clones, repo browsing etc are all working.

/pull/:id/changes is not working, but /pull/:id.diff _is_. It seems parts of the API is also ok? some gh commands I tried worked.

Files changed works on the mobile app somehow. I guess a particular api version or something is down.

Anyway if someone needs files changed and merge status, check the mobile app. Past actions are also loading on that.

[Android, India]

linhns
“Can’t get latest commit”

Ffs GitHub, get your act together.

greatgib
"degraded performance" is the new 2026 term to say that your platform is down unusable.

On my side it is funny,I'm encountering the following errors: "no server available to process your request".

Maybe azure is out of capacity in its datacenters... Or they didn't pay the server bill.

hohoangphu
Cannot retrieve latest commit at this time. :<
jubilanti
Almost 3 hours in and still "We are still working to identify the root cause"

Can't even view diffs in the web interface!

I'm out! I am willing to pay ~$5-10/mo for a reliable host I can swap as painlessly as possible. Mostly just building little static sites and little PWAs, no more than 500mb a repo and 10gb total. Need a repo with PRs and issues, where people can register free account and contribute or raise issues, GH actions CI, and pages host to deploy static sites. I don't use more than 100 mins of CI a month. Any recommendations?

ChoGGi
No idea about some of that stuff, as I have more basic needs; nearlyfreespeech.net is cheap and stays out of my way.
memonkey
wow totally forgot about nearlyfreespeech. so glad they're still around. wonderful host.
tom1337
Update 16:36 UTC - We identified the problematic component and have taken corrective actions. There are strong signs of recovery but we are still working to completely restore service, with error rates still remaining slightly elevated. We will post further updates as recovery continues.

just shy of 3 hours to find the issue

unknownfuture
Weird, I've been told you can just throw AI at all your logs and metrics and it can magically solve all the things...
akkad33
Maybe they are dedicated hand coders!
rurban
My blackest Monday today. Deepseek is off/too expensive, Claude tokens out for 3 days, kimi refuses, and Github down for the whole afternoon. Maybe I'll take the week off
ALLTaken
hmm very interesting combination of services being down. I've been using DeepSeek v4 Pro and Qwen3.8 now daily for close to 2 weeks. Damn good and reliable. Anytime I need to verify math I check with Kimi k3 tho, it still feels like being superior to these both, not sure if actually true.
ajmurmann
It's crazy that it wasn't that long ago that all we worked with was a compiler on the local machine and maybe a SVN server you'd interact with every few hours
intrasight
Not that long ago that I used a local CVS server for SCCS. OK, almost 40 years - but still.
timcobb
Nvidia build might have DeepSeek capacity
hombre_fatal
Well, it would be a mistake on your part to assume that everything in an organization, especially a large one like github, is a reflection of the capabilities of the state of the art in anything (AI, architecture, programming language, organizational management, operations, etc.).

You can't even be sure that updating the status page as quickly as possible is something github cares about.

esseph
[delayed]
bethekidyouwant
Laws? (Is the first one you must have a status page?)
esseph
Yes, laws, regulations.

They're fucking useful tools of curbing shitty but not yet illegal behavior that is at BEST a dishonest attempt to avoid contractual accountability.

Yes, a fucking law. Maybe a whole set of them!

unknownfuture
It would be a mistake to judge AI based on the performance of one of its strongest pushers and promoters? Interesting take.
therealdrag0
My company is heavy on AI promotion, but many many many people still do things sub-optimally, and many systems are still not well integrated/able with AI.

“The future is here it’s just not evenly distributed”, and that applies inside companies too.

Hard to actually judge without being in the room and knowing the team that owns the incident. And as other said, it’s pretty common IME to have a root cause but slow roll communication of it until remediation has more confidence.

Brian_K_White
It's very easy to judge in the context of how all other service providers are performing on average.

It's very easy to hold any service provider responsible for their own words and deeds that no one else demanded nor commanded (selling themselves to MS, moving onto Azure, copilot stuffed into everything, the architectural garbage that is gh actions). It's very easy to judge when most of their difficulties are self-inflicted.

You excuse and understand the exception, not the norm.

therealdrag0
Sure all that can be true. I’m just saying none of that implicates AI capabilities.
[deleted]
(comment deleted)
hombre_fatal
You're basically saying that before you can know if AI is useful to you and your dealings, AI must solve all of a given corporation's problems including all the human/organizational components and even the trivial ones like how serious Github is about updating their status page, something that probably came down to one guy.

It's just really weak "AI bad" snark.

hombre_fatal
You're basically saying that before you can know if AI is useful to you and your dealings, AI must solve all of a given corporation's problems including all the human/organizational components and even the trivial ones like how serious Github is about updating their status page, something that probably came down to one guy.

It's just really weak "AI bad" snark.

rurban
So just 3 more hours to fix and restart and process the queues
adamddev1
Furiously prompting Claude, "please, please fix this!!!!"
19skitsch
I mean to be fair it’s a huge production system, there are many moving parts. Having worked on outages before in complex systems it is very difficult to identify root causes under pressure
ofjcihen
Especially when you’re worn down because it happens weekly right?
sippeangelo
"Strong signs of recovery" is a crazy way to describe the status of a computer system. Wow.
throw-the-towel
IDK, when a system is very complex — which GH certainly is — this phrasing sounds kinda fine to me? It's recovering, but not yet operating fully as intended.
bargainbin
It’s all just ones and zeroes, and maybe a two. You never can tell with these computers!
adamddev1
I was in a very similar situation and this is what I did:

- put Forgejo on a little $5 Vultr VPS (shared w other things)

- got a 8GB ram cheaper Hetzner VPS (you have to wait for their availability) for around 8 EUR and used that to install the CI runner

- Use Netlify for hosting static sites and PWA

It takes a bit to set up but everything works snappy, clean, and flawlessly. I have unlimited CI minutes. As a bonus I get free Docker image and registry hosting because it's included with Foregejo. I am so, so happy with it all.

Also with Forgejo you just feel like you're using something clean and solid. GitHub started to feel more and more bloated, slow, and unstable.

epistasis
I've been very happy with forgejo for a personal, private server.

I don't care much about the social aspects of GitHub, but I do care a ton about hosting code for long periods of time, long after I've completely changed my personal homelab.

GitHub's most essential service for me is it being the equivalent of the Internet Archive, but for code.

ALLTaken
Great setup, two questions, do you have 2 VPNs now or all on one Hetzner VPS? And how do you setup security?

Just recommend swapping vercel/netlify with something else, we had quite a few issues and high costs with them. Not opiniated here, currently using Cloudflare and BunnyCDN (which is great)

adamddev1
I just SSH into the servers when I need to. I don't have any VPNs with these servers. I just try to follow normal server security setup practices, disabling password authentication etc.
chasd00
That’s what I do. Sensible firewall rules with ufw, turn off services I don’t use, follow the docs for the services I do. For a handful of servers the maintenance overhead is fine.
ALLTaken
I worried being behind in server-security, since all those 0days now being so easy to pull from claude/chatgpt, that maybe you guys found a risk-minimizing front-end to reduce the attack-surface.
cortesoft
This sounds great for personal projects, but what about open source work? Or if you want to collaborate with more than a few people?
skydhash
Unless you want those people to have merging rights, collaboration can be easily done with exporting a read-only version of the git repo via http (even a browsable one using cgit). Then anyone can work on their copy, generate patches (git format-patch) and send the files via any means of communication (email, slack,...).
cursuve
Similar! I was tired of the outages and crazy runner bills so I just finished moving everything to my own server hosting Forgejo. I've had to recreate a couple things (like a dependabot replacement) but it's been a week so far and it's going well - I haven't had any outages nor have I been finding ways to lessen my Runner billed minutes.

Forgejo has some quirks (like no ability to share workflows between private repos) but I'm happy with my choice so far.

hinkley
Sounds like they don’t have a fine enough lens on some particular class of problems. Been there before. The source of the meta problem is so old few people will learn the lesson that hadn’t known it already.
Aeolun
I’m working on it. Unfortunately ‘high reliability’ also means ‘tons of testing’, and ‘finding all the performance holes before your customers’.
__float
Architecturally, how are you storing Git data? If it isn't fundamentally different from bare-repo-on-disk with libgit2 or wrapping the git binary, you're not going to scale any better.
Aeolun
Blobs are unique. Whether you have 700 forks or not, data doesn’t increase. I’m unpacking the git stuff, keeping all the blobs on disk as alternates. Then syncing to object storage. This proved really slow so the blobs are still repacked to sync to object storage. It was fun finding out that some of the repos I test with have 500k blobs, I didn’t know either, but half a million individual put calls made it excessively obvious that wasn’t going to work.

Between the bare git hosts things move around based on available space, IO pressure and whether it’s been recently accessed or not (spinning disks are far cheaper these days). Theoretically there will be a point where the routing becomes a problem, but probably not until there’s several hundred git hosts (I’ll probably have bandwidth problems before that happens).

I’m not sure what the largest git repo is that github has, but I can theoretically keep repos up to 1TB around (plus a sheer unlimited number of forks for the same). I imagine the API’s get problematic before git does at that point (e.g. shuffling data around is much harder than storing it).

stephenway
Forgejo is a great fit for what you're describing. I run Fjord, which is managed Forgejo. Our dedicated instances are above your $5-10 budget, but we also run a free public Forgejo instance if you want to try it without operating anything yourself.
bad_username
> GitLab free tier has all I need ... their next tier is $29/mo, which is too much.

Pay every Nth month, or M months per year :-) Use the free tier features throughout.

imglorp
$work just moved a few dozen repos to GitLab using their import tool, including issues, PRs, comments, etc. It got everything over but stumbled on a few comments.

I guess the message is whichever place you go, you can find or write a migrator that will capture most of what's in GitHub, assuming their API is up long enough...

juliend2
Check out `git init --bare`.

You can basically self-host your repos on a cheap VPS. I did that some time ago. You'll have to create a ssh user that has only 'git' as a shell, iirc, on that box. Works okay. But doesn't provide a web ui.

I think that's where https://git-scm.com/docs/gitweb becomes handy.

pigbearpig
Are you paying for GitHub? Are you complaining about an outage from a product you aren't even paying for? You know the whole point of git is you don't _need_ a central server.

I don't understand the entitlement in your post here.

throw-the-towel
And I don't understand why some people feel they need to be the Criticism Constable.
lowercased
Not OP, but I pay. And it was an annoyance today for several hours that did impact some work.

I don't need one, but I do have that because it's doing a bit more work than just git hosting (administratable access for others on the team, build actions, etc) and yes... I'm paying for it.

sellmesoap
Are you the user or the product? You can be both with GitHub! GitHub provide a compelling product, enough to become the defacto forge for many. But their reputation has taken a lot of hits, as a grey beard I'm salty since the Microsoft takeover, the downtime is just the leaky diaper icing.

To diagnose at a distance it feels like they need a stronger QA process, it's a product that scales horizontally, but somehow they find a way to break it for everyone on the regular! They need less hub and more node!

jubilanti
> I don't understand the entitlement in your post here.

Excuse me, I'm literally asking where I can pay to get the specific cloud/web-based features I listed. I'm trying hard to NOT rely on a free tier because I want to pay to have the entitlement to complain when I don't get what I paid for. You know, the way that commerce has existed for millennia. [0]

I don't know where this aggression is coming from. You seem to want me to stay on the free tier and shut up or go all locally-hosted. Great if either of those work for you, buddy.

[0] https://en.wikipedia.org/wiki/Complaint_tablet_to_Ea-n%C4%81...

throwaway2037
I'm not here to shill, but would SourceHut[1] work for you?

[1] https://sourcehut.org/pricing/

jubilanti
This is the closest I'm finding! CI, web-based ticket submission for anyone with a site-wide login, static hosting, good price... web-based patching would be nice, good to see it in development.

Some honest feedback: 10 minutes of searching and reading left my ADHD brain confused about pricing and quotas. It doesn't have any numbers for what I'm getting for $4/8/12/mo. And it took me 3 reads of that page to notice that the tiers are not actually the standard tiers, but pay-what-you-can. I first jumped to the prominent table then hunted for the fine print for the quotas, didn't even the read "there is no difference between the subscriptions besides price." Then read it but my mind thought "OK, all features are available to all tiers, but at what quotas?" A normal VC backed company might hide this because they're trying to pull one over on you...

Also, "Sourcehut’s public alpha... What we consider an alpha passes for a production-ready system to some." That's a choice, I get it, but it makes me feel like I'd be jumping from one unreliable platform to another. Maybe link to your uptime? (And the real github uptime side by side :P)

Because another honest thought: unlike GitLab, which pretty much has the same shaped features and affordances that you can one-click import and things mostly work, this would be a pretty breaking change to that would be difficult to revert. I do see full data autonomy is on the roadmap too though.

throwaway2037
I was also confused when I first found that page. The second sentence at top of the page, says:

    > [C]hoose the subscription plan most appropriate to your means — there is no difference between the subscriptions besides price.
So, you can pay just 4 USD or EUR per month to get full service. That seems like a steal.

About uptime, what level do you expect? If you Google for "Sourcehut uptime" you will find a bunch of pages with estimates. Example: https://sourcehut.org/blog/2020-07-03-how-we-monitor-our-ser...

systemvoltage
Problem with Gitlab is that they will never stop adding features.
jm4
This, in a nutshell, is why Github is the way it is and there's a lack of good alternatives. I'm not knocking you in particular because we're all pretty much in the same boat here.

"I am willing to pay ~$5-10/mo for a reliable host... Need a repo with PRs and issues, where people can register free account and contribute or raise issues, GH actions CI, and pages host to deploy static sites."

Who wants to build something to acquire that $5-10/mo business? A VPS powerful enough to host a small repository costs more than that these days. There's no opportunity here for someone to build a Github alternative and make any money. The best parts of Github that make it worth using in the first place all come down to the size of the network and that would be extremely difficult to replicate.

It comes down to wanting to develop a SCM hosting service and hitting a $5-10/mo price point. To put that in perspective, people pay more than $5 for a latte and COGS on that is about $1.50 or less. A good customer will come in weekly or more often. I'd rather sell coffee than SaaS SCM.

jubilanti
Good points. I guess I keep just wanting 'you are the product' / enshitification / free tier capture to not be the dominant business model of our time. You're right that a YC VC would likely not funds this. There are many alternative business models though. (edit: https://sourcehut.org seems promising if very barebones and oldschool)

> The best parts of Github that make it worth using in the first place all come down to the size of the network

Not intrinsically, only as a side-effect of economies of scale. At least for me. The 'social' features make it more compelling and explain its widespread adoption and thus economies of scale, but not what made github worth using for me at least. It Just Worked (TM) to unite all the work we used to do in the good ol' days via repos, patches, tickets, mailing lists, IRC, build servers, and host servers.

I don't need my own VPS running 24/7, I said maybe 100 mins of CI a month, even if I'm pushing and building while others are commenting and PRing multiple times a day, it's going to still take a fraction of that VPS capacity.

jm4
Same. Currently, I'm spending $10/mo on a VPS from OVH to host Forgejo. I have hourly backups to an rsync.net account I was already paying for. It's not the most efficient way to do things, but Forgejo is actually pretty nice and I have data sovereignty.
chillfox
You can absolutely build a solution that works for that range, you just have to pick your tech stack for efficiency and not rapid development.

CI will be the tricky one, but I am pretty sure it can work if you keep the limits low, or price it based on usage.

jm4
Why not do it then? Seems you would have plenty of customers judging by this thread and the others today asking about alternatives.
chillfox
Because I don't have the money to start it.

I can't do it in my spare time, because then the company I work for will own it. And I can't just not be employed, because I have bills to pay.

I would love to run a small software company making products individuals can afford.

iamcoder18
Check out https://code.storage/

+-------------------------+-------------------------------+

| Total storage limit | Unlimited |

| Git request rate limits | Unlimited (Seen: 500+/s/repo) |

| API rate limits | Unlimited (Seen: 15k+/s) |

| Max repos | Unlimited |

| Concurrent writes | 30+ |

| Repository size limit | 32TB |

| Git LFS | Supported |

| GitHub/3rd party sync | Supported |

+-------------------------+-------------------------------+

r0b05
There is no reason to be on github anymore.

Your code will used for training and the platform enshitification has been accelerated by Microsoft. It would be wise to move to another platform or even better, host your own Gitlab/Forgejo/Gitea on a vps or at home.

ivanjermakov
I have a habit of reading the source code via github web interface instead of cloning and reading it in a much more comfy text editor.

Today I was forced to clone one because of "the incident". Which is a good thing.

docmars
How are those layoffs treating you, Microsoft?
kingjimmy
Another day, another github incident
cebert
GitHub can’t focus on the basics like a stable platform. However, when they release stack commits a few weeks ago somebody had time to make the hamburger menu an unprofessional stack of pancakes…cute.