HN Explorer

114 comments

[ 2.8 ms ] story [ 33.4 ms ] thread
thesdev
Finally a GitHub outage during European workday. Yaaay.
nope1000
Although it is lunchtime :/
y-curious
Isn’t most of the European workday lunch time? :P

I joke, the QoL is really nice

esher
Again and again?
crote
GitHub is down only on days ending in -day.
chinathrow
git clone works, git checkout not so much.
computerfriend
Unauthenticated fetches seem ok. Not sure what you mean about git checkout, it's purely local.
sognetic
Can't even pull no more using my key. That's a new low, I'm used to the website being janky nowadays but this is seriously essential
ayhanfuat
At least they have a very clear error message so you know what is going on.

> git@github.com: Permission denied (publickey). fatal: Could not read from remote repository.

Please make sure you have the correct access rights and the repository exists.

chinathrow
That's standard git, not coming from GH.
b1tshift
Just rolled new keys and though I'm going crazy that it's not working
igleria
Can't they just... sell github back like when the germans sold the nuclear plant back to mr burns?
solyin
As of 2026-11-04T10:25:34Z the domain MicrosoftPleaseGiveGithubToGoogle.com is still unclaimed.
zbentley
Ah yes, Google, known good steward of acquired companies’ projects and notable luminary in the developer tooling and workflow space.

Wait…

eknkc
Our deploys are tied to GitHub actions and it is becoming a liability at this point. What alternatives do people use? I mean for FOSS there are some decent stuff but we need a reliable provider for private repos.
pelagicAustral
My deployments are tied to their container registry, so I'm equally screwed... Been thinking of going back to Dokku, I started working on a third wrapper around Dokku (yeah, I've literally done this twice already), and then self-host and self-provision... I honestly thinking seriously about this.
saturn_vk
Self-hosted Forgejo with a bunch of runners might also be interesting for you. The workflow syntax is _almost_ identical to the github one.
import
gitea is amazing.
raddan
Seconded. Also very easy to set up. I’ve been running it for years, along with webhooks to kick off testing and website deployment jobs. I especially like the swagger browser that generate API calls for you to try in curl. I use the API pretty extensively.
rozularen
self hosted forgejo has been my saviour. im thinking in acquiring a new NUC server to increase compute and storage to my homelab to add some new runners
myaccountonhn
Post receive hook on a bare git repo with gitolite for access control.
jelder
Git itself usually works, but we gave up on GHA and use https://www.rwx.com/ for tests, previews, and deployments. It’s 2x to 10x faster than GitHub Actions and testing uncommitted changes on real CI infrastructure is a total game changer.
embedding-shape
You don't even have to migrate off GitHub if you don't want to.

Begin by refactoring the deployments so they're driven by checked in shellscripts, that accepts a git reference to deploy (prefer immutable commit SHAs over mutable branches/tags). Make it use the stuff it already has checked in, and make the actual GitHub Actions stuff as small and concise as possible, ideally each "run" should just call a shellscript with the git reference.

Then move so the deploys happen not on GitHub Actions itself, but on a "intermediate deploy host" that has the source code and access to the credentials you need, and make this host have one secure entrypoint, where someone (or something) can enter and run shellscripts, after pushing their code as a normal git push. Then switch so the deploys from GitHub Actions instead use this new host to do the deploys. Lock down this host hard.

When GitHub and/or GitHub Actions go down, the developer responsible for deploying stuff, should be able to get temporary git and ssh access to this host, can push any git data straight to the host, and their user on this host can run the deploy scripts. Keep in mind that nothing in your deployment process should depend on stuff that isn't also available on this "deploy host", if you have multiple synced repositories and so on.

Now things mostly work the same as people are used to, with the added benefit that when things go down again (like next week), you'll still be able to continue to deploy stuff.

I've done this refactor/migration for tens of companies in the last year, more or less following this exact template for all of them, with no issues reported from either of them, but lots of praise :)

SkiFire13
Some of our deploys use CircleCI instead, but they were still broken because they couldn't checkout from Github.
bonetruck
Forgejo backed by Woodpecker CI running on Kubernetes (Talos) has been amazing! Ditch Github Actions syntax for the simplicity of Woodpecker and containers. You'll thank me later.
paaloeye
How do you run Docker containers inside your K8S? Let’s say you need a test database or test ClickHouse???
stack_framer
We use GitLab where I work. The only problem is it calls pull requests "merge requests," and I don't think I'll ever get used to saying "MR" instead of "PR."
hhh
I have the opposite
jaapz
Why are all you github people pulling on your requests?

I am requesting to merge my changes into the main branch.

dutchbart
Or you are requesting me to pull your changes into my branch.
crote
Because in the good old days when git was still distributed, a merge often started with sending someone an email to obtain your newly-coded commits via `git pull`ing your "gpu-drivers-for-linus" branch from some URL pointing to your personal GitWeb install. In other words: it was a request to pull your code.

Similarly: Why are the Gitlab people so happy to merge? Where is my "Rebase Request", "Squash Request", or "Cherry-pick Request"?

chrisweekly
good bit of lore.

related tidbit: in git, "pull" is basically shorthand for "fetch + merge".

ajasmin
You can still do this on a GituHub PR. Pull the branch locally, read the code, try things out, rebase onto main…

Once all is good, push into remote HEAD and GithHub should auto-close.

Though, larger project tend to leave merges to GitHub itself. Better access control and it’s easier to serialize merges through a single queue for very active projects.

Roark66
This is exactly why I don't understand companies (like some of my clients) that move from self hosted Jenkins to GH Actions.

You're literally replacing a well designed resilient system with a single point of failure run by a different company...

Chris2048
> a well designed resilient system

Wait, Jenkins?

iamjake648
Like with all things, it really depends on having someone competent stand it up and run it. I think we've already reached the point that this is totally worth it for mid-sized companies as opposed to giving MS money for GHA.
hiccuphippo
I get to tell my manager it's github's fault.
shimms
Buildkite with self hosted runners
maccard
Which is 4x the price of github, _plus_ the cost of the runners on top of it.

It's a great service though

karolist
Self hosted Jenkins, been working wonderfully.
maratc
I use Jenkins too, however any pipeline starts by checkout from github, so it failed this morning just the same.
karolist
Same for me, ssh access was intermittent, but at least I have reliable local builds when GitHub actions is out but access works. With AI running Jenkins is not that hard. I build large cuda containers so cost alone ruled out hosted builds for my usecase. There's a bit of update churn but nothing broke so far. I ran around 2000 builds last year. I run it on epyc 7v12 128gb ddr4 host.
icy
We're shipping private repos for Tangled very soon (tangled.org). Shoot me an email anirudh@!
fauzanhilmi
For a moment I thought my ssh key got breached. Fricking hell Github!
dude250711
These days, perhaps, normal undisrupted operation should start counting as an incident...
cashbreaker
I was thinking I'm going crazy with me ssh keys not working on both of my machines.
larsnystrom
Why haven’t they asked an AI agent to fix all their bugs yet? /s
Ampersander
Why does anyone even use Github anymore? Just generate a better forge with AI
feukon
The day GitHub will no longer be free is approaching... they just can't support the abnormal amount of things being pushed rn
crote
For most people here GitHub is already a paid platform. Our companies rely on it as a core part of our development workflows, and we pay a decent bunch of money for it. If it was only the free part of GH having issues, it wouldn't get even remotely near this kind of exposure.

Developers are expensive. If I can't ship a feature on time because GitHub issues keeps forcing me to switch to different tasks to work around git/PR/ticket/Actions downtime, then that costs my company real money - despite us not being the source of the AI DDoS that GitHub has happily been welcoming.

To me, the situation is very simple: either 1) rein in the AI deluge, or 2) isolate paying customers from it, or we'll move to an alternative.

For businesses the current state of unreliability would be unacceptable if it was completely free, the fact that it is actually paid just adds insult to injury.

rarisma 

       - - >  i wake up - -
       |                  |
       |                  V
       github has an outage
dodslaser
Do you fall asleep every time GitHub has an outage? Are you narcoleptic?
shepherdjerred
You have it wrong. GitHub goes down because they fall sleep
binsquare
A manager might say you need to stop waking up.
GrantMoyer
Point of clarification: do you work at GitHub?
Chris2048
Perhaps they are a process/thread?
gib444 

  def is_github_down(day):
      return day[-1] == "y"
sigmarule
All will be solved by tomorrow!
lta
It must be Tuesday, that's why
grishka
I managed to push a commit over ssh but then the client threw an error before the push fully completed. I can see the commit on the web, but my client still thinks it isn't pushed. And now when I try to fetch I get "access denied" after a long wait, which is scary. I first thought something got hacked.

Maybe they need a better AI model for their vibe coding. That'll fix everything, right? At least that's what my friends keep telling me when I say that the free LLMs I played around with a bit are incredibly dumb.

misnome
At this point, even when there isn't an incident, push/pull have been so unreliable as to fail about 30% of the time.

I can live with the occasional outage but this is a constant problem.

malikolivier
GitHub outages can be a drain on productivity. If your stack is pulling artifacts from GitHub, and this consistently fails during GitHub outage, you can consider having a file-cache in between.

Some of our customers unexpectedly started using the service we're running as a file-cache around GitHub to mitigate GitHub outages.

We wrote about this use case in our blog: https://www.stablebuild.com/blog/protect-your-builds-from-gi...

jaapz
Self promotion is generally frowned upon here
juped
no, self-promotion is most of the content here, but it has to be somewhat aware of prevailing norms
a-squirrel-comm
It looks like today there was maintenance scheduled to deprecate support for certain categories of SSH keys: https://github.blog/changelog/2026-09-22-security-improvemen...

> November 4, 2026: We’ll have a brownout of the removal of the ssh-rsa signature type (i.e., RSA keys using SHA-1) and the diffie-hellman-group-exchange-sha256 key exchange algorithm.

I guess something went sideways.

pksunkara
If this is truly the cause of the outage, then the incident looks like a result of GitHub's own vibecoding instead of the increase in usage.

Disclaimer: I am building a code forge at https://juju.bi

shawabawa3
That's not a disclaimer it's just an ad
fredsted
Just set up a Gitea instance, pushed my code and switched my deployment script to it. Took 30 minutes. Github pulls still aren't working.
alper
Same old same old.

What were the 60'000 people exiting Microsoft doing if not improving Github?

colek42
Why do we need centralized CI? We just need a gate to make sure the code passes the right checks before we move the tip.
falcor84
Well, at the very least, that gate needs to be centralized in some way, right? Unless ... are you suggesting using a blockchain ledger for that?
nalekberov
I think blockchain ledger wouldn't be enough, there needs to be AI agents on top of that, that combined with some kind of harness layer, plus, ideally agents gossiping with each other and sometimes fighting with each other to get best results.
icy
That's the idea behind Tangled's spindles.