270 comments

[ 3.8 ms ] story [ 252 ms ] thread
GitHub git operations are down. Any git operation returns a 500 error response.
Can't pull anything from my repos which is not great.
Yeah, this is not good. I've experienced other outages on their other products (e.g. GitHub Actions), but not even being able to do a basic git pull is an entirely different level.
I'm effectively dead in the water here. I guess I'll go touch some grass. Thank you to the SRE's who are looking at this.
Looking forward to a blogpost or writeup from github on this.
Is it possible to download a zip of a repo through the GUI?
SSH access appears to be rejecting my key too.
GitHub Actions are also failing with 500 (gateway timeout) errors when trying to fetch the repository.
Yep, 100% of git operations are failing. CI/CD and dev operations are at a halt.
Yup, SSH rejecting my key, thought it was something at my end but no. Can't run any Yocto builds... I guess a good opportunity to take a walk or do some life admin.
Back up now for me.
I hope you didn't wait until now to push your day's work!
Experiencing this as well. I was a bit concerned something was wrong with my key but alas it is not my key but an outage.

Developer "snow day".

GIT is down today, Code rests in snowy silence, Developers play.
Spent like 30 mins tryig to figure out why ssh auth was not working. Compared sha256 signatures, doubted my reading abilities and pulled my hair.
this coincided with an os upgrade for me.. made for a very confusing 15min until i checked gh status
same here I was scratching my head trying to figure out if something had corrupted my system or what
damn, why couldnt it happen during east coast business hours?
lmao i deleted and readded a new key thinking my key was fucked
It's almost as if only Git is distributed, but people sold out to Github for convenience. Too bad Git lacks a distributed bug tracker and wiki system like Fossil. Guess Github has to fail a lot more for things to change.
people could not be more clear that their preference is for reliable and easy-to-use centralized services maintained by professionals, and not decentralized systems that require a great deal of user expertise
(comment deleted)
Ironically github is sort of the exception that proves the rule: decentralized in the one way that really matters (decoupled development on individual systems), but centralized for easy interaction in the way the market demands.
Similar thing that happens with crypto exchanges, also swift transfers and federal states for that matter.
Yes, subsidiarity is a great principle.. in-principle, but in practice it often gets the curb. See https://en.wikipedia.org/wiki/Subsidiarity

So far, having individual small countries seems to keep the centralisation at bay for longer than just having states in a federation.

(Look at Germany, Austria, Australia, the USA for examples of the latter. Interestingly, the UK is legally not made of federal states, but in practice they have granted more autonomy to eg Scotland over the years. And everyone knows that Scotland would secede and get away with it, if there was a power grab by London. In that sense, they are more federal than the US, where secession is very much verboten.)

But it demands both. The ability to develop in a parallel, decentralized way, and the ability to integrate things at a central point, an authoritative source and a blessed official destination.

It's similar to how databases allow to begin parallel, concurrent, even contradictory transactions, and also guarantee serialized, consistent database state, and rejection of invalid updates, at commit time. Both aspects are utterly important.

Thats mostly because of herdmentality not because of a considered approach. Decentralization assumes the majority are anti social.
what a convoluted way to say you do not agree with this. when you are dealing with people of all skill levels, every step that you put on the development process costs real money. over that, not everything is and has to be open source. we all have source code that is proprietary and has a value for us as it is. making that decentralized and private is not something easy to achieve (and I don't really see a benefit). the problem from my point of view is that we have a single player (github) that managed to attract a huge percent of the market and the competition, while it exists, it's minor.
How so?

Decentralization can be hidden from the user, it's an implementation detail.

There's literally a popular decentralized social network.

It's less about the tech, and more about the execution.

Historically we can look at LimeWire or PopcornTime as an example.

Both decentralized, both popular due to the ease-of-use.

> There's literally a popular decentralized social network.

No there isn't. Not a single one.

There are a few federated social networks, which is a fancy way of saying that they are centralized networks that have (or can have, in principle) more than one "center".

In practice, the overwhelming majority of users of such networks gravitate towards one or a handful of large providers. And many of those providers actually refuse to federate with other providers unless they follow an ever-growing list of politically-charged rules. This is just centralization with extra steps.

bluesky has over 27 million users
Bluesky is federated, not decentralized.
(comment deleted)
No, it's because complexity comes with a cognitive cost, and delegating responsibility to other entities minimizes this

If you don't account for the benefit, it looks irrational, but this is true of absolutely anything

I actually don't care whether it's centralised or decentralised, or who's managing it.

But you are right that I want reliable and easy-to-use services. And centralisation is often one way to go there.

As an interesting counterpoint: Git itself is decentralised and replaced centralised services like Subversion. And that made git easier to use, especially easier to get started with: no need for a server, no need to be online, just do `git init` in any old directory.

A GitHub-clone could be more decentralised, but they'd need to use that decentralisation to drive those other features that people actually care about day to day.

(comment deleted)
> no need for a server, no need to be online, just do `git init` in any old directory

svn doesn’t require a server and there is no need to be online. It works perfectly fine over the file:// protocol.

Interesting.

Was that always the case? I remember it being quite a hassle to set up (following tutorials online), but that was about 15 to 20 years ago or so.

And a `git remote add name url` and you are setup to use another remote server.
Only if your repo doesn't have any other critical integrations like CI/CD, jira, etc
Why? We use github and its CI/CD system, but locally I still only need to add the git remote to work with it.
And I'm sure they'll continue to feel that way right up until the first time they experience "the" internet from a minority partition for more than a few days. I just hope that the distributed stuff is easy enough to limp along with if/when that happens.
No, they'll continue to feel like that even after that. GitHub being down once in a blue moon is more acceptable to the vast majority of users than having to cobble together your own nerdy distributed version of everything.
I was imagining something a bit more disastrous than that. A big enough solar flare could take parts of the planet offline for months. Years if they can't source enough replacement transformers. There are also political reasons that countries go offline.

Then it'll be up to the nerds who manage to cobble together their own distributed version of everything--even if it's a significantly reduced definition of everything.

If large parts of the planet lose their digital infrastructure for months, I really don’t think that “finding a good platform to host my code” is going to be one of my biggest problems.
Especially if the problem was due to a lack of electric grid as they're suggesting.
(comment deleted)
I think that how big those problems get is going to depend on how much critical infrastructure we can hack back into a working state despite the fact that it can't phone home, which is going to be a problem if nobody knows how to work offline anymore.

Or, if it's a political scenario, it may depend on how well we can coordinate en masse without the cut connection. If we can exceed a certain threshold then we'll have removed the incentive to cut it in the first place.

Even if that's a concern in a doomsday scenario, self-hosting gitlab is super easy and a good (some would argue better) alternative.
Self-hosting gitlab is not and has never been easy if you do it right, it's very heavy on resources and take lots of time and effort to upgrade. It's also extremely complex and has many moving parts. Stick to gitea or forgejo, they upgrade just by restarting the daemon. MySQL for the database if you want comparable ease of maintenance (same thing: upgrading between major versions requires replacing the binaries and restarting the daemon).
ok fine, ill thank the nerds in that case.
I doubt it, I don't think the distributed stuff is anywhere near ready. Instead it'll be time to kiss the ring of whoever manages to grab control during the gap.
You've described a scenario disastrous enough that my primary concern would be drinkable water.
At that point the enormously powerful central players like big tech and militaries and tax collectors will be more than incentivezed to use every remaining resource they have to bring everything back online and re-centralize power. And if they can't - society and power wasn't exactly more distributed in the distant past lol. Your local warlord/military dictator/whatever will probably not be supportive of nerds acting independently.
(comment deleted)
There is not actually a need to choose between a single centralised monopoly and going full-on techno-prepper and running all your own services in your garage as an individual . We could have intermediate points, such as having services run by professionals but based on portable open standards.
git is a great portable standard. But if I were GitHub, I'd make damn sure I didn't have issues, actions, and the rest of it be based on something users could just yank out in a portable format and take elsewhere though.

Portable VCS is simple. Portable anything with the integration everyone expects (issues connects to source which connects to builds which connects to releases) is hard. Git being so open and portable means it isn't a moat.

this should be on a wall somewhere
I've been keeping an eye on radicle[1] but the documentation for setting up a peer and web frontend is a bit complex. It seems to offer what you're describing: a "decentralized" Git frontend with issue tracking. Seems to be missing wiki functionality, however.

[1] https://radicle.xyz/

The GitHub wikis are actually git repos.
I forgot about that, I wonder if they were broken too.
> Too bad Git lacks a distributed bug tracker

Email and mailing lists?

That's the same as not having one.

Otherwise you can claim Facebook is distributed because you can email people links to Facebook pages.

But git has built in email features

And it was developed itself by email.

> That's the same as not having one.

That's the way the Linux kernel (the first Git repository) and Git [2] itself manage their codes. There's even a git send-email command, that prepare the commits as patches and send them following the using the correct template.

[1] Linux kernel, IIO subsystem: https://lore.kernel.org/linux-iio/

[2] Git mailing list: https://lore.kernel.org/git/

I agree that Git has more of a claim to this than Facebook, but that's kinda like saying a turtle is more of a car than a banana is a car.

Like, yes, it's true. Unlike a banana, turtles have 4 movement-enabling things, they use them to move mostly forward and backwards and not sideways, and other things can ride on them. It's probably more of a car. But it's not a car.

Git has no issue tracker. It's really not a controversial statement. The git community has common practices using something else to work around that, but if that's all you need to say "therefore git has X" then you can claim git has a CI framework because everyone and their dog uses GitHub. Which also has email integrations.

A CI infra just needs to communicate. It can communicate over email, like it does for the kernel.

A bug tracker is just assorted communication. One can easily build it over email.

You're just indulging in hyperbole for the sake of it. Nobody said git has an issue tracker in it.

> You're just indulging in hyperbole for the sake of it. Nobody said git has an issue tracker in it.

Yes they did. That's what this comment thread is about. https://news.ycombinator.com/item?id=42691624

(Unless you're splitting some really fine hairs about what "in it" means?)

They just mentioned the most common communication tool used with Git? You're the one splitting hairs here.
They mentioned them as a counterargument to "Too bad Git lacks a distributed bug tracker".

Which makes it a claim that those tools are git's distributed bug tracker.

A bug tracker and an issue tracker are basically the same term. So that's a claim that git has an issue tracker.

So when you come along and say "Nobody said git has an issue tracker in it." you are either wrong, or you're saying the words "in it" completely change the meaning of the sentence.

If it's the latter, that is a very unhelpful way to communicate, and is definitely splitting hairs. And honestly it's a strawman too because the comment you replied to wasn't using the words "in it". They were saying that you shouldn't say "git has" email. Which is a direct reference to the ancestor comment's claim. It was not hyperbole.

I'm not splitting hairs anywhere. I'm saying that the ancestor comment has the same meaning as "git has an issue tracker". That's not splitting. It's the opposite of splitting.

Is there a world in which GitHub used an open protocol for the social network part of their product like BlueSky's AT protocol[0]?

[0] https://docs.bsky.app/docs/advanced-guides/atproto

...this is an interesting thought exercise, thank you.
not p2p, but federated: https://forgefed.org (ActivityPub extension)

I believe Gitea has support for it, not sure to what extent.

Forgejo, a Gitea fork that I use, has support for it according to the page you linked. But the FAQ for Forgejo mentions it's on the roadmap so not sure how complete ActivityPub support is in Forgejo either.

https://forgejo.org/faq/#is-there-a-roadmap-for-forgejo

I only use my Forgejo instance for myself currently so I haven't looked at the ActivityPub features of it before.

Forgejo (Gitea fork) has been working for multiple years to add support for this. It will still take a lot of effort to finish, I doubt we will see anything usable this year.

Originally the plan was to PR the federation support to Gitea as well. I'm not sure if this is still the case, considering the rising tensions between the two projects and the fact that Forgejo is now a hard fork.

Yes, conveniences like:

- A canonical name and place on the web;

- Access policy enforcement (who can commit and when);

- The whole pull request thing, with tags, issues, review, discussion, etc linked to it;

- Code review, linked to said policy enforcement;

- Issue tracking, even as basic as what GitHub offers;

- A trusted store for signing keys, so commits are verified;

- CI/CD triggers and runners;

- A page with releases, including binary releases, and a CDN allowing to use the download links without fear.

This is way more than distrusted version tracking. Actually the above is not even married to Git; it could be as valuable with Mercurial, or even Perforce.

This is a large product, actually a combination of many potentially self-contained products. It should not be compared to Git, but rather to Gitea or BitBucket. Not all of this can be reasonably decentralized, though quite a bit can.

> - Access policy enforcement (who can commit and when);

Interestingly, what GitHub mostly enforces is where your branches point to. Not who can make commits. That's mostly because of how git works, not because of any grand design on GitHub's part.

It controls who can push commits to the main branch hosted by GitHub (and other branches if you want to configure that). You can have OWNERS files to control who can push commits touching particular parts of the tree, or who must approve such a push / merge (see "pull request").

Out of the box, git does not offer that, and this does require a single point of enforcement.

My point is that in git branches are just mutable pointers to commits. Tags are internally nearly the same, but socially they are meant to be immutable.

Anyone can make any commit they want in git. That includes merge commits, too. GitHub mostly lets anyone push any commits they feel like, too. (What restrictions are there on pushing commits is mostly to deal with denial of service and people being a nuisance.)

Where the policing comes in is in giving rules for how these pointers (aka branches) can be mutated. OWNERS files, PR reviews, CI automation etc is all about controlling that mutation.

See also the new-ish merge queues[0], which really bring out that difference: the merge queue machinery makes the merge commit of your approved PR branch with 'main', runs the CI against that, and iff that passes, moves the pointer that is 'main' to point to the newly created commit.

It's exactly the same commit (with exactly the same hash), whether it passes the CI or not. The only difference is in whether it gets the official blessing of being pointed to by the official 'main'.

It really speaks to the design of git, that conceptually the only thing they need to lock down is who can mutate this very small amount of data, these handfuls of pointers. Everything else is (conceptually) immutable, and thus you don't need to care about who can eg make commits.

[0] Really a re-implementation of bors-ng.

The above is why my company left mercurial years ago for git. Mercurial of 10 years ago was better version control than git (git still can't track a branch and how it changes) - but the rest of github is much better.
The git reflog tracks branches and how they changed. Is that roughly what mercurial has? (It’s been more than a decade since I switched from hg to git as well, so my remaining memory is minimal). GitHub also has an API for querying historical branch info, which is more permanent than reflog, though quite annoying to parse for that info if I recall right.
Hg names the branches and keeps the name. The other day I was looking at a sequence of commits trying to figure out where they came from and knowing the branch would have helped.

mg always kept history though. Git has always encougaged squashes and rebase to keep a linear history so that information was lost.

We all want history information to be lost. (Unless you are running a version control system that timestamps every keystroke.) Reasonable people may disagree on what information should be kept.
While true I still find it strange that git still doesn't address this which hg has always used to say why it is better. There is something to it that git fans who have never had good brancing don't know what they are missing.
- CI/CD triggers and runners;

I've used up 17h of CI time these two (slow) January weeks, for free, testing stuff across ~20 different OS/CPU combinations.

That's on just one "personal" project; a bigger dependency of that, of which I'm a maintainer, spends an order of magnitude more.

Can you (GP post, people complaining, not parent) blame us? Should we instead self host everything and beg for donations just to cover the costs?

People said that when GitHub got bought out, and only more people ended up moving there. It's really not fun to manage your own Git servers, and when things go down, they get fixed much slower than in-house hosted version of it.
I don’t know anyone beyond hobbyist hackers who want to set up and maintain this stuff for themselves.

As a professional software developer, I want tools that just work that I can rely on. GitHub 99.99% uptime is something I can rely on.

As a professional software developer, you rely on software written by those hobbyist hackers.

Whenever you do a clone or an npm install or apt get or pip install, etc...

You choose github because your dependencies chose git

Sure, but as you say those hobbyist developers aren't responsible for keeping a specific server up. They are 'just' writing some software.

(And even among professionals, there's a big difference between Site Reliability Engineering and Software Engineering.)

(comment deleted)
It was down for ~45 mins according to the linked page page, which would put it at 99.897% uptime for the month, assuming no other downtime.
They had some more downtime a few days ago, too. And that's just the one I happened to notice.
And if your central hub for your distributed vcs needs more than 2-3 9s of uptime for your service to be reliable, honestly you’ve done something really wrong in the design phase like using version control as a database.
A bloke called Linus Thorvalds created git for Linux development when the commercial service used for that ceased to be useful, for one non technical reason or another.

github basically shoves a webby frontend and workflows on top of someone else's work. That's all fine and good but github is not git.

As a professional IT consultant, I want tools, I use lots of other's and I also create my own and I also generally insist on hosting my own. I'm also a fair carpenter and several other trades. I have tools for all my trades and I look after all of them. That way I can guarantee quality - or at least reproducible and documented quality.

I'm sure you are fine with abrogating responsibility for stuff that you are not able to deal with - that's all good: Your choice.

EDIT: Sorry, forgot to say: "Yay cloud"

Linux development is also centralized. Instead of Github they use an email list that has patches sent to it. If that goes down, your change isn't going into Linux today.
(comment deleted)
I contend that if you submit a patch under any circumstances your change isn’t going into linux today.
You do know what's a mailing list right? You seem to be confusing it with GitHub.

A mailing list can go down and nothing would happen. The main point is to post patches to the maintainer. The mailing list is for a public record of things.

The only centralised thing is repo hosting on kernel.org. And that isn't the only official place, you can get the repo published on googlesource or GitHub, so it isn't exactly central enough.

I suppose that's true. In any case, though, getting a backup mailing list going is much much easier than something like GitHub, and you can always mail patches directly if maintainers allow it.
I believe he found the open source tools being used (CVS?) weren't good enough, and he started using a commercial closed source tool called "bitkeeper", which rankled the ire of the FOSS community who wanted to eat their own dogfood.

So Torvald's opted to "clone" the features of bitkeeper into an open source version he named 'git'.

That's the story I heard, no idea if it's true.

four 9's uptime https://uptime.is/99.99

Daily: 8.6s

Weekly: 1m 0.48s

Monthly: 4m 21s

Quarterly: 13m 2.4s

Yearly: 52m 9.8s

If you assume that "uptime" means all tools are available

https://statusgator.com/services/github

this appears to be 45 minutes in just one day

Incident with Git Operations 30m Jan 14, 2025 9:01 AM Down

Incident with Git Operations 10m Jan 14, 2025 8:51 AM Down

Incident with Git Operations 5m Jan 14, 2025 8:46 AM Down

Not much margin to hit four 9's left for the rest of the year.

> GitHub 99.99% uptime is something I can rely on.

Their enterprise level SLA is only 99.9% (measured quarterly) and the remedy is a 10% credit, increasing to a 25% credit if they drop below 99%.

Contrary to popular belief, sarcasm makes you harder to understand and is no longer cool
Counterpoint: it's a 45 minute outage once in a blue moon. Very small price to pay for the convenience of a centralized VCS with many features that aren't easy to reliably set up in standalone installations.
Once in a blue moon? It’s like, monthly at best.
Well, despite our intents, we may have established the moon might be blue then.
When was the last time you noticed? Not saw a HN post, but it actually stopped you from pushing commits or commenting on an issue or whatever?
Once a full moon maybe? There's one tonight.
Git is distributed. Distributed system does not guarantee 100% uptime or real time consistency. You can take the whole history with you and push to a different remote.
> Too bad Git lacks a distributed bug tracker

Not your point, really, but fortunately, git is easily extensible. This in-repo issue tracker is surprisingly feature complete: https://github.com/git-bug/git-bug. Anyone else given it a whirl?

It's this type of negativity ruining the internet. Nothing thoughtful to say, nothing to add, and hoping for failure. I hope everything is okay over there...
They are advocating for decentralized tools. That’s actionable and hoping to prevent failure.
There is no reason to host 'decentralized' tools besides regulation. It's considerably cheaper to use GitHub (or other alternatives like GitLab) than hosting your own and hiring people to maintain and support the solution. Their issue tracking system is very convenient for small teams too.
Exactly, hire a team of 3 and pay 500K in compensation, or spend 100K on a system that works and you get a support person to call in the event of an issue. The math is so simple.
Except you’re not considering the cost of when you can’t deliver something on time for a customer because infrastructure you don’t control is down.

You don’t outsource things that prevent you from doing your core competency.

And for most companies, building and managing an SCM is absolutely not their core competency. Your point is valid, but not in the way you're trying to convey it.
> building and managing an SCM is absolutely not their core competency.

Building their software is - Github being down is currently preventing that for many companies.

Nope, sorry. Github offers cloud and on premise offerings. If you choose cloud and your company can't handle a 45 minute service outage, that's just a bad purchasing decision. You do realize they make most of their revenue from on premise enterprise customers and that none of those customers were impacted? The solution was there the entire time but they can't force people to use it.
It's surreal to imagine most companies not being able to handle a 45-min service outage of a VCS to begin with. Sounds to me like a GitHub mandated break for all SE 's.
The costs would be trivial for the vast majority of Software Engineering companies. Talking about corner cases is useless as they often need a custom specialized solution anyways and wouldn't be using GitHub in the first place.
> It's considerably cheaper to us

This is not true. The cheapest option is to not have services that require servers to maintain. Git continues to work if GitHub is down. So do shell scripts when CI is down. So why can’t we have an issue system where the underlying data is text files in a git branch?

I understand at scale you can pay people to optimize a process for the larger team, but there is a ton of unnecessary fragility before getting to that scale.

If you're collaborating with a small group of people (or you're not running a huge amount of CI/CD) then you can make almost anything work. Once you get big it's another story entirely.
They want people motivated to design systems that can handle github going down. That doesn't strike me as negativity, and especially not negativity ruining the internet. It's not the most thoughtful thing in the world but it's a reasonable opinion, and most comments are also not the most thoughtful things in the world.
Makes me wish for the good old days of git-ssb
I mean I could very easily still push to another remote until it comes back? I do not feel locked in at all.
git was never really decentralized though. The whole system is aggressively a funnel towards central repositories, it's just that because it deals in whole repos, every git repo has the potential to be promoted to that role.

Nothing is built into git to let it actually run decentralized: there's no server or protocol where someone can register say, an identifying public key and then just have a network of peers all communicate updates to each other. It's even pretty damn unsafe to just run a repo through basic file-sync (I do this for myself with syncthing with bare repos in a special folder, which seems to work fine but I'm hardly loading it up to chase down why it doesn't).

If you have a local development setup, few hours of not being able to push should be a not problem most devs.

Except if you have a release planned but most don't at that time, statistically.

Problem is that people get comfortable with pushing to branch -> deploying in dev and testing from there.

Panic for a minute SSH rejected a key!
I just reset my ssh key lol.
They are going to get DDoS's by people trying to do this haha.
I guess it's a good time as any to setup a backup or check out alternatives that could mirror or standalone. Gitlab, Gitea, any others worth checking out?
Gitea works for us.
I use a fork of Gitea called Forgejo. It works nicely as well.

https://forgejo.org/

What drove you to the fork?
First I was using Gogs. Someone forked Gogs and made Gitea, because Gogs was under control of a single person and some other people found that frustrating.

https://blog.gitea.com/welcome-to-gitea/

I was using Gitea for a long time, and then someone forked Gitea to create Forgejo. At this time, my installation of Gitea was already out of date a bit because I had previously been manually building and installing Gitea from source. Soon after Forgejo was created, it landed in FreeBSD ports and then it became available in the FreeBSD package manager.

So at this point, and having read a bit about Forgejo and seeing that Forgejo was maintained by people with connection to Codeberg, I thought “hey I need to migrate my current Gitea setup anyway. Either to Gitea installed from FreeBSD packages, or to something else. I might as well try Forgejo.”

And that’s how I ended up installing Forgejo and I’ve stuck with it since.

Aha, makes sense. Thanks for explaining. I see that Forgejo was created at least partially due to concerns that Gitea was trending towards freemium as well. Good to know.
What's the point of having a mirror? It'll only lead to conflicts that'll be difficult to resolve once the main service you use is back up again.

For example, I doubt you would be able to easily merge any pull-requests or use the same CI/CD code for the same services without hacky solutions.

JFYI, the next version of gitea (which should come out in April) will have full GitHub mirroring functionality (which will let you set up a mirror and have it pull code/issues/wiki/PRs/etc. every few minutes). The current version can either migrate the full repo once, or mirror the code and nothing else.
I was going insane doubting my SSH knowledge, stopped short of creating new keys thankfully!
Same - but created new keys :) ... which also didn't work, and then I went to check... oh well.
I thought my account was hacked and ssh keys removed. Panicked a little bit and then double checked that my keys were still there.

Then went to github status and calmed down.

I briefly entertained the notion that I was fired and my ssh keys revoked. But I still had access to slack and email etc, so I banished that thought.
well on my side I'm the proud owner of a new ed25519 key.. The status page didn't update quick enough
Was trying to install homebrew on a device and kept wtfing because the clone step kept failing… thought I was crazy
I had just upgraded to Windows 11 last week, and for some godforsaken reason earlier today, I could SSH via WSL but not from the host OS even though they were both using keys served from the Windows OpenSSH agent! I'm just going to blame this service outage and hope for the best tomorrow.
I for a second thought was I getting fired
As someone who found out that way twice in the last two years, I also thought I had been let go. Anxiety is a hell of a drug.
Sorry for you, it should be illegal for companies to fire with no notice!
This is the case in many places, but companies would often rather pay you to not work for the notice period rather than have access to digital or physical company property after having been given notice.
Just the info I was expecting to see on HN. Unfortunately (or Fortunately?), a better source for these kinds of things than anywhere else.
It's just a hub at the moment.
Update: looks like the problem has been identified.

"We've identified a cause of degraded git operations, which may affect other GitHub services that rely upon git. We're working to remediate."

It's odd to me that there's nothing indicating an issue is ongoing on the github.com homepage
I thought it was my internet. Oh well, time to finish that game of Civ 6. Edit: and it literally came back up the moment I posted this.
Damn you azure! Or is Microsoft not even running their precious GitHub in azure
Who said it was an infrastructure failure?

There's innumerable causes of this kind of failure that aren't rooted cloud service provider shenanigans

(Admittedly the duration of the outage does "feel" like an infra outage)

lol I changed my ssh key 2 times until I figured out pubkey permission denied was an issue on their side
All checkmarks just turned green and my push went through. Snow day over.
Not sure if I am just noticing GitHub's issues more often as I am using their tools pretty much every day but their availability is kinda not great. Be it Actions failing or something "not core business" (read: git operations) but I can't remember a month in the past where I was not annoyed by any outage on GitHub
Agreed, I wonder what their downtime percentage is. If guess it's down on the order of one hour per month, so 1/1000.

Update: They promise >99.9% on a quarterly basis for enterprise customers - https://github.com/github/docs/blob/main/content%2Fsite-poli...

99.9% uptime corresponds to about 2 hours downtime per quarter, if my maths is correct. If that is indeed the guarantee, based on the experience at my company, GitHub has failed its promise recently (or is getting damn close). I recall 2 decent outages in the past few weeks alone. It's making me begin to doubt if GitHub's reliability is appropriate for an enterprise service.
I'm not sure whether the uptime guarantee is 99.9% independently or jointly for each service, i.e. if service A is down for 0.06% and service B another 0.06% but not at the same time, will this count as overall uptime >99.9% or <99.9%.

Rereading the SLA, it looks like Github can have each service feature like issue, pull requests, git operations be down 0.1% and still not reimburse. In your head you might not account separately for each feature, but Github does.

The whole product feels like it’s getting g progressively jankier.

The front-end is glacial nowadays and frequently has issues actually loading the page, actions frequently has some kind of panic attack and breaks or just grinds along a glacial speeds. The UX has gotten worse (why no merge-queue button?).

Agree with the second part. Github actually has merge queues, but only for the enterprise tier…
I wouldn't say glacial. If you want glacial try Jira. Mind-bogglingly slow.
Oh I know, I have to use both at work. God forbid you mis-click in Jira.

Navigating around takes so much time, it should probably have its own timesheet code.

I'm not sure if it's just me but I swear GitHub UI is one of the slowest web app I've used.
the incident has been marked as resolved; I am now able to push