HN Explorer

GitHub was down (githubstatus.com)

117 points by bombcar ↗ HN
The status page shows stuff about slack https://www.githubstatus.com but it seems authentication is down hard, can't pull, comment, or even view public repositories when logged in.

Edit: https://www.githubstatus.com is updated as of 19:08 UTC and everything is on fire

43 comments

[ 3.3 ms ] story [ 95.7 ms ] thread
athorax
Getting 404 on all github.com repos, oh boy lol
bombcar
I got a glimpse of some code on a reload of notifications, and then - poof.
adtac
We, uh, fixed the glitch^W^W all bugs
hobofan
Yup, I'm also seeing very sporadic issues with 404 pages and some API actions not being able to complete, though it seems mostly related to private repositories for me.
uncomputation
Seems intermittent. Sometimes a refresh will work to display files and commit history.
ezekg
Possibly related? https://github.blog/changelog/2023-11-03-multi-account-suppo...

Implementing that type of multi-account support likely had some big authz changes (I've done a couple of those projects before), and GitHub defaults to a 404 for authz errors instead of leaking information with a 403. Just a wild guess.

jrochkind1
The fact that https://github.com/ (which requires no specific authz check) seems to be up, and that some users report logging out and in again at least temporarily restores access, seems to support that hypothesis.

Still seems odd that no CI/QA would have caught such a widespread bug.

grepfru_it
Oh if you only knew the code base they are up against. Or in other words “what QA <laughing.emoji>”
netcraft
spot on:

> Update - We’re in the process of rolling back an authorization-related change that is causing 404s and other errors.

ezekg
Like I said, not my first rodeo for implementing a multi-account system on top of a single-account system lol.
hirsin
No, it was an authorization internals upgrade we were testing, not the account switcher.
ta1243
Nobody would deploy such a change on a Friday afternoon though
alfanick
It kinda makes sense though to do it on Friday afternoon, as on the weekend enterprises and most people just go skiing or do other things so less of customer base is affected in case of (unfortunate) outage.
ta1243
Even if that were a fact, then that's the one time github should not be updating. Their customers will be doing deployments on a Friday afternoon, thus require their repositories for their internal processes to work
paulddraper
What day is the best day to have downtime?
ta1243
Depends on your business. For my workflow we are happiest with risk for weekdays 0930-1130 and 1400-1600, and Saturday from 0500-1100, but it varies depending on the affected systems and workflows. If I'm doing work on one systems used by Department A then a Saturday is great. If it's on Department B though Saturday is the worst possible time. Department C's systems are mainly used from 2000-0700. Department D's systems are used 24/7, which is why we have geographic resilience.

But you don't get to choose this with XaaS. It goes down when it goes down, it comes up when it comes up. Our connectivity supplier sends us a notice that "circuit from X to Y will be going down from 0200-0500" and we have no choice but to accept it.

Typically updates are best done on Tuesday morning given that most people are awake and working and thus if there are problems they can fix it, but you wouldn't go ahead while the resilience is down (say due to a fibre cut, or a power outage)

[deleted]
(comment deleted)
[deleted]
(comment deleted)
mparnisari
This is nature's way of telling us to stop working on Friday, y'all.
casenmgreen
I run my own git server and users clone from that.

smug ;-)

rlizzo
first cloudflare... now github... goodbye internet
seneca
Is there a major service that has a worse uptime than Github? I can't think of one that feels less reliable.
[deleted]
(comment deleted)
bombcar
Logging out and back in seems to .... somewhat make it work? So maybe that could help.
vb-8448
I've tried multiple times + incognito while trying to access a public repo and still randomly get 404.

I bet there is something wrong with caches.

layer8
Good thing that Git is decentralized.
derwiki
I mean yea, I'm still happy that I can commit locally. With a centralized VCS like Subversion, I wouldn't even be able to do that.
lakomen
I left that platform when M$ took over. Only have an account because of the many stubborn people still clinging to it and I have to tell them when I find a bug.

Especially the forced 2FA.

Whatever will you people do now, that you can't use co-pilot to code for you? It's the end of your career I'm sure. I'm sorry but I can't help being cynical on HN especially.

mparnisari
> Especially the forced 2FA.

How dare they protect their users?

caeruleus
Notice that there were issues with Slack notifications only before:

> The delayed Slack notifications should be fully processed in about 30 minutes > (Nov 03, 2023 - 18:25 UTC)

Exactly 30 min after:

> (Git Operations|Issues|Pull Requests|Actions|API Requests|Codespaces|Packages|Actions|Pages) is experiencing degraded performance. We are continuing to investigate. > (Nov 03, 2023 - 18:55 UTC)

Might be coincidence, might not be...

AndrewOMartin
What we need is something like github but decentralised somehow, if such a thing is even conceivable.