151 comments

[ 0.18 ms ] story [ 140 ms ] thread
Issues seems to be completely broken, just getting unicorns.

Interestingly the API still works for creating an issue (but webhooks weren't fired).

Welp, workday over in Europe early!
On the bright side, they were pretty fast on the Incident Declaration this time.
We're starting to have the conversation of "do we need to replace GitHub as our CI/CD pipeline?"

No idea if/when we decide to pull that bandaid but I think once we can't merge a hot fix into main because of GH being down, it'll be a big org push to replace it.

What's everyone else using? Self hosted Gitlab? Gitea?

It's not the most elegant thing ever, but I've been using (self-hosted) WoodpeckerCI and it's been dead simple. It doesn't have many bells and whistles, but honestly I think that's good. Forced me to put complex behavior in better systems as opposed to doing bash scripts in YAML.
>What's everyone else using? Self hosted Gitlab? Gitea?

Self hosted github enterprise and runners.

1) Issues showed unicorn. Reloading page.

2) Issues showed Not found.

Anyway, GitHub Actions for building new release has progress. Good enough.

how long can they survive like this?
As the saying goes, 'No one gets fired for choosing Microsoft solutions'.

I've talked to a number people in IT and commonly found they use Linux or Mac at home, as their daily, and Windows only at work.

Talking one of the IT works in the company I work for, they were forced to move from GitLab to GitHub because the CTO wants Microsoft everywhere. When all you know are screwdrivers, a personal will gladly use them as a hammer, punch, ....

Back in my IT days, we were pushing for marbled solutions. Example, 50% of the computers have antivirus brand Y and 50% have antivirus brand Z. Or the Windows servers used a different antivirus solution than the desktop. I don't see this anymore in the industry. One basket and all eggs go inside.

Irony, I'm updating code so our main products no longer require Windows as the host OS.

You have to pay me to use Microsoft products and solutions.

It's getting to where we should just post when it's working fine, to reduce noise.
If you are required to use a Microsoft tool is Azure DevOps now a better choice? Our Azure hosted instance rarely (if ever) has had an outage that effects our ability to work.
Can anyone shed some light why GitHub is having so many problems lately?
As I said 4 days ago GitHub is in chronic operational decline which is killing trust in hosting repositories [0]

You can use as many AI agents as you want (it won’t solve the problem), but with that many outages ever since getting bought by Microsoft, it is more than enough time to give them to change. They don’t care and there is no CEO of GitHub to help either.

A self hosted instance would have much better uptime than GitHub and GitHub Actions at this point as I predicted 6 years ago. [1]

[0] https://news.ycombinator.com/item?id=49276571

[1] https://news.ycombinator.com/item?id=22867803

I had a lot of goodwill for GitHub but I think today is the tipping point.

Looking at a unicorn page, I feel this lingering hope that it's transient (like it usually was in the old days) but my mind reassures me it's probably going to be a long full outage again.

The hope is dead.

I think the same. But the GitHub CLI is the biggest block for me. It is a killer feature. No other tool is close. To self-host the forge is easy. To replace gh in my scripts and agent workflows is not.
Any good LLM should be able to come up with the equivalent git and curl commands fairly easily.
The Unicorn page rubbed me the wrong way. Thankfully I was able to revert changes to my prod instance using another approach. But I count on GH for reliability.
This is getting ridiculous. Issues, PRs, actions... nothing's working
Githubs a great advert as to why you shouldn't use Azure. What an utter mess its become since they took over. Their argument that its scaling issues due to increased traffic just makes them and Azure look incompetent.
So whats the alternative? I'm thinking I'll migrate my company to something self hosted, but what about personal projects?
Why haven't we all moved off GitHub yet?

Or let me ask a more useful question: if we were to move, then to where? Ideally somewhere new (not Gitlab)

Ah, so that's what the agent meant by summarizing a documentation page as "Unicorn!" I'd thought it was a very colorful hallucination (it's just GitHub's (unicorn-themed) failure page).

    Gathered details on embedding request syntax from
        Unicorn! · GitHub
I was querying docs to figure out how to set up RAG to query docs locally. Alas.
So glad I moved my OSS org mostly over to a self-hosted Forgejo + Woodpecker CI. I get that we are operating on a completely different scale but I swear the GitHub outages are far too regular than they ought to be.
> We are seeing an approximate 20% error rate across numerous experiences including Pull Requests, Issues, and others. Investigations are currently under way and we will be posting updates as they become available

Well it's 100% down for me, so does this mean 20% of all users are fully down?

If you load an issue page, you'll see 1 failed request to:

    /project/product/issues/<number>
And sure, that's what you care about, but consider the working requests to:

    /in-product-messaging/copilot-budget-request-banner
    /in-product-messaging/code-scanning-ai-findings-preview-banner
    /github-copilot/chat
    /_private/browser/stats
    
See, only 20% requests failed!
And I just wondered why I couldn't update my nix flake lol
Will our hug of death now make githubstatus.com crash