Again? Last time a serious incident happened was just 11 days ago [0] with reports of actions going down again.
Like I said, I expect them to go down once a month guaranteed. But this is like 6 times in one month. Now I expect something in GitHub to go down twice in one month.
Not looking good at 'going all in' and 'centralizing everything to GitHub' as I predicted [0].
That can be complicated in itself, but GitLab's operational overhead is minimal ( both their omnibus and Helm deployment methods are highly automated) and IMHO it would be hard to have availability lower than Github's is at the moment ( last few months).
We run our CI Cd on GitHub and all our code is there and I’ve never felt there outages really affect things much ever (except the one time actions was down proper for a few hours). If anything it’s more stable than our ragged old Jenkins crap, or our airflow clusters. I’ll happily trade them to GitHub if they can take care of it.
Of course we did make sure we can still deploy our code if GitHub was down but that sounds like a good redundancy practice anyway so why not?
GitHub has ruined my lunch or dinner time at least once a week or so for the last three months in some capacity. I am absolutely fucked off with it now.
Centralising on this infra was a stupid technical decision.
No I have to field a thousand questions about why nothing is working from just about every part of the organisation. Then spend several hours after it has recovered assigning blame back to the engineers when they do something stupid rather than blaming it on the GitHub outage.
It reminds me of a time many years ago - before SaaS days - that a co-worker was similarly disrupted due to unreliability of a vendor system...and his anger towards internal senior staff for (what many obviously) considered to be a poor decision to go with said vendor. So my buddy was tempted to automate email forwards to a partiocular senior leader...so they somewhat feel the pain of all the messages being sent on a partoicular topic...plus my buddy acvited his email's out of office message to both assure folks that he was aware of the issue and to contact the senior leader most responsible for the poor vendor decision...again, to share the paion. Ultimately he didn't do that direct of a blame for fear of getting fired...but he did implement the out of office in a general fashion at least to lessen/avoid him having to answer questions about the numerous outages, etc. He didn't name any names...but some savvy folks in the org who understood the management hierarchy eventually figured things out and knew to start pionging that senior leader...and my friend didn't get fired...So there could be some mildly automated ways to lessen disruptions to your life. ;-)
This is like the third downtime in a month. Hosting your own infra is easier than people think, but it is completely trivial if weekly downtime is the expectation.
Maintaining gitlab has been nothing more than sudo apt-get update && sudo apt-get install gitlab-ee.
Their releases often have security vulnerabilities though. But other than that, even the community edition is packed with features, it's a no brainer for me ;)
So is there a general idea for why so many outages?
I’ve used GitHub for 10+ years and don’t remember these outages historically. I’m sure there’s more usage now, but usage has been high for a long time.
My initial guess is Microsoft monkeying with the tech stack to use inferior solutions from their own stack rather than proper architecture design and evaluation.
A slightly more generous interpretation — GitHub was pretty much stagnant (for better or worse) until Microsoft took over and started adding new features and making improvements are a pretty considerable rate. This has led to more complexity and bugs.
I at least have found the recent improvements and additions pretty great — GitHub’s newish PR review UX is way ahead of BitBucket (where my employer currently hosts our codebases) for example.
My employer just moved from Bitbucket (on-prem) to Github (cloud). I have to say, as much as I hated the Bitbucket review system, there's a bunch of things we run into with Github that just are surprisingly lacking compared to Bitbucket.
- In Bitbucket, you can reply to _any_ comment as a thread, rather than a quote-reply, making conversations easier to follow. Github only lets you do this if you're replying to a comment on a file.
- Though this has been added very recently as a beta addition, the lack of a file tree made large reviews in our monorepo hard to follow.
- If someone force-pushes, the UI doesn't tell me what commits were removed; just the new branch-head.
- Our devops team has had to make it so we can re-trigger our CI (or add additional pipelines) via comment-commands, while in BitBucket we were able to add custom buttons to the PR menu. This means there's a lot of noise of comments that just say "/retrigger-ci" or some such.
- This one is not Github's fault per-say, but because we used to add our users via LDAP, we used to just be able to type <first-initial + last name> to add a user to a PR, but now because everyone has bespoke usernames, it's a lot more annoying to add people because I now have to also remember their username.
- In our repo, we work with lots of release branches for maintenance (committing to `develop` is not rare, but quite frequently we're adding features to old branches), and Github's PR view does not show the target branch at a glance, which can be annoying if someone has multiple PRs to cherry-pick a commit across branches and you don't know which PR you're clicking on.
- Being a cloud solution, we can't add pre-receive hooks. We previously used to reject commits at push-time that didn't follow the format we use for time/issue tracking. Now, we have to enforce this in CI which means rebasing/force pushing to fix it. While pre-commit hooks can help, this depends on every developer keeping their hooks up to date, which is a challenge.
That said, Github's "review" system, rather than limiting you to only individual comments, is great for email noise.
I think your answered your own question with saying you have a monorepo on GitHub. Like it’s not designed for that at all? That’s the main reason I’m not yet pushing for monorepos in our org, because it looks like it’ll be inviting trouble if we are still using GitHub for it.
Also you’re not doing trunk? I mean I get it it’s hard but sounds like a no brainer thing to stop doing as soon as possible..
The PR workflow certainly isn't, but we're far from the only company using them (Dropbox comes to mind). It's surprising to see it not fully supported.
> Also you’re not doing trunk? I mean I get it it’s hard but sounds like a no brainer thing to stop doing as soon as possible..
No, unfortunately not. Trust me, you're preaching to the choir :) But unfortunately, long-lived branches is the workflow we're stuck with at the moment (we constantly merge commits forward, but it's a manual process). That said, even if we were using short-lived feature-branches, you wouldn't be able to know from the PR view if a commit was to a feature branch or not, so the criticism still applies.
> - Our devops team has had to make it so we can re-trigger our CI (or add additional pipelines) via comment-commands, while in BitBucket we were able to add custom buttons to the PR menu. This means there's a lot of noise of comments that just say "/retrigger-ci" or some such.
Job re-runs are more than one click away unfortunately. But I'd be curious, outside a failing job due to flakiness or GitHub instability; for what other reason do you use this functionality?
The big one that I can think of is triggering our "packaging" pipelines. We have some packages published separately from our main releases (which are not published via CI), so we have to trigger separate pipelines for them. We could _probably_ have some kind of filter based on directories, and honestly I'm not sure why we don't (probably so it's deliberate).
Maybe the gh command line can be useful for your use-case. Although you might want to keep the comment trigger style for historical/audit reasons, or if non-devs are allowed to trigger those downstream jobs as well.
41 comments
[ 3.1 ms ] story [ 94.9 ms ] threadLike I said, I expect them to go down once a month guaranteed. But this is like 6 times in one month. Now I expect something in GitHub to go down twice in one month.
Not looking good at 'going all in' and 'centralizing everything to GitHub' as I predicted [0].
[0] https://news.ycombinator.com/item?id=31023941
Lots of talk about moving stuff on-prem.
That can be complicated in itself, but GitLab's operational overhead is minimal ( both their omnibus and Helm deployment methods are highly automated) and IMHO it would be hard to have availability lower than Github's is at the moment ( last few months).
Of course we did make sure we can still deploy our code if GitHub was down but that sounds like a good redundancy practice anyway so why not?
Centralising on this infra was a stupid technical decision.
GitHub is still a no brainer for me.
GitHub downtime is something I can just tell managment and prepare things to put it out of high critical path.
Like I would not pull images from GitHub on my k8s cluster.
Gitlab works absolutely but you know I don't want to do gitlab updates, do backup, do regular backup/restore test, deleting backups etc.
I prefer to create new things.
Their releases often have security vulnerabilities though. But other than that, even the community edition is packed with features, it's a no brainer for me ;)
You also need to do backups, try restore, you might need to announce a maintenance window.
And for what? Getting stress from mgmnt when it breaks? With a GitHub downtime that's just it. No one cares, everyone accepts it as given.
I’ve used GitHub for 10+ years and don’t remember these outages historically. I’m sure there’s more usage now, but usage has been high for a long time.
My initial guess is Microsoft monkeying with the tech stack to use inferior solutions from their own stack rather than proper architecture design and evaluation.
I at least have found the recent improvements and additions pretty great — GitHub’s newish PR review UX is way ahead of BitBucket (where my employer currently hosts our codebases) for example.
- In Bitbucket, you can reply to _any_ comment as a thread, rather than a quote-reply, making conversations easier to follow. Github only lets you do this if you're replying to a comment on a file.
- Though this has been added very recently as a beta addition, the lack of a file tree made large reviews in our monorepo hard to follow.
- If someone force-pushes, the UI doesn't tell me what commits were removed; just the new branch-head.
- Our devops team has had to make it so we can re-trigger our CI (or add additional pipelines) via comment-commands, while in BitBucket we were able to add custom buttons to the PR menu. This means there's a lot of noise of comments that just say "/retrigger-ci" or some such.
- This one is not Github's fault per-say, but because we used to add our users via LDAP, we used to just be able to type <first-initial + last name> to add a user to a PR, but now because everyone has bespoke usernames, it's a lot more annoying to add people because I now have to also remember their username.
- In our repo, we work with lots of release branches for maintenance (committing to `develop` is not rare, but quite frequently we're adding features to old branches), and Github's PR view does not show the target branch at a glance, which can be annoying if someone has multiple PRs to cherry-pick a commit across branches and you don't know which PR you're clicking on.
- Being a cloud solution, we can't add pre-receive hooks. We previously used to reject commits at push-time that didn't follow the format we use for time/issue tracking. Now, we have to enforce this in CI which means rebasing/force pushing to fix it. While pre-commit hooks can help, this depends on every developer keeping their hooks up to date, which is a challenge.
That said, Github's "review" system, rather than limiting you to only individual comments, is great for email noise.
Also you’re not doing trunk? I mean I get it it’s hard but sounds like a no brainer thing to stop doing as soon as possible..
The PR workflow certainly isn't, but we're far from the only company using them (Dropbox comes to mind). It's surprising to see it not fully supported.
> Also you’re not doing trunk? I mean I get it it’s hard but sounds like a no brainer thing to stop doing as soon as possible..
No, unfortunately not. Trust me, you're preaching to the choir :) But unfortunately, long-lived branches is the workflow we're stuck with at the moment (we constantly merge commits forward, but it's a manual process). That said, even if we were using short-lived feature-branches, you wouldn't be able to know from the PR view if a commit was to a feature branch or not, so the criticism still applies.
Job re-runs are more than one click away unfortunately. But I'd be curious, outside a failing job due to flakiness or GitHub instability; for what other reason do you use this functionality?
Anyway, hope that answers your question :)
https://news.ycombinator.com/from?site=githubstatus.com