OT but why did it capitalize the title like that? I submitted "GitHub is having issues". I don't see that for other submissions (unless people manually edit them afterwards).
EDIT: Before the change to "GitHub was having issues" it had converted my original submission from "GitHub is having issues" to "GitHub Is Having Issues".
There are some rules for when this happens and when it does not. Don't know what they are exactly. Maybe the camelcase in GitHub may have triggered. Anyway, as you say, afterwards you can edit the title and give the desired case format.
From what I can tell from looking at older submissions, it might be doing it if the title is 4 words or less. Not sure why that would be the case though..
BTW you can see changes that occur like this on https://hackernewstitles.netlify.app/ - it seems like the automated title casing is being overridden by mods later on, perhaps.
Aren't these pretty heavy duty and any issue sticks very prominently ?
A ton of companies are using these tools literally 24/7, at most of my jobs Github/Gitlab being down meant most of us would just be spending half of our time faking working (rare are the people who could just do "offline" dev for half a day without any external input/output)
Just like every company, they sometimes have things break, push out code that doesn't behave as expected, hit limits on various services they weren't aware of, etc.
The complexity of technical systems is massive especially when they are constantly changing. It's not helped that these companies have high availability requirements and need to be online 100% of the time even when making updates.
I love github as a product but for the past months, issues are a weekly occurance, often multiple times a week, and always in the middle of the day (Europe).
GitHub were pretty open about having issues with Bitcoin mining abusing the free compute available via Actions. I think it's an ongoing battle and one which is having a wider platform impact
Cue the age old argument chain about outsourcing responsibility vs having control of downtimes.
I’m tired of them now, anyone using hosted solutions should know this by now and they’ve made that choice.
Personally I believe people underestimate how easy it is to host your own services.
But I’m sick of having the conversation over and over, if people aren’t willing to investigate then, what’s the saying: “you can lead a horse to water but you can’t make them drink”
But the argument isn't static. GitHub has become much more of a target to large-scale attacks as it's grown in popularity and has become mission critical to so many companies and projects.
Security risk isn't a one-and-done decision. It's a changing landscape and, arguably, there are far too many eggs in the GitHub basket for this not to be something people might want to think twice about now.
GitHub also arguably just has a good product that's a good fit for a lot of use-cases, regardless of if there's better options.
Yeah, you might be able to self-host gitlab/gitea/etc but their free offering for organizations is so good that for my 30 commits/week open source project, I can deal with half an hour of downtime once in a while.
Combined with free hosting, it's really not that much of a stretch that people are willing to put up with this, especially as there have been periods of time with less outages before, though that's been a while now.
self hosted won't insulate you from your VCS instance failing.
I used to work at a medium-sized company with Gitlab 8 years ago, it was running into several issues and required frequent maintenance.
To give another data point, in my company we're running Gitlab (we started 3,4 years ago maybe?) with some pretty heavy use of CI and we've had no issues, even with automatic upgrades enabled (we have tested backups, we're not crazy).
Confirmed, on a ubuntu host GitLab is actually the product that I have the best self hosted experience with. Apt packages running their Chef Cookbooks always impressive!
Well going all in on GitHub / GitHub Actions now really sounds like a bad idea. Even some others had issues [0] and had no external backup plan to counter GitHub's regular outages.
The best example to learn from is ReactOS who while has moved to GitHub they at least have a self-hosted backup just in case GitHub falls over. [1]
For the ones who are not on GitHub and are self-hosting (OpenBSD, Mozilla, Linux Kernel Project, GNOME, xfce, wireguard) I don't hear any complaints from them.
The UI is much nicer than gerrit (or at least the gerrit UI I knew from 5-6 years ago not sure if they have changed). It’s more “GitHub”-y. It’s a bit confusing at first but once you get into it it’s pretty damn powerful.
Side note, I honestly seriously think we need much better tooling than just these 2 for code reviews.
Gerrit has gotten a new UI in the last few years as far as i know. I wouldnt nesecarily describe it as pretty, but its my personal favourite review system
I'm building a much better code review for tool for GitHub, check out https://codeapprove.com
It combines concepts from GitHub, Gerrit, and other good review tools out there to make it easy and fast to reach consensus (which is what it's all about).
If you want to hear more you can email me at sam at habosa dot com
Not even that necessarily, control of downtime is also worth something. i.e. lots of downtime is tied to system changes/updates, especially in a conservative setup, which you can schedule as appropriate on a self-hosted setup.
Someone said the same thing to me more than a year ago [0] and here we are the same issues once again.
This is why it makes no sense to go 'all in' or try to "centralize everything to GitHub". At least (if you're a company or an organisation) have a self-hosted backup and don't jump into the 'goin all in' hype train.
I completely agree with you, to reiterate what I said last time we had this discussion 3 months ago:
> I wouldn't call it prudent to go all-in on any cloud service [0]
I've never used Github as anything more than a managed git service with a nice web interface. At work, any time the topic of moving our CI/wiki/project management/issue tracking onto GitHub has come up, I've been vocally opposed.
If GitHub went down, and didn't come back up, it would be trivial to migrate to Gitlab/CodeCommit/Bitbucket. The most time consuming thing would be setting up new web hooks for our CI.
I'd wager that there's more self-hosted CI's running code from self-hosted repositories than from GitHub (however you measure it: e.g., lines of code, build time, etc)
All the time. It’s not even that reliable when it’s up. Total embarrassment for us when we have to explain to management.
If I started again I’d probably use something standalone, light weight and 100% self hosted ie GoCD. I’d change the operating model to keep GitHub as a collaboration place but not have any dependencies on it for pipeline. We have other issues as well like the GitHub flow mod being inadequate for maintaining stability on high traffic repos too. I’d pull from master into a production branch and run that off line entirely.
From what I heard it was fine until Microsoft bought it and decided to bring over .net code (and people) from azure devops to replace a bunch of the pre-existing github actions stuff for some reason.
I read this as more a situation where an existing architecture got stuff shoehorned in for non-technology reasons.
I don’t think there’s a problem with .NET inherently. But swapping out components for new tech stacks for no user or tech reason is likely to introduce bugs and at best will negatively impact schedule.
Although it would be interesting to hear any new company that chose .NET over other alternatives. I really likes how Spolsky explained why they used the MS stack [0] to build stackoverflow, among other things.
I'm assuming that the fundamental flaws* of GoCD haven't been fixed, based on the knowledge that in 2015 they (ThoughtWorks) still hadn't addressed them, and I think 5-6 years isn't long enough for them to be addressed.
*Pipelines, as modeled in GoCD, were so inflexible that any kind of flow resulted in having many, many, pipelines configured. To the point that the number pipelines you operate and maintain is actually the multiple of any fork in logic you need.
Hate to pile on GoCD. But I second this. In 2015 they were a contender when Jenkins was showing it's age and Gitlab was in its infancy. Development was fairly stagnant and felt 'closed' to external contributions.
Very rarely, and (so far) not in any meaningful way. Having to wait a few hours for a CI process isn't a problem 99% of the time. Deploys are affected by issues on AWS and Azure far more often than GH in my experience.
I'd probably change my mind if I ever get unlucky enough to need to deploy a critical hotfix at the same time GH is unavailable though.
If you're OK with waiting multiple hours for CI, I suppose these issues wouldn't affect you very much. Where I work we make relatively small PRs and deploy to production tens of times per day. The recent outages have been a nightmare for us.
Not the OP but in a continuous release process you treat main (master) as “the source of truth” and merges to main trigger a new release to production.
At the company I work at, we have about 160 full time engineers with about half our repos on continuous release. We average right now somewhere around 50-55 releases per day, with some hot repos getting up towards 20 releases per day on their own.
The point of small releases is that it makes rollbacks super easy (you always know what broke), roll forwards easy (you can easily test post rollout to find out if things aren’t functioning as expected), and you can mentally move on to your next task sooner because your code gets released when you want it released- not when the next “big develop release” happens.
To pull this off requires a few things. You need good tests that run against PRs. You need fast deploys (under 10 minutes is a good starting point, but get them faster if you can). And, ideally, you have ephemeral environments for each PR so that changes can be tested fully in isolation without requiring a shared staging environment.
Devs develop locally. Can test their code (manual test and most automated tests).
They push and open a PR. They can deploy ephemeral environments with any dependent microservices spun up along side. This is an in house tool we developed that builds and deploys ephemeral docker, CDK, and terraform based apps.
We then run CI automation against their PRs. This can include E2E tests courtesy of these ephemeral environments.
Once their PR is green and signed off, they merge to master. We’ve done some wizardry behind the scenes so that their code is live in production in around 3 minutes.
They verify their changes in prod and we run any automated regression tests against prod.
Next person merges as they see fit.
That’s it! It’s not easy to get to that point, but once you’re there it’s a great process. We’ll have capacity issues once we get up to 30 or 40 releases per day in a repo, but I’ve got some ideas about how to work around that.
Where I work we make relatively small PRs and deploy to production tens of times per day.
We do that, sort of. Obviously we're not PR'ing to get things in to production. Changes go through QA, Staging, UAT, etc first. The final point is that they go to prod though, and if a few hours of changes end up queued to all go at once that's fine.
If not seeing a change in production for a few hours counts as "a nightmare" for you then I certainly don't envy you. That must be incredibly stressful. How do you do good work under that amount of pressure?
Maybe these are just a bunch of coincidences, but I have a feeling that GitHub is being poorly run recently. Apart from quite frequent uptime issues, it seems that Github Actions is pretty half-baked and they completely botched the migration of Dependabot (for example, take a look at this issue and especially, how it was handled: https://github.com/dependabot/dependabot-core/issues/3253)
A shame they removed all their uptime graphs on the status page, since it's much harder to get a read on things now.
However, scrolling back through the status page's history for the last year shows that there between 6 and 12 incidents every month, and it often hovers at the high end of that range.
That has to be an utter misery for anyone who's on call.
> A shame they removed all their uptime graphs on the status page, since it's much harder to get a read on things now.
I've started using that (uptime/performance/latency graphs being removed from status pages) as a signal to for when companies feel embarrassed over their quality of service but don't have enough interest/resources/profits to fix it.
It doesn't seem like Microsoft should have any of these issues. At most "interest", but even that should be a pretty easy sell for a service as massive as github.
...so that customers start tracking it themselves, so that everyone can waste time on discussing their differences in numbers and measurements, and who is right and who is wrong.
I would bet on it being related to their identity server. It seems to almost work in browser(s) where I was signed on - but in a new browser, it gives HTTP 500 after SSO.
We're looking at switching to self-hosted runners, so I'd be very interested in hearing if it would shield us from some of Github downtime.
We're also looking into Github enterprise for the IP allowlist feature which unfortunately means we'll have to migrate our whole CI/CD to self-hosted since Github-hosted runners don't work with this feature enabled.
Things like this make me want to have more free time to start my own paid CI service. I can likely start from a few languages I know and take it from there.
I wonder how hard it is to write your own CI/CD though? Anybody has thoughts on this?
If you Google around you’ll see a crap ton of companies trying to do this.
IMO the only “serious” threat to Jenkins are platform integrated services like Gitlab CI/CD, GitHub Actions, Google CloudBuild etc. Everybody else seems to continue to use Jenkins.
I'm not scared of competition, I'm mostly wondering how viable and profitable such a business would be. I happen to believe people would pay for good specialized CI (say, Elixir-only, Rust-only, Golang-only etc.) but you're also quite right that the real value-add seems to be in integration.
As long as you don't provide a huge degree of additional value you will be more likely to lose money through free plans which are needed to onboard people then you are to make money.
While specialized CI sounds great you would need a good general purpose CI which also sports specialization. Ort people will need to change CI once they e.g. need to build and link against some FFI. Or have some project in another language. Which is quite a bit reason not to just it even if you currently only need specialized CI.
Plug: after almost a full year of hosting alone (for $50) I recently started using srht's builds, somewhat equivalent to github actions. It's been awesome so far. Look at the downloadable coverage.html: super easy to set up!
95 comments
[ 3.7 ms ] story [ 169 ms ] threadEDIT: Before the change to "GitHub was having issues" it had converted my original submission from "GitHub is having issues" to "GitHub Is Having Issues".
https://status.azure.com/en-us/status
Reminds me of the time when Twitter had an outage, because an FB outage made people go on Twitter to ask whether FB is down
A ton of companies are using these tools literally 24/7, at most of my jobs Github/Gitlab being down meant most of us would just be spending half of our time faking working (rare are the people who could just do "offline" dev for half a day without any external input/output)
Slack could be down it had less impact overall.
[1] https://git-scm.com/book/en/v2/Distributed-Git-Distributed-W...
Looks like some code issue from their side.
The complexity of technical systems is massive especially when they are constantly changing. It's not helped that these companies have high availability requirements and need to be online 100% of the time even when making updates.
Makes me want to do embedded programming.
Time to reset the counter once again. Now everything there is degraded performance. Oh dear.
Consider a self-hosted alternative or have one as a backup like I have said before.
[0] https://news.ycombinator.com/item?id=27192869
[1] https://news.ycombinator.com/item?id=27172443
[2] https://news.ycombinator.com/item?id=26666843
[3] https://news.ycombinator.com/item?id=26667101
I’m tired of them now, anyone using hosted solutions should know this by now and they’ve made that choice.
Personally I believe people underestimate how easy it is to host your own services.
But I’m sick of having the conversation over and over, if people aren’t willing to investigate then, what’s the saying: “you can lead a horse to water but you can’t make them drink”
Security risk isn't a one-and-done decision. It's a changing landscape and, arguably, there are far too many eggs in the GitHub basket for this not to be something people might want to think twice about now.
Yeah, you might be able to self-host gitlab/gitea/etc but their free offering for organizations is so good that for my 30 commits/week open source project, I can deal with half an hour of downtime once in a while.
Combined with free hosting, it's really not that much of a stretch that people are willing to put up with this, especially as there have been periods of time with less outages before, though that's been a while now.
Github hasn't been reliable for a while now.
P.S. Most people here attribute being a "hacker" to curiosity.
The best example to learn from is ReactOS who while has moved to GitHub they at least have a self-hosted backup just in case GitHub falls over. [1]
For the ones who are not on GitHub and are self-hosting (OpenBSD, Mozilla, Linux Kernel Project, GNOME, xfce, wireguard) I don't hear any complaints from them.
[0] https://news.ycombinator.com/item?id=26301523
[1] https://github.com/reactos/reactos#code-mirrors
I dont know reviewable so it might have more sauce than the recommendations
Side note, I honestly seriously think we need much better tooling than just these 2 for code reviews.
It combines concepts from GitHub, Gerrit, and other good review tools out there to make it easy and fast to reach consensus (which is what it's all about).
If you want to hear more you can email me at sam at habosa dot com
This assumes that your self-hosted Git service will have better uptime than Github.
Piece of cake. But it will also have better performance.
This is why it makes no sense to go 'all in' or try to "centralize everything to GitHub". At least (if you're a company or an organisation) have a self-hosted backup and don't jump into the 'goin all in' hype train.
[0] https://news.ycombinator.com/item?id=22868406
I completely agree with you, to reiterate what I said last time we had this discussion 3 months ago:
> I wouldn't call it prudent to go all-in on any cloud service [0]
I've never used Github as anything more than a managed git service with a nice web interface. At work, any time the topic of moving our CI/wiki/project management/issue tracking onto GitHub has come up, I've been vocally opposed.
If GitHub went down, and didn't come back up, it would be trivial to migrate to Gitlab/CodeCommit/Bitbucket. The most time consuming thing would be setting up new web hooks for our CI.
[0] https://news.ycombinator.com/item?id=26302232
I'd wager that there's more self-hosted CI's running code from self-hosted repositories than from GitHub (however you measure it: e.g., lines of code, build time, etc)
If I started again I’d probably use something standalone, light weight and 100% self hosted ie GoCD. I’d change the operating model to keep GitHub as a collaboration place but not have any dependencies on it for pipeline. We have other issues as well like the GitHub flow mod being inadequate for maintaining stability on high traffic repos too. I’d pull from master into a production branch and run that off line entirely.
I don’t think there’s a problem with .NET inherently. But swapping out components for new tech stacks for no user or tech reason is likely to introduce bugs and at best will negatively impact schedule.
Although it would be interesting to hear any new company that chose .NET over other alternatives. I really likes how Spolsky explained why they used the MS stack [0] to build stackoverflow, among other things.
[0] https://channel9.msdn.com/Blogs/Seth-Juarez/Joel-Spolsky-Tal...
Don't. Just don't.
I'm assuming that the fundamental flaws* of GoCD haven't been fixed, based on the knowledge that in 2015 they (ThoughtWorks) still hadn't addressed them, and I think 5-6 years isn't long enough for them to be addressed.
*Pipelines, as modeled in GoCD, were so inflexible that any kind of flow resulted in having many, many, pipelines configured. To the point that the number pipelines you operate and maintain is actually the multiple of any fork in logic you need.
I was hopeful, but ultimately disappointed.
I'd probably change my mind if I ever get unlucky enough to need to deploy a critical hotfix at the same time GH is unavailable though.
Any chance you're willing to share more?
It sounds like there's something interesting to learn here.
At the company I work at, we have about 160 full time engineers with about half our repos on continuous release. We average right now somewhere around 50-55 releases per day, with some hot repos getting up towards 20 releases per day on their own.
The point of small releases is that it makes rollbacks super easy (you always know what broke), roll forwards easy (you can easily test post rollout to find out if things aren’t functioning as expected), and you can mentally move on to your next task sooner because your code gets released when you want it released- not when the next “big develop release” happens.
To pull this off requires a few things. You need good tests that run against PRs. You need fast deploys (under 10 minutes is a good starting point, but get them faster if you can). And, ideally, you have ephemeral environments for each PR so that changes can be tested fully in isolation without requiring a shared staging environment.
This is legit what I do for a living, so AMA.
Can you describe at a high level your development toolset, CI/CD toolchain and workflow, and deployment environment?
Devs develop locally. Can test their code (manual test and most automated tests).
They push and open a PR. They can deploy ephemeral environments with any dependent microservices spun up along side. This is an in house tool we developed that builds and deploys ephemeral docker, CDK, and terraform based apps.
We then run CI automation against their PRs. This can include E2E tests courtesy of these ephemeral environments.
Once their PR is green and signed off, they merge to master. We’ve done some wizardry behind the scenes so that their code is live in production in around 3 minutes.
They verify their changes in prod and we run any automated regression tests against prod.
Next person merges as they see fit.
That’s it! It’s not easy to get to that point, but once you’re there it’s a great process. We’ll have capacity issues once we get up to 30 or 40 releases per day in a repo, but I’ve got some ideas about how to work around that.
We do that, sort of. Obviously we're not PR'ing to get things in to production. Changes go through QA, Staging, UAT, etc first. The final point is that they go to prod though, and if a few hours of changes end up queued to all go at once that's fine.
If not seeing a change in production for a few hours counts as "a nightmare" for you then I certainly don't envy you. That must be incredibly stressful. How do you do good work under that amount of pressure?
However, scrolling back through the status page's history for the last year shows that there between 6 and 12 incidents every month, and it often hovers at the high end of that range.
That has to be an utter misery for anyone who's on call.
I've started using that (uptime/performance/latency graphs being removed from status pages) as a signal to for when companies feel embarrassed over their quality of service but don't have enough interest/resources/profits to fix it.
It doesn't seem like Microsoft should have any of these issues. At most "interest", but even that should be a pretty easy sell for a service as massive as github.
Has been left up to the open source community to implement basic functionality.
We're also looking into Github enterprise for the IP allowlist feature which unfortunately means we'll have to migrate our whole CI/CD to self-hosted since Github-hosted runners don't work with this feature enabled.
I wonder how hard it is to write your own CI/CD though? Anybody has thoughts on this?
IMO the only “serious” threat to Jenkins are platform integrated services like Gitlab CI/CD, GitHub Actions, Google CloudBuild etc. Everybody else seems to continue to use Jenkins.
I'm not scared of competition, I'm mostly wondering how viable and profitable such a business would be. I happen to believe people would pay for good specialized CI (say, Elixir-only, Rust-only, Golang-only etc.) but you're also quite right that the real value-add seems to be in integration.
While specialized CI sounds great you would need a good general purpose CI which also sports specialization. Ort people will need to change CI once they e.g. need to build and link against some FFI. Or have some project in another language. Which is quite a bit reason not to just it even if you currently only need specialized CI.
It has a steep learning curve, as with any sufficiently advanced system.
[1] https://buildbot.net/ [2] https://john.scummvm.org
https://builds.sr.ht/~motiejus/job/517389
And unlike any megacorp, Drew is approachable. I recommend having a look at it for anyone re-evaluating their software forge.