When the movement toward status pages started, it was great. Early adopters were eager to communicate their status, to the extent they technically could.
Nowadays it feels like most status pages show green even when things are decidedly not green.
Conspiracy theorists abound, but in most cases it's probably just ordinary human tendency to not broadcast one's foul-ups eagerly and broadly at the first opportunity.
Investing could mean hiring more experienced candidates rather than more candidates. Investing could mean having a fraction of the devs and far more resources for each one of them. Whatever the problem this is absolutely a GitHub management issue they gotta fix.
Even more likely that most SRE and infra engineers have no ownership over that either, since it's all Microsoft-authored code (in case you didn't know: GitHub Actions is Azure DevOps Pipelines is VS TFS Pipelines).
Yes but management are responsible for allocating resources. Pointing to another team that has no bearing on the issue at hand doesn't nothing to absolve management from this.
Yes and it's a poor allocation of resources. Instead of hiring n designers they could have hired m SREs and it stands to reason we'd have a less-gamified GitHub and a more stable GitHub. So plusses all around, I think.
Again? Last time that happened was 12 days ago. [0] Worse part is, it hasn't reflected on the status page. Typical of status pages.
Looks like relying on and going all in on Github and GitHub Actions really doesn't make any sense and I now expect this to go down once or even twice a month.
Oh dear. Perhaps self-hosting like how ReactOS, ARM, RedoxOS, GNOME, etc have done seems like a good idea.
Unfortunately it's not fully-fledged enough to work properly with all CI configs. I had trouble with some conditions but also I note that it doesn't seem to implement reusable workflows yet.
It's also a bit fiddly to set up so you don't run into API rate limits.
I think that people who are interested in having resilience should write their GHA CI config for act and then be happily surprised when it works on GitHub too.
Or write them as something more portable (Earthfiles are interesting, but a bit clunky when I tried) that can also be executed locally — having the GitHub-specific stuff just be a very light wrapper to 'call into' the portable scripts.
Now that everyone and their dog is moving to "serverless" hosting, how does one institute redundancies in case something like this happens again? It would suck if half the Internet and its open source projects go down because of a bad server configuration. Does anyone know of a "load balancer" for Github projects or source management in general? It looks like a solution many could benefit from.
We don’t rely on GH for our production systems. We have a clone of our Git repo hosted in-house for anything that needs a Git repo.
We don’t rely on GH for our Primary CI/CD. We can make changes and push them through CI/CD without GH.
We have an understanding of the business impact of GH being down. It’s an enabler that we love using, but we can operate without it for a known period of time, albeit without certain workflows and value added tools.
If our tolerance for it being down we’re much greater, then it would be worth the money and overhead to host it on our own, which they support.
The simple answer is "you don't," because a couple hours downtime on your CI/CD platform shouldn't matter. You should be able to deploy by hand in an emergency.
The less simple answer is "by doing Disaster Recovery planning". Go outside and throw a rock and you'll hit a company doing hosted CI/CD. Set up two of them; when one fails, use the other.
GitHub could mitigate this by making its services region and AZ specific, and then only one region should go down at a time, letting you switch regions. Not sure why this hasn't happened, other than GHA is more of a "consumer product" and they may assume customers are too dumb to figure out how to switch regions. (This may sound harsh, but this is the same company that wouldn't update documentation about how to generate ssh keys to make them more secure because it might confuse their users)
For example, if you are using AWS, you could use Code Pipeline for your CI/CD alongside GitHub actions.
Use GHA for “PR open” triggered tasks, but Code Pipeline for tag or “PR merge” triggered tasks.
This way your deployment is decoupled from GH availability, but you get the benefits of GHAs first-party integration with your core software development loops.
maintaining bi modal behavior introduces a new set of problems. I'd take "should be able to deploy by hand in an emergency" a step further: deploy by hand, frequently. make sure that your production outage isn't the first time you are exercising this runbook.
Git is already decentralized, and you can push your code to any of the dozens of competitors easily. I often keep important projects on both Bitbucket and GitHub.
I think a hosted mulit-tenant CI platform appears fairly simple on the surface, but there's actually a lot of complexity under the hood when your product is arbitrary code execution as-a-service - and you want to offer a smoother user experience than something as general-purpose as a container runtime.
Especially when you throw in a large and generous free tier.
None of the recent examples of unbundled multi-tenant CI/CD appear to have wildly successful exit stories, or even continuing to run with great growth and profitability stories. I’m thinking of companies like Travis with that comment (which was, IMO, delightful to use for its era). It’s probably pretty hard to sell this to the average enterprise when you’re competing against product suites which do 100x more? Definitely feels like a situation where those suites are “Jack of all trades, and master of none” though, and collectively mediocre vs. individually excellent.
We are _besieged_ by caching issues. A good 1/4 to 1/3 of a particular AMI build action we run will fail because of a package mirror at GH being out of date.
I'd be interested in reading a source for both of these points, if you have one. By "moving the Azure DevOps engineers to GitHub" do you mean they switched internal tooling on their own team? or that engineers from the ADO team switched orgs and now work for GitHub?
As an individual sure, but as an organization? No way a team of 100 can just pivot to another provider.
CICD providers are not interoperable, once one is picked, it's a complete lock in.
If I was evil, I would offer a cheap service to employees of any potential contributors, one MBA types don't understand, then once they've made themselves dependent on it, make it unreliable. Boom. There goes your productivity and your credibility as competition.
GitHubs status page went from industry leading with uptime graphs and instant updates to a static “everything’s ok” page (as the whole service 500s) page overnight with the Microsoft acquisition. They are burning good will like it’s a fuel.
That's what happens when you join a trillion-dollar company. Somebody lets it slip that the service might not be reliable and suddenly you lose a billion dollars on the stock market.
Source? Any graphs to look at, for example. That covers years before and after the acquisition?
I've been thinking that GitHub has pretty good uptime.
(I wouldn't personally want to rely on a specific 3rd party like GitHub or Circle CI or anyone, for deploying my software, though. Wow, 560 people have followed the githubstatus.com link (I see in the Discourse forum) that's many people affected I think!)
Lots of complaining here. I'd be ticked if I was a paying customer.
But I just started using Actions yesterday. On my free open source project. And I've been using it all morning (no problems). And the whole time I was thinking: it's bloody amazing that I'm getting this for free.
(I used to work on a bit of a similar system at Google [internal remove build execution system for non-google3 projects] and ... there's a lot involved.)
76 comments
[ 2.9 ms ] story [ 158 ms ] threadNowadays it feels like most status pages show green even when things are decidedly not green.
Conspiracy theorists abound, but in most cases it's probably just ordinary human tendency to not broadcast one's foul-ups eagerly and broadly at the first opportunity.
Customers who are affected? Appreciate the acknowledgment.
Customers who aren't? Appreciate that if they are next time, even if it's only a small subset, they'll be made aware.
But that assumes that the people assigned to achievements could never help the platform to be reliable. If that’s the case, why hire them at all?
Either way you slice it, money is being spent here.
We currently don't have any issues and the status page also doesn't list anything.
Looks like relying on and going all in on Github and GitHub Actions really doesn't make any sense and I now expect this to go down once or even twice a month.
Oh dear. Perhaps self-hosting like how ReactOS, ARM, RedoxOS, GNOME, etc have done seems like a good idea.
[0] https://news.ycombinator.com/item?id=31579932
I think that people who are interested in having resilience should write their GHA CI config for act and then be happily surprised when it works on GitHub too. Or write them as something more portable (Earthfiles are interesting, but a bit clunky when I tried) that can also be executed locally — having the GitHub-specific stuff just be a very light wrapper to 'call into' the portable scripts.
The (unfortunately named) Dagger is also an entry into that space: https://github.com/dagger/dagger#readme (Apache 2)
We don’t rely on GH for our production systems. We have a clone of our Git repo hosted in-house for anything that needs a Git repo.
We don’t rely on GH for our Primary CI/CD. We can make changes and push them through CI/CD without GH.
We have an understanding of the business impact of GH being down. It’s an enabler that we love using, but we can operate without it for a known period of time, albeit without certain workflows and value added tools.
If our tolerance for it being down we’re much greater, then it would be worth the money and overhead to host it on our own, which they support.
The less simple answer is "by doing Disaster Recovery planning". Go outside and throw a rock and you'll hit a company doing hosted CI/CD. Set up two of them; when one fails, use the other.
GitHub could mitigate this by making its services region and AZ specific, and then only one region should go down at a time, letting you switch regions. Not sure why this hasn't happened, other than GHA is more of a "consumer product" and they may assume customers are too dumb to figure out how to switch regions. (This may sound harsh, but this is the same company that wouldn't update documentation about how to generate ssh keys to make them more secure because it might confuse their users)
Use GHA for “PR open” triggered tasks, but Code Pipeline for tag or “PR merge” triggered tasks.
This way your deployment is decoupled from GH availability, but you get the benefits of GHAs first-party integration with your core software development loops.
Avoid the temptation to code up all the details of your build and deploy process in a proprietary single vendor CI tool.
Code your build using a build tool, script you deploy using tools suitable for that.
Now you can run the same process on multiple different CI services, or locally.
- [1] https://github.com/actions/setup-node/issues/516
- [2] https://github.com/actions/cache/issues/820
As of now (11:28 UTC) the status page has been updated.
https://www.githubstatus.com
Especially when you throw in a large and generous free tier.
- GitHub Actions Is Down (github.community)
+ GitHub Actions Is Down Again (github.community)
```
As it stands CI/CD is a bit of a commodity, so you can go with any of the solutions and they're largely similar.
CICD providers are not interoperable, once one is picked, it's a complete lock in.
If I was evil, I would offer a cheap service to employees of any potential contributors, one MBA types don't understand, then once they've made themselves dependent on it, make it unreliable. Boom. There goes your productivity and your credibility as competition.
I've been thinking that GitHub has pretty good uptime.
(I wouldn't personally want to rely on a specific 3rd party like GitHub or Circle CI or anyone, for deploying my software, though. Wow, 560 people have followed the githubstatus.com link (I see in the Discourse forum) that's many people affected I think!)
But I just started using Actions yesterday. On my free open source project. And I've been using it all morning (no problems). And the whole time I was thinking: it's bloody amazing that I'm getting this for free.
(I used to work on a bit of a similar system at Google [internal remove build execution system for non-google3 projects] and ... there's a lot involved.)
Me too