Kinda - what we're seeing at Depot (https://status.depot.dev/clyrvud6i57402igofm6jtb7id) is that the APIs for managing Actions runners or retrieving work are receiving rate limit errors, interestingly missing the regular rate limit status headers[1]. We do have some jobs processing, but overall the GitHub API and Actions control plane seem very overloaded by the outage.
Instead of saying means things about GA, can anyone chip in about ways they've made their work more resilient and fault tolerant in case such incidents happen?
We run Bitbucket pipelines, but a similar notion applies. Everything we do is in a container. If we wanted to run our CI manually, we can just run our steps as a script on anyone’s workstation (given keys/permission). Staying away from the proprietary run environments and being easily to run them ourselves has saved us a number of times. Especially when running on Atlassian where things go down often.
Of course there are plenty of self-hosting options if you want that responsibility, but a very rare outage of CI/CD doesn't seem like something that's worth spending resources planning around. On the scale of possible disruptions, it's pretty minor.
We looked into it, very briefly, and decided that a few hours or even a whole day of downtime per year wasn't really worth the time and cost it would take to mitigate this in our case. If our pipelines happen to fall over, and it also happens to occur during our working hours then the solution for us is to either work on something else or head down to the pub.
20 comments
[ 3.2 ms ] story [ 46.2 ms ] thread[1] https://azure.status.microsoft/en-us/status
Something very critical has fallen over.
Edit: I don't think this is for everyone though - maybe only if your ADO instance happens to be hosted in US Central?
However, the server we would SSH to to probably has less uptime than Azure.
[1] - https://docs.github.com/en/rest/using-the-rest-api/rate-limi...
Just another Thursday :shrug:
https://warpbuild.instatus.com/