I realize this isn't about GitHub Actions, but I wish they would get on with different machine sizes for GitHub's own runners[0]. Many huge builds can end up taking 2x-3x the time on the slow 2 core machine they give you compared to a local high-end CPU.
I feel like this is a common problem with almost all CI/CD offerings. At my company we ended up just installing a local build agent on a powerful pc, and using that for the 'frequent, but not super important' builds, such as building/test for a PR.
The builds for acceptance/prod we still do on remote CI/CD, but those happen much more rarely. Also the advantage of a local CI/CD is that it is much easier to setup caching for things like Node Modules, NuGet etc.
We use Azure Devops which has a pretty good local CI/CD story, installing an agent was quiet easy for us.
You should check out this [0] GitHub action workflow. It uses docker machine to create a high resourced runner on demand in the cloud and then runs the subsequent job on it. The workflow is intended for machine learning but I'm sure it could be adapted. I must disclose that I haven't actually used it myself yet (because I haven't felt the need) but it does look very promising.
With GitLab CI you can task any machine you want to your builds. I've slaved my workstations to my CI builds because they're ~2-5x faster than the AWS hardware most companies use.
This doesn't seem like all that great of a solution to me. The strict requirement to have a passing build is replaced with the fear that you'll be the one that broke the build. So you'll end up running these tests before you merge anyways, right? What has changed?
I think the difference is that they expect to not break tests most of the time. So most of the time this just saves time.
It’s like when you merge a documentation PR while the tests are still running: You’re pretty sure that the tests won’t break. With this solution however you’ll still be pinged if they do break, later.
If one developer doesn't break it 98% of the time, then for 200 devs you have a chance of .98^200=2% of not breaking it. And for one person trying to quickly get the fix ready while he might have an urgent family situation you have 199 devs who can't get their stuff in, of which at least 20 are under pressure to deploy their fixes quickly themself. It sounds like creating a hard problem for individuals for solving a mediocre problem for the team.
I dont see why this would block other devs merging in their work. And what kind of teams do you work in where you can't politely ask for someone's help when you have to deal with a personal issue?
From personal experience, statistical clustering seems to set the narrative.
You don’t even need to hit a 2% success rate to get people agitated. Consecutive failed builds that happen every few weeks will eventually happen when someone really needs to get something out Right Now and that incident will come to define their experience, especially if it happens two or three times. Even if it’s just to people they know instead of themselves.
Anything that happens once a day happens “all the time.” For some people, that’s true for once a week. For others, if it happened twice in two weeks and once every six weeks thereafter.
The dev does have the option of reverting their change.
If significant numbers of devs found the whole process stressful, I wonder if it would help to make the reversion happen automagically after, say, 24 hours, if the dev hasn't committed anything in that time
Devs who have called in sick or over the weekend or attending a wedding or long-planned vacation don't have to think about it.
> The dev does have the option of reverting their change.
Stopping the world seems like an automated process. So should reverting be. In fact, there should be no stopping the world at all, just reverting if the dev can't fix it within 72 hours. There might be cases where automatic reverting doesn't work, but those should still be managed through organizational processes, not by stopping the world.
I'm an unusual American, in that I consider healthy work-life balance to be top priority, at least for everyday employees (founders and such have a different path).
I do try to make room for the idea that my understanding of this is likely superficial, and that there may be things that I am missing, here.
What has changed is that deployments happen 3x faster.
I would guess that this wouldn't introduce more or less build breaks. People will do as they've been doing. The psychological argument can be made that people could be either less safe with their commits, since they have 72 hours to fix any problem breaking a long-running CI job, or they could be more safe with their commits, fearing they'll break a long-running CI job. Personally, I don't think that either position has a lot of strength.
Stopping the world after 72h of CI fails seems like a dangerous decision, especially if you have a large team working on a monorepo. Note that fails can easily 'pile up' and the 72h are not necessarily all caused by the same change. I would probably get paranoid, feeling the constant pressure not to be part of the cause for such an occourance.
Yes, I didn't understand this. I've even been a part of workflows that halt all subsequent PRs until the CI tests pass, and I still don't understand the situation this is designed to prevent.
Why not simply disallow the PR to merge unless CI passes? Why stop all integrations? There must be something I'm missing.
Edit: I think I get it. Normally PRs to Github.com non-enterprise go through as normal. 45 minutes later, after that merge, the 2 expensive Enterprise CI tests complete and that's when the developer gets the 72 hour timer. The dev has the option of reverting their change and trying again.
It's not very clear but it seems to suggest they deploy to production before all the tests pass? That seems silly. Why not just wait until they've passed?
Why should deploying to production be be prevented by needing to wait for tests that do not apply to that environment, but only to a separate product (GitHub Enterprise Server)?
GitHub takes continuous deployment very seriously, with dozens of deployments every day. My understanding is that they try to avoid having code sitting around only deployed to an internal environment any more than truly necessary. They want to be able to fix code issues discovered in production within a shockingly short time frame if the issue is not complex.
24 comments
[ 2.6 ms ] story [ 55.2 ms ] thread0: https://github.com/github/roadmap/issues/95
The builds for acceptance/prod we still do on remote CI/CD, but those happen much more rarely. Also the advantage of a local CI/CD is that it is much easier to setup caching for things like Node Modules, NuGet etc.
We use Azure Devops which has a pretty good local CI/CD story, installing an agent was quiet easy for us.
0: https://github.com/iterative/cml_cloud_case/blob/master/.git...
0: https://docs.github.com/en/free-pro-team@latest/actions/host...
It’s like when you merge a documentation PR while the tests are still running: You’re pretty sure that the tests won’t break. With this solution however you’ll still be pinged if they do break, later.
Read the article. It says:
> If the CI job remains broken for more than 72 hours, all deployments to GitHub.com are halted
Edit: it even creates the perverse incentive of trying to get your own stuff in quickly before the 72h window closes that s/o else caused.
You don’t even need to hit a 2% success rate to get people agitated. Consecutive failed builds that happen every few weeks will eventually happen when someone really needs to get something out Right Now and that incident will come to define their experience, especially if it happens two or three times. Even if it’s just to people they know instead of themselves.
Anything that happens once a day happens “all the time.” For some people, that’s true for once a week. For others, if it happened twice in two weeks and once every six weeks thereafter.
If significant numbers of devs found the whole process stressful, I wonder if it would help to make the reversion happen automagically after, say, 24 hours, if the dev hasn't committed anything in that time
Devs who have called in sick or over the weekend or attending a wedding or long-planned vacation don't have to think about it.
Stopping the world seems like an automated process. So should reverting be. In fact, there should be no stopping the world at all, just reverting if the dev can't fix it within 72 hours. There might be cases where automatic reverting doesn't work, but those should still be managed through organizational processes, not by stopping the world.
I'm an unusual American, in that I consider healthy work-life balance to be top priority, at least for everyday employees (founders and such have a different path).
I do try to make room for the idea that my understanding of this is likely superficial, and that there may be things that I am missing, here.
I would guess that this wouldn't introduce more or less build breaks. People will do as they've been doing. The psychological argument can be made that people could be either less safe with their commits, since they have 72 hours to fix any problem breaking a long-running CI job, or they could be more safe with their commits, fearing they'll break a long-running CI job. Personally, I don't think that either position has a lot of strength.
Why not simply disallow the PR to merge unless CI passes? Why stop all integrations? There must be something I'm missing.
Edit: I think I get it. Normally PRs to Github.com non-enterprise go through as normal. 45 minutes later, after that merge, the 2 expensive Enterprise CI tests complete and that's when the developer gets the 72 hour timer. The dev has the option of reverting their change and trying again.
Or does "deploy" really mean "merge"?
GitHub takes continuous deployment very seriously, with dozens of deployments every day. My understanding is that they try to avoid having code sitting around only deployed to an internal environment any more than truly necessary. They want to be able to fix code issues discovered in production within a shockingly short time frame if the issue is not complex.