This seems to skip the idea of stacked commits plus automatic rebasing, which have been around in Gerrit and other tools for quite a while.
If you read between the lines, the underlying problem in most of the discussion is GitHub's dominance of the code hosting space coupled with it's less than ideal CI integration - which while getting better is stuck with baggage from all their past missteps and general API frailty.
while the article does not provide specific dates for the “prehistory” the Not Rocket Science piece refers to an automated integration system working circa 2000~2001:
> February 2 2014, 22:25 […] Thirteen years ago I worked at Cygnus/RedHat […] Ben, Frank, and possibly a few other folks on the team cooked up a system [with a] simple job: automatically maintain a repository of code that always passes all the tests.
(I work on a project that uses Chromium’s commit queue infrastructure)
I think there’s a big difference between Chromium’s approach and the “not rocket science” rule. AIUI Chromium’s model there are still postsubmits that must pass or a change will be reverted by a group monitoring the queue. This is a big difference in practice vs having a rotation or team that reorders the merge queue and rolls changes up to merge together. In the commit queue model you land faster at the expense of more likely reverts than in the merge queue model.
This covers part of the problem, the part where your tests are enough to indicate whether the changes are good enough to keep. In that scenario, relying only on fast-forward merges is good enough.
One trickier problem is when you don't know until later that a past change was bad: perhaps slow-running performance tests show a regression, or flaky tests turn out to have been showing a real problem, or you just want the additional velocity of pipelined landings that don't wait for all the tests to finish. Or perhaps you don't want to test every change, but then when things break you need to go back and figure out which change(s) caused the issue. (My experience is at Mozilla where all these things are true, and often.) Then you have to deal with backouts: do you keep an always-green chain of commits by backing up and re-landing good commits to splice out the bad, and only fast-forwarding when everything is green? Or do you keep the backouts in the tree, which is more "accurate" in a way but unfortunate for bisecting and code archaeology?
I think this is ignoring a lot of prior art. Our deploys at Yelp in roughly 2010 worked this way -- you flagged a branch as ready to land, a system (`pushmaster` aka `pushhamster`) verified that it passed tests and then did an octopus merge of a bunch of branches, verified that that passed tests, deployed it, and then landed the whole thing to master after it was happy on staging. And this wasn't novel at Yelp; we inherited the practice from PayPal, so my guess is that most companies that care at all about release engineering have been doing it this way for decades and it was just a big regression when people stopped having professional release management teams and started just cowboy pushing to `master` / `main` on github some time in the mid 2010's.
I don't know what it's like now, but GitHub's internal merge queue circa 2017 was a nightmare. Every PR required you to set aside a full day of babysitting to get it getting merged/deployed - there were too many nondeterministic steps.
You'd join the queue, and then you'd have to wait for like 12 other people in front of you who would each spend up to a couple hours trying to get their merge branch to go green so it could go out. You couldn't really look away because it could be your turn out of nowhere - and you had to react to it like being on call, because the whole deployment process was frozen until your turn ended. Often that meant just clicking "retry" on parts of the CI process, but it was complicated, there were dependencies between sections of tests.
> The motivation was to avoid "merge skew," where changes appear compatible when reviewed in isolation but break once merged into an updated main.
My opinion is that this situation of a merge skew happens rarely enough not to be a major problem. And personally, I think instead of the merge queues described in the article, it would be overall more beneficial to invest in tooling to automatically revert broken commits in your main branch. Merging the PR into a temporary branch and running tests is a good thing, but it is overly strict to require your main branch to be fast forwarded. You can generally set a time limit of one day or so: as long as tests pass when merging the PR onto a main branch less than one day old, you can just merge it.
Yeah, there is way more prior art. We were doing this at FB early (here is a talk from 2014 mentioning it but it was much earlier: https://www.infoq.com/presentations/facebook-release-process...) and we definitely did not invent it...it was already the industry standard for large CI.
Mergequeues generally solve (1), but not for git merge conflicts. It's for when the combination of two in-flight changes would result in a regression (aka "logical merge conflict" https://trunk.io/blog/what-is-a-logical-merge-conflict)
With a semi-linear merge strategy, you rebase (without --fast-forward) before merging, so the history ends up looking like this:
* c8be0c632 Merge pull request #1538 from my-org/api-error-logging
|\
| * 80ecc4985 Fix security warning, bump nokogiri
| * 750613638 Log and respond with more detailed validation errors in the API
| * 0165d6812 Log code and details when rendering an API error response.
| * 1d4daab48 Refactor email validation result to include a descriptive message
| * 635214092 Move media_type logging into context_logging
|/
* 1cccd4412 Merge pull request #1539 from my-org/profile-clarify
|\
| * 87b342a32 Rename profile default to migration target
| * 2515c1e59 Fix disallow removing last profile in company
|/
* b8f3f1658 Merge pull request #1540 from my-org/customer
|\
| * 064b31232 Add customer-specific taxed allowance reduction
|/
* 3cf449f94 Merge pull request #1528 from my-org/console-logging
|\
| * 99657f212 Don't log to rails console in production
|/
* 8c72e7f19 Merge pull request #1527 from my-org/gemfile
It makes it easy to look at the Git history both at the 'PR level' kind of like a change log (`git log --merges --decorate --oneline`) or dig down into each PR to see all commits.
It’s a good idea and a deserved momentary rub-in-the-face blog post title back then to call it “not rocket science” (not for a decade+ name though).
I don’t use this because we just use the standard forge stuff. But yes, there shouldn’t be a reason to “break the build” and for small teams you shouldn’t need advanced tech to solve the issue (like for a bigger team with a large test suite it is tricky).[1]
But it would also be nice with some nice, readable history at the top level. Doesn’t have to be “curated” but you could have highlighted quotes and whatnot... I don’t know. Look at the Git project; the merge commits are decent.[2] The Rust project’s merge commits look like some dogfood factory ground floor where the coolers have malfunctioned. Bunch of weird command-like `r? @ghost rollup=...` which look like they were vacuumed up from GitHub via contributors or maybe also Tina Fey-bot. Then the commit message will the ill-suited MarkDown dump from the PR, including paragraphs with no hard-wrapping,[3] inline links (you can use reference style but okay), or how about some crazy ascii tables that look like a MySQL console[4] when you query a table with 20 columns (exaggeration). But I guess this is the not-rocket-science aesthetic of real serious business going on here.[5]
[1]: Shouldn’t need to buy some company software like whoever is selling this via their history lesson
[2]: A typical one would be “the frobinator in the config parsing segfaulted ..., which has been corrected”. Then with an inline subject-only “log” of all the commits. Pretty old-school.
[3]: The stuff they put in the commit message even uses an arrow to indicate “oh this was too long for me”: `feels like a regression as →` https://github.com/rust-lang/rust/pull/146121
[4]: 1ed3cd7030718935a5c5e5c8f6581f36d8be179f
[5]: Closer to the Linux Kernel style than the Git project, certainly
18 comments
[ 2.9 ms ] story [ 47.3 ms ] threadThis blog post about choosing which commit to test is also relevant and may be of interest: https://sluongng.hashnode.dev/bazel-in-ci-part-1-commit-unde...
If you read between the lines, the underlying problem in most of the discussion is GitHub's dominance of the code hosting space coupled with it's less than ideal CI integration - which while getting better is stuck with baggage from all their past missteps and general API frailty.
> February 2 2014, 22:25 […] Thirteen years ago I worked at Cygnus/RedHat […] Ben, Frank, and possibly a few other folks on the team cooked up a system [with a] simple job: automatically maintain a repository of code that always passes all the tests.
I think there’s a big difference between Chromium’s approach and the “not rocket science” rule. AIUI Chromium’s model there are still postsubmits that must pass or a change will be reverted by a group monitoring the queue. This is a big difference in practice vs having a rotation or team that reorders the merge queue and rolls changes up to merge together. In the commit queue model you land faster at the expense of more likely reverts than in the merge queue model.
One trickier problem is when you don't know until later that a past change was bad: perhaps slow-running performance tests show a regression, or flaky tests turn out to have been showing a real problem, or you just want the additional velocity of pipelined landings that don't wait for all the tests to finish. Or perhaps you don't want to test every change, but then when things break you need to go back and figure out which change(s) caused the issue. (My experience is at Mozilla where all these things are true, and often.) Then you have to deal with backouts: do you keep an always-green chain of commits by backing up and re-landing good commits to splice out the bad, and only fast-forwarding when everything is green? Or do you keep the backouts in the tree, which is more "accurate" in a way but unfortunate for bisecting and code archaeology?
You'd join the queue, and then you'd have to wait for like 12 other people in front of you who would each spend up to a couple hours trying to get their merge branch to go green so it could go out. You couldn't really look away because it could be your turn out of nowhere - and you had to react to it like being on call, because the whole deployment process was frozen until your turn ended. Often that meant just clicking "retry" on parts of the CI process, but it was complicated, there were dependencies between sections of tests.
My opinion is that this situation of a merge skew happens rarely enough not to be a major problem. And personally, I think instead of the merge queues described in the article, it would be overall more beneficial to invest in tooling to automatically revert broken commits in your main branch. Merging the PR into a temporary branch and running tests is a good thing, but it is overly strict to require your main branch to be fast forwarded. You can generally set a time limit of one day or so: as long as tests pass when merging the PR onto a main branch less than one day old, you can just merge it.
Mainly I'm confused what this check is gating. Based on the article it's hard to tell what they mean.
1. Code changes that may conflict with each other in the repo, in the sense of a merge conflict.
2. Regressions (test failure, build breakage) caused by recently checked-in code.
3. Preparing and verifying a new release prior to deployment.
4. Monitoring/canary a release candidate with real users.
In my mind, these are all very different things, but the article seems to mix them up.
I Gitlab and Azure DevOps (also owned by MS) supports it, and even talked to an employee now working at Github, that implemented this in Azure DevOps.
More background: https://github.com/orgs/community/discussions/14863
With a semi-linear merge strategy, you rebase (without --fast-forward) before merging, so the history ends up looking like this:
It makes it easy to look at the Git history both at the 'PR level' kind of like a change log (`git log --merges --decorate --oneline`) or dig down into each PR to see all commits.I don’t use this because we just use the standard forge stuff. But yes, there shouldn’t be a reason to “break the build” and for small teams you shouldn’t need advanced tech to solve the issue (like for a bigger team with a large test suite it is tricky).[1]
But it would also be nice with some nice, readable history at the top level. Doesn’t have to be “curated” but you could have highlighted quotes and whatnot... I don’t know. Look at the Git project; the merge commits are decent.[2] The Rust project’s merge commits look like some dogfood factory ground floor where the coolers have malfunctioned. Bunch of weird command-like `r? @ghost rollup=...` which look like they were vacuumed up from GitHub via contributors or maybe also Tina Fey-bot. Then the commit message will the ill-suited MarkDown dump from the PR, including paragraphs with no hard-wrapping,[3] inline links (you can use reference style but okay), or how about some crazy ascii tables that look like a MySQL console[4] when you query a table with 20 columns (exaggeration). But I guess this is the not-rocket-science aesthetic of real serious business going on here.[5]
[1]: Shouldn’t need to buy some company software like whoever is selling this via their history lesson
[2]: A typical one would be “the frobinator in the config parsing segfaulted ..., which has been corrected”. Then with an inline subject-only “log” of all the commits. Pretty old-school.
[3]: The stuff they put in the commit message even uses an arrow to indicate “oh this was too long for me”: `feels like a regression as →` https://github.com/rust-lang/rust/pull/146121
[4]: 1ed3cd7030718935a5c5e5c8f6581f36d8be179f
[5]: Closer to the Linux Kernel style than the Git project, certainly