Is anyone doing collaborative cloud-based editing with the ability to run all tests at any time? That sounds pretty great, although you still need deployment gating.
Sorry to disappoint but I think Jenkins will be here far beyond 2025.
However on the topic of real-time integration that is somewhat already here for no-code systems that are already divorced from traditional version control.
He hasn't been in the industry long enough to see that most of the pillars of software engineering are built on top of massive tech debt.
When I say tech debt I mean design flaws created by designer who simply could not anticipate the future. I'm not talking about bugs, mistakes or intentional shortcuts.
Whether jenkins is "tech debt" is a different topic. But whether Jenkins is here to stay past 2025 has nothing to do with how well it's designed.
Thought experiment: taking real-time pre-commit workflows to the logical extreme, humans just become slaves to the machines, sitting at keyboards making near-instantaneous real-time changes to running systems in response to changing conditions and needs.
I always have to be careful responding to things like this because I get sucked into the trap of thinking “we” means myself, and it just needs to mean “most of us” for whatever they’re talking about to be worrisome or even true.
I do worry that a lot of people are missing the first principles where CI is concerned, but a few of us still remember. Whether that’s enough is always a question.
> First, CI rarely knows what has changed, so everything needs to be rebuilt and retested
CI is the impartial judge in your system. That is the ultimate goal and much of the value. Don’t believe me when I say that your check in broke something. Believe the robot. It does not think, it does not interpret errors (which is why intermittent failures poison the well). It just knows that whatever you did was not good.
We make it stupid simple because the more ways you try to interpret things, the higher the likelihood the system will either miss an error or report imaginary ones. And in this case I am very sorry Mr Lasso, but it is in fact the hope that kills. Optimism makes us try things, things others would give up on. That’s what makes a piece of software good. Better. But too much and you start ignoring very real problems and make optimism into a liability. I’ve seen it over and over again, take a build with intermittent failures and people will check in broken code and assume it’s the intermittent problem and not their change.
Ideally, when you start breaking builds you’ve found your limits. But the build has to break when I break the code, and it needs to do it repeatably, because if I can’t reproduce the error I risk telling myself stories, and foisting off debugging to other people, which is a big no no.
If self delusion is our biggest sin, learned helplessness is probably number two.
A small but important part of my motivation to build better DevEx is that it puts a squeeze on the foot draggers. If you can’t blame the tools for being shit (and believe me, I know a lot of them are), then the only other explanations are that you are either too stupid to use them or just not a team player. An asshole.
Developers will pick being called an asshole over being seen as stupid any day, but the former eventually gets management involved. It’s the closest I’ll probably ever get to solving a social problem with tech. Make it easy for people to participate, make them feel silly for not, then ramp up the pressure if that doesn’t work, and when it’s clear they only care about themselves you have actionable evidence to involve HR.
What if it was possible to have a local build system that would run tests etc and provide a cryptographically secure hash proving that the tests passed, the build was good, etc then a “centralized” CI system just verified that before permitting merge/deploy.
There’s a lot of hand waving there but it doesn’t seem wildly implausible.
Local use of containers or something like nixos could provide sufficient protection against the “works on my machine but not in production” problem.
Running tests locally is pretty much a no-no for many codebases because they are so large and tests take a long time if they don't run on a beefy test server or in parallel.
Common examples of this are multi-million line C++ codebases (e.g proprietary game engines) and monorepos in any language.
Running tests on my computer uses up valuable CPU cycles that I can use to work on something else while the CI servers are running my tests.
However I can see this working for small to medium sized codebases that really don't need CI for testing.
Partial suite is certainly an option. You can build tools that only test code that has changed and whose changes can have cascading effects on other code (i.e dependent code). Do you know if such tools exist, or would I (as in the software engineer working on the codebase) have to build them?
Partial is the most valuable when you have red tests that are resisting simple fixes. I would definitely encourage running the entire tests every hour or two at the least.
We have a project where the team split the tests into chunks and eventually I figured out the reason why is because they had coupling between tests, and running them all together ran into problems. I worry about other people opening that door, because it’s damned hard to close again.
This is actually the default result you get if you leave the defaults alone in some CI systems! My current company runs CI this way, in fact. A checkout is re-used between builds, and Gradle computes the minimal set of tasks that need to be run based on changed files just as it would locally. TeamCity will set things up this way by default. If you want a clean build on every run, you have to opt-in by checking a box.
How well does it work? Well .... it sorta mostly works. Sometimes a build will fail for inexplicable reasons because Gradle/Kotlin incremental builds don't seem fully reliable. You re-run with a clean checkout and the issue goes away.
How much time does it save? Also hard to say. Most of the time goes into integration tests that by their nature are invalidated by more or less any change in the codebase. That's not exactly incorrect. Some changes in some modules do avoid hitting the integration tests, though.
TC can also do test sharding in the newest versions when you use JUnit. We don't use this yet though.
This is an oversight that lands in circle of “the things I know that are difficult to sell because everyone else is just ignoring it.”
Capacity planning applies to tests. As your test count goes up your budget per test goes down. Every CI tool should plot build and test duration over time and only a
couple do.
There’s a reason most test frameworks can mark slow tests. You need to not only use that but ratchet down over time. Especially when you get new hardware.
I haven’t run benchmarks lately but my old rule of thumb developed over many projects and with several people better at testing than I, was a factor of eight for each layer in the testing pyramid.
That certainly puts a lot of runtime pressure on the top of the pyramid, but that’s by design. You don’t want people racing to the top, because that’s how you get a cone.
I'm actually working on something to solve the slow test problem. We go down the massively parallel route. We run everything pre git commit by syncing your project directory to our servers and parallelizing the runs. Here is a demo https://brisktest.com/demos running against the react test suite.
It wasn't top of our list but it is definitely something we could support if there was interest (fairly quickly I'd imagine). My personal experience with go tests is that they tend to be very fast but perhaps I just haven't seen a large enough project. Is it something you'd be interested in? If you want to chat more about it, my email is in my profile.
I always found it interesting how git was adopted by industry where a large part of its complexity and feature set is built around suopporting fully distributed developer, developers who could lack internet access for months at a time but need to work collaboratively in a team. Then, industry took that distributed approach and strapped project and team management systems with web based front ends like Github, GitLab, etc. around git and created CI/CD pipelines that built ontop of or in parallel to those environments--all of which were centralized. While it's of course still true that your underlying repo instances still give you a lot of flexibility away from those centralized environments, the centralized environments themselves are critical in regular daily workflows, negating a lot of the flexibility git as a VCS adds.
In large part, industry could largely use a significantly simpler model than git for VCS since many features are largely moot on a daily basis. Many could probably get away with SVN in large part, for example (although branching isn't nearly as good).
I think you raise a great point in that we need to look at how processes have evolved ontop of version control and look at adapting those to a similar model. In some cases it's just not practical because the way infrastructure testing an deployment works but it's the direction I think we should be going. In fact, the first step would be to make issue systems and project management interfaces provided by efforts like GitLab and GitHub available in a local capable, distributed fashion. Clearly communications often require some degree of centralization or at least peer message propagation but there's no reason that information can't be separated from the infrastructure that displays and interacts with it.
Apparently, many people disagree with you; otherwise, we would not be in this situation.
I agree that centralization is probably not a hard requirement for these properties, but currently no one has come up with decentralized equivalents that provide users what they actually want - and remember, you don't get to tell them what they want!
Why is this not best of both worlds? You have centralized portals like GitHub that have bunch of additional proprietary features, but if you don't need them or don't want to have centralized solution you have freedom to interface with it from, uh, anything.
It's a recurring and perhaps inevitable problem. We've collectively been building software for a long time but any given individual hasn't. No-one has seen all the good ideas that have gone before and sometimes been forgotten. Everyone is influenced to some degree by whatever ideas are currently popular within their group.
We developed distributed VCSes and the ability to do smart things with branches and combining contributions from people all over the world at different times. Then we somehow still ended up with centralised processes where everyone has to merge to trunk frequently and if GitHub goes down then the world stops turning. It turns out that just using git as git wasn't such a bad idea after all.
We were told that dynamic languages like JS and Python are so much more productive and that this was essential for fast-moving startups to be competitive. Today the industry is moving sharply from JS to TypeScript and even Python now has similar tools. It turns out that static typing was better for building robust software at scale than relying entirely on unit tests after all.
Just wait until lots of developers who have only ever worked with JS or Python learn what a compiled executable is. If you tell them that xcopy deployment even to a huge farm of web servers is no big deal when all you're copying is one executable file and maybe a .env with the secrets, it'll blow their minds...
Sure. But anecdotally I don't think I have ever worked somewhere with a centralised VCS/CI/CD for more than a few weeks and not had at least one major outage that basically caused all development work to grind to a halt for everyone for at least several hours, all because a key service went down. I've seen it happen to GitHub, GitLab, some of the big dedicated CI providers...
You can certainly argue that there are other advantages to using those systems. However it's an inescapable fact that if those teams had been using git-as-git and had a good local environment for each developer then most of the developers would have been able to carry on with most of their work during all of those outages. Sometimes single points of failure fail.
Wow, that seems to be the other side of the lucky stream I am on :-D
Seriously, a well set up gutlab is pretty much undownable. And even if you do, what really is the impact? If the pipelines don't run you cannot integrate, sure if the core service breaks down you will probably resolve to exchanging patches the old school way. But the good part is, in order to deploy those decentrally sourced changes you still will go through centralized CI and gain its quality assurances by doing so. Where's the drawback?
Unfortunately a hosted GitLab is downable when GitLab itself goes down. Same for all the other hosted source control and CI systems.
Given that many companies now tie their entire deployment process into their source control and CI/CD systems that means you really are reduced to exchanging patch files. For any non-trivial change in a large system that quickly becomes impractical and so development slows to a crawl or everyone literally gives up and goes to the pub.
> Then, industry took that distributed approach and strapped project and team management systems with web based front ends like Github, GitLab, etc. around git and created CI/CD pipelines that built ontop of or in parallel to those environments--all of which were centralized.
A different view on this is that my CI system is a kind of colleague, responsible for a lot of what testers, ops, and build eng people toiled at before. It is another aspect of the decentralized system where a robot can also check out and work with the code even while humans continue to write more, which is considerably more painful under e.g. the SVN model. My human colleagues and I still share plenty of code directly, via git pulls and other tools.
Industry uses both the centralized and decentralized features. Just because I want to configure team permissions doesn't mean I want to be blocked if the network is down.
I think a lot of the adoption of git in the industry at large is in _spite_ of its distributed nature - it just so happens that the decentralized design made it care much more about branching and merging, and it got adoption because even for a small local team those improvements over SVN were worth the pain of dealing with git's UX warts.
Containers alone don’t fix the “works on my machine” issue. CI should be isolated to provide proper gate keeping such that any change required for code to function on CI machine will also be automatically loaded into production and other developers machines.
In theory you could get something reasonably close to this working locally, but the it’s a serial process so that’s pointless.
>will also be automatically loaded into production and other developers machines.
I’m assuming you are only submitting this as theoretical, because in the real world (where I’ve directly experienced this workflow) it’s a nightmare:
* Someone merges main which deletes working files in your feature branch. Automatic pushing/rebasing of main onto your feature branch creates needless work the moment you try to push upstream when git tries to ascertain state of local main.
* countless times I’ve hit bugs where if I try to merge “patch-a” from a common “feat-1” branch (meaning patch was cut from feat, not from main), but then main is updated by the auto-updater, I then have a messy working directory in which main’s new files are treated as unknown orphans and I have to spend time deleting these by hand.
I’m all for having my feat branch be up to date at merge time. But making it a rolling target is something git (from my perspective) hits the boundary of what git can reasonably do, and creates more pain than any type of positive DX
This is actual experience from multiple projects. Ideally starting from a blank machine/VM you should be able to run a single script that get’s the latest, builds it locally, runs it successfully, and passes all tests.
I'm going to be annoyingly pedantic here, apologies in advance. To me, what you're describing is an end state where the fetch/build + test is a state in which the developer is done with some task. But in my working reality, testing/building is a non-linear process that is iteratively run after some certain self-determined point-in-time.
>Ideally starting from a blank machine/VM you should be able to run a single script that get’s the latest, builds it locally, runs it successfully, and passes all tests.
This isn't automatic, and is a practice that is fairly standard today (And one I agree with). You have placed a requirement/line in the sand that says, "Only when I have ideal state should this pipeline run in linear time and output the final result, which are the return codes from tests." Your example has the user initiating the update of the upstream main, not some other process that runs git fetch on the Developer's behalf.
Your original comment, as I understand it, is contradictory to this point:
>or code to function on CI machine will also be automatically loaded into production and other developers machines.
All of us (I think) agree that auto-deployment to production is a desirable goal. But we all (I think) know that broken commits are routinely delivered to production, where "production" represents the sum of all production environments in the world. So while we can have a reasonable assumption that "Production is, or should be deployable all the time," that doesn't mean the state that is represented by Production is safe to run locally in my environment, unless I *specifically* request it. Since git doesn't have file-locking, some other team/PM/developer can decide it's time for <MASSIVE REFACTOR> that blows away my work/branch mistakenly (Or maybe even intentionally, especially if I work in an org that is terrible with communication), creating unnecessary merge conflicts/mental load. This happens in short-lived and long-lived feature branches.
In no setup, do I think it's ever safe to take away the developer's agency and let some other process keep my local machine in "sync." There are so many variables to account for that some daemon/service can't be aware of, to allow for automatic updating (and again automatic updating != user running `git fetch`).
> But in my working reality, testing/building is a non-linear process that is iteratively run after some certain self-determined point-in-time.
That’s not actually involved here. The actual process of coding can take place on a separate standalone project or even a whiteboard. But somehow all that code and everything associated with it needs to be packaged up for the team or it’s never making it to production. Further your process needs to minimally interrupt other team members or team efficiency tanks.
How that’s done is up to the team but it needs to happen somehow and automation avoids headaches. I briefly worked on a project where we kept passing around updates to a VM, slow and bandwidth intensive but it did actually work.
> "Production is, or should be deployable all the time,"
That’s a separate question. I am saying the code should be bundled with any needed environment configuration required to run that code. CI is a direct test of the process.
> In no setup, do I think it's ever safe to take away the developer's agency and let some other process keep my local machine in "sync." There are so many variables to account for that some daemon/service can't be aware of, to allow for automatic updating (and again automatic updating != user running `git fetch`).
Capacity is not a requirement.
For developers it’s about being able to hit a big red button and get your local environment working rather than something you automatically do day to day. Onboarding, or coming back after 3 months on another project, etc shouldn’t involve someone trying to piece together all the little environment changes that people need to apply since the last time someone updated the onboarding document.
In practice you might be reading a diff of some script and just apply that manually. But at least the sharing process is automated.
It’s the same problem with tests in general. Developers think it’s all about getting a green build. It’s not. It’s about fixing red builds as fast as possible. I have a whole other rant about making zero build failures a metric.
A red local build probably means a red CI build, but a red CI build doesn’t necessarily mean a red local build. Now you’re fucked because there’s a build failure you can’t reproduce.
Reducing variance helps. Having the person who set up the CI system also copy edit the onboarding docs helps a lot with this.
It’s a matter of scale. As the team grows and particularly as you hit the steep part of the S curve of development, you’re going to have lots of builds and that 1:50 error is going to go from every two weeks to once a day.
Humans interpret every day as “all the time”. Some do this for every week, especially if it coincides with their most important commits. It’s not the ratio of failures that bothers people. It’s the frequency, and the clusters.
This and the original article feel very much like solutions in search of problems. Beyond the issues with the original article, what's the point in a cryptographically secure hash from an untrusted environment? If someone were really determined to maliciously push broken code for some reason, they could just tamper with whatever generates the hash.
> provide a cryptographically secure hash proving that the tests passed
It’s unlikely that verifying such a hash is possible without rerunning the tests, and not at the same time enabling someone to trivially compute that hash without having run the tests in the first place.
Local testing doesn't tend to work outside the small scale. Even if you use containers for everything you'll run into edges where a local development machine can't accurately test a production-like environment. Deploying to production without testing in that environment is just tragedy waiting to happen.
Distributed signing of artifacts is only effective if you've got fully reproducible builds. If you don't, because almost no one does because it is a huge effort, then all you have is attestation. If a broken/malicious artifact gets deployed the damage gets done and you only know who to blame afterwards.
Local tests are useful for not knowingly committing broken code insofar as your tests can determine. Outside of that a full test suite run by a beefy cluster with ready access to assets and network resources is better suited to test for deployment.
"What if it was possible to have a local build system that would run tests etc and provide a cryptographically secure hash proving that the tests passed"
On who's system? Your? Mine? Someone else's? The purpose of CI is consistent continuous integration in a same like manner, not relying on developer A, B or C's systems which may vary greatly.
If your tests ran in a sandbox, for instance the Bazel sandbox or a Docker container, and this has the same configuration on CI and on the local machine, then it does not matter if the tests ran in CI or on a developer's machine.
This falls apart for things like integration tests, which may be too large/complex/interconnected to work on a local machine, but most of the time this would be more than sufficient.
It's not enough to just validate the tests: you would also want a tamper-proof record of exactly what source code was actually built. Probably also the full environment. 99.99% of the time this will be irrelevant trivia, but that one time you need to know, there's no substitute.
There’s a lovely feature of TeamCity that I had to enforce on one exuberant but doubly sloppy programmer.
In trunk based development you can do a conditional commit, where TC runs your code and only pushes it to trunk if the build is green. This allows you to push something before lunch or a meeting for someone who is blocked without coming back to angry faces because you ding-dong-dashed.
Breaking trunk is fundamentally a problem of response time. If you’re not in the office you can’t fix a red build in a timely fashion. He did this regularly and got put under house arrest.
I started using it on myself so people didn’t have to wait two hours for me to get out of a meeting and fix their api bug.
A previous employer used a passport analogy. Your code gets a visa (cryptographically signed) to run in a specific environment for a fixed time. You can renew your visa as long as all the assumptions remain true, but once they’re falsified you must create and submit a new build that makes the assumptions true again.
Right, we should just edit and test directly in production.
Jokes aside, one of the reasons why we still need a CI is it proves we can do a full clean build. Optimizations that rely on what actually changed are good for local dev.
One way to truly get rid of CI is to get rid of the need to do a full clean build. If we structure our build to be constructed from a series of immutables input files (easy with version control) with a series of pure (as in no side effects) build steps and hash the output file by its input + builds steps, then the concept of a clean build is meaningless.
Then you could even allow regular dev machines to push the build assets to a centralized build cache. However I would still want an independent "known good" oracle to rebuild and check the hashes. One could call this oracle a CI...
The complaint doesn't even make sense, the CI platform has all the change informations (the diff tells you that), running the tests locally won't give you any more useful input, and so makes no difference to the change-detection logic.
I fail to see the point being made in the article. CI done right is the best thing that ever happened to me in terms of collaborative productivity.
Of course there are many pitfalls. Like with any tool it becomes trivial to use it wrong. Integrating less than once per day and deferring all testing and linting to the CI process is an anti-pattern and the issue is not the CI process but how the team chooses to use it.
For example I worked in a team of 10 and we were doing multiple production deploys per day. This was made possible by a great CI workflow. Everyone was running tests precommit and focusing on keeping the pipeline green. Yes I've seen the opposite as well, but that usually is a symptom of issues with the team not the general concept of CI.
Not saying that there isn't room for improvement but all the real-time collaboration features I used on low-code platforms feel like a step back.
I think the point is about far shorter feedback loops than possible with today's pervasive tech.
> something even more continuous than continuous integration. It might look like real-time integration of your changes, collaborative editing, or smarter build systems that produce near-live feedback
Yes exactly, I was also thinking about how darklang does it. The feedback loop can be much shorter, and I'm sure it's not just limited to how darklang does it either.
Well, how short the feedback loop needs to be is something that can be controlled by the process, and the team sets the process.
I can have the normal flow, which takes minutes and runs all the tests and other stuff I might have on my pipeline (security, performance, etc). I use this for typical day to day work on features. I don't care if the deploy takes 50ms or 5 minutes.
I can have a fast-track for critical production patches. I skip all the main CI steps and just get my code quickly in production. If done right this takes seconds.
I didn't know about dark but I've seen this type of promise too many times, so I'm pretty sure there are many tradeoffs hidden under the nice shiny exterior. I can't know until I try it, but that kind of complexity doesn't all just disappear, i always has a cost even if it's out of sight.
Here's the thing: we have hundreds of thousands of teams and organizations who currently each need to select and define those processes.
Perhaps some of those teams and individuals are experts at determining the correct, minimal continuous integration suite to run on a per-commit basis to minimize time and energy expenditure without compromising correctness.
But I can guarantee that not all (in fact, not many) are, and that they pay maintenance and mental overheads to adhere to those practices.
It feels to me like there is a potentially massive opportunity to design an integrated language-and-infrastructure environment that only re-runs necessary checks based on each code diff.
- "Altered a single-line docstring in a Python file? OK, the data-flow from that infers a rebuild of one of our documentation pages is required, let's do that... done in 2ms"
- "Refactored the variable names and formatting within a C++ class without affecting any of the logic or ABI? OK, binary compatibility and test compatibility verified as unchanged, that's a no-op... done in 0ms"
- "Renamed the API spec so that the 'Referer' header is renamed 'Referrer'? OK, that's going to invalidate approximately a million downstream server and client implementations, would you like to proceed?"
(examples are arbitrary and should imply no specific limitations or characteristics of languages or protocols)
Doing this effectively would require fairly tight coupling between the syntax of the language, ability to analyze dataflows relating to variables and function calls, cross-package dependency management, and perhaps other factors.
Those properties can be achieved during design of a programming language, or they can iteratively be retrofitted into existing languages (with varying levels of difficulty).
Bazel[1] attempts to achieve much of this, although to my understanding it offloads a lot of the work of determining (re)build requirements onto the developer - perhaps a necessary migration phase until more languages and environments provide formats that have self-evident out-of-date status and dependency graphs.
We'll get there and the (sometimes uneasy) jokes will be about how often the software industry used to re-run huge continuous integration pipelines.
Honestly diversity is a good thing. It's good that different people are trying to solve problems at the heart of software in different ways as it creates new perspectives and possibilities. Now, after reading these replies I'm really curious about stuff like darklang and Bazel.
What happens is that people don't understand that most of these higher level solutions are very leaky abstractions, they aren't the silver bullet marketed in medium articles. Yes, they can save you a lot of time and headaches in specific scenarios but when you encounter one of the leaks it can take you weeks to get to the bottom of it.
If the teams doesn't understand what problems the tool is solving and if they have that problem then they might be just cargo-culting. An example of this is kubernetes. I know teams that used kubernetes just because everybody else is using it, they didn't actually need it for their monolithic java spring app. They think they avoided accidental complexity by using kubernetes but in fact they just added accidental complexity to their project. And then they add a new tool that makes it easier to manage the complexity of kubernetes, and so on.
Anyway, I'm probably just a rambling fool and I should appreciate that all this generating and shifting around of accidental complexity will actually mean future job safety for guys like us.
> deferring all testing and linting to the CI process is an anti-pattern
I'm confused as to why this is an anti-pattern? My understanding is that the CI pipeline should run unit tests and linting for every commit. But at the same time, developers should run their tests before pushing code.
Linters and tests should pass before code is rebased onto master. YOLOing code onto master is an antipattern because if your code breaks the build then you are holding up everyone else from being able to make changes until yours gets reverted. If linters and tests are already passing there is a very good chance the rebase won't break anything.
You can go a step further. Instead of merging anything, you can tell the ci to merge it. Then ci can make a "merge-test" branch, run all the tests on it, and if they pass, then ff-merge it to master for real. No need for "good chance" or rebasing just to keep up with master.
It does take some extra work though, because GH and others don't really support this out of the box.
Gitlab does support this with “Merged Result pipelines”[0]. We use them extensively alongside their merge train functionality to sequential is everything and it’s fantastic.
Isn’t this the way everyone works? Write code, run some tests/linting locally, then create a PR to master and the CI server runs all the tests and reports pass/fail. Changes without a pass can’t be merged to master.
The extra step is that once the change is accepted the process of rebasing it or merging it into master is done by a bot that checks if it would break master before preceding to do it.
My issue with this approach is that it becomes tricky to scale since you can only have one job running at a time. Allowing master to potentially break scales better because you can run a job for each commit on master which hasn't been evaluated. Technically you could make that approach work by instead of rebasing onto master rebasing onto the last commit that is being tested, but this adds extra complexity which I don't think standard tooling can easily handle.
https://zuul-ci.org/ and some other systems solve it by optimistic merges. If there's already a merge job running, the next one assumes that will succeed. And tests on merged master + first change + itself. If anything breaks, the optimistic merges are dropped from the queue and everything starts from the second chance only. Openstack uses it and it works pretty well if the merges typically don't fail.
This can result in a broken master if there were new commits added to master since the pull request was submitted (but before it is merged).
The solution is to require that all PR must be rebased/synced to master before they can be merged. GitHub has an option for enforcing this. The downside is that this often results in lots of re-running of tests.
Of course the CI should always run them, but that should normally be as a confirmation/safeguard.
I've seen too many cases where the devs wouldn't even run the code locally. They would push it and expect the CI to do all the work. That's how you get shitty CI that is always broken.
That’s why you use branches though. You can break the CI on your own branch as much as you want, it’s nobody’s business. But a broken CI on a dev branch MUST prevent merging to a release branch.
If you allow devs to push directly on release branch, thus breaking the CI, you’re absolutely doing it wrong.
Not that I don't see the value proposed by TBD, but I think you can have >90% of said value and none of the downsides using a well thought out branching strategy.
> Depending on the team size, and the rate of commits, short-lived feature branches are used for code-review and build checking (CI). [...] Very small teams may commit direct to the trunk.
TBD doesn't mean you have a red CI main branch. Of course it is always green on main. It means you have short lived feature branch and rely on runtime checks for feature gating. A broken main will halt TBD. You are mischaracterize what TBD is.
Well, I believe "absolutely doing it wrong" is a bit strong-worded.
Of course you can do it like you said but that means longer feedback loops in general. If the team wants to integrate more often and reduce feedback loops then that model evolves.
I'll give you an example.
In the team I mentioned in my top comment we were initially using a branching model with master, releases/, hotfixes/, dev, features/, which gradually evolved into master, dev, features/, which finally ended up as master, features/*. With the important mention that for small changes/fixes that needed to get deployed quickly nobody would bother with a branch they would just push to master.
This allowed us multiple production deploys per day per developer with no risk. That's why I said I don't get the point of the article, you can absolutely get those short feedback loops and continuous integration if you want it, just need to setup the process that way.
> Of course you can do it like you said but that means longer feedback loops in general. If the team wants to integrate more often and reduce feedback loops then that model evolves.
Longer feedback loops =/= long feedback loops. You can definitely wait 5 to 10 minutes if it means doing it right.
> With the important mention that for small changes/fixes that needed to get deployed quickly nobody would bother with a branch they would just push to master.
From my experience, the 1-2 lines fixes are the ones that benefit the most from automated CI because you’re doing it in a rush. In my team just last week a junior dev asked us to review their PR quickly because it was just 2 lines, and it didn’t even compile. We told them to be more careful in the future, but in the end it didn’t impact anything. It couldn’t possibly have impacted anything thanks to CI, it just makes it impossible to fuck up too recklessly.
From the perspective of the dev, if my local CI isn’t worth anything towards a merge and upstream CI is gospel why run locally? If I’m reasonably certain that the two jobs are duplicative in output it could be seen as wasted time, especially if I have a PM hounding for features. I don’t call that lazy, I call that a trade off. (coming from someone who is constantly running tests locally before pushing upstream)
Isn't that slower and less efficient? Usually the CI has to run a full build from scratch before it can run the tests, but locally for me it's going to be an incremental build that takes a second or two. I can also run the subset of tests that I know are affected by my changes to get fast and reasonably reliable feedback vs waiting for CI to run the whole test suite.
It wouldn't need to be pre commit but pre push. Integrate vc with test runner and don't allow a push to certain remotes without an associated green test suite.
Maybe this will happen, but I really doubt it will come anytime soon, especially since the original tweet mentions that local, laptop-based testing is enough, which is laughingly ridiculous in terms of resources, time and coverage for anyone who's worked on large, real-life, many-year, bloated projects.
: which is pretty much every project that survived the POC phase
"Most" and "everybody" on HN often means "we at the few megacorps around".
Let me assume instead that most software projects happen in the long tail of small companies, low LOC numbers, low developer head counts (even one or zero - consultants working only when needed.) In that world I saw deployments run when tests pass on one PC, deployed from that PC. In the case of scripting languages even with a git pull on the server and a restart. That works surprisingly well until the team gets larger. Then customers often resist the extra costs and burden of CI. Then some of them cave in after something bad happens.
I think these tiny or zero teams benefit even more from CI. As people come and go the CI process remains and people can use it without any local setup. Use GitHub actions to avoid main thing yet another thing (assuming that’s where the source code lives)
I like CI tools that run locally on a developer machine and run end to end tests with all Microservices locally. The secret is integrating early and often.
That's why I wrote mazzle and platform-up.
Mazzle is a run server that is configured as a Graphviz file (.dot) and defines an end to end environment. Here's an example graph file of a pipeline:
It's largely a prototype of an idea. It's infrastructure and pipelines as code but incorporates every tool you use from terraform, chef, ansible, puppet, Kubernetes, packer and shell scripts. My example code spins up a Consul and Kubernetes cluster with hashicorp vault and a Debian repository server, configured SSH keys on every worker, bastion and Java application. And Prometheus exporter and grafana. I haven't got around to adding ELK yet. But it didn't take long to do all these things due to Mazzle meaning it's very easy to test complicated changes together.
Platform up is a pattern for local development that tests your Microservices locally all together. You use vagrant-lxc and ansible together to deploy everything locally. So you can test your secret management locally and deployment process. If your ELK stack is ansible driven you can even run your ELK stack locally as I did on a client project.
https://GitHub.com/samsquire/platform-up
1. I still wouldn't put it past developers to lie occasionally.
2. You don't control the test environment. What if the tests pass on the developers machine but not on the CI machines?
3. Difficult to test on multiple platforms. What if the developer uses Mac but you need to support Windows too?
4. Testing can take a long time. Who wants to wait an hour to submit a minor PR? Plus it ties up resources on your machine. CI can automatically scale.
5. There's no way to avoid races between testing and integration (i.e. merge queues).
CI isn't going anywhere. I think the biggest scope for improvement to CI is
1. Only testing things that have changed. Most people don't do that because it requires a build system that properly isolates everything. Basically only Bazel and it's derivatives do this. You can't do it with CMake or NPM or Cargo or ...
2. Make it easier to run CI on your own machine. I don't think there's really any technical barrier to this, it's just people don't usually bother.
This reminds me of a brief discussion/feature-request on the nektos/act GitHub repository (a project that allows developers to run GitHub Actions workflows locally):
> Trust in the developer is required ... but it is likely the case that developers are incentivized to illustrate that they are reliable, careful and trustworthy -- and that should strongly encourage accurate test result signing
100% guaranteed you're going to see "oh I just made a stupid typo that broke one test but I can see what is wrong so I'll just fix it and use the previous test results".
Also since it's entirely based on trust anyway I'm not sure what additional benefit signing gets you. I don't think there's really a way to prove you ran the tests.
Either way, all my other points still stand. CI isn't going anywhere.
> Also since it's entirely based on trust anyway I'm not sure what additional benefit signing gets you. I don't think there's really a way to prove you ran the tests.
It's a reputation and trust-building exercise, essentially - and that's one of the reasons that continuous integration is particularly useful. "Entities X, Y, Z all say that commit <ID> looks good".
> Either way, all my other points still stand. CI isn't going anywhere.
It seems like a core argument is the pre-commit tests that runs as commit hooks on the developers computer.
I have worked in a place where they did that, and I think the cons heavily outweighed the pros. I can not push incomplete work to a remote, I can not separate feature development and chores (eg. linting) because I _need_ to fix stuff in order to commit an push, etc.
> Continuous Integration is a model thats tightly tied to how we run version control.
I would say that a pre-commit testing world is much tighter. CI, as many know it, as a passive observer. When stuff fails, it will let you know. You can still merge.
One thing that would be nice, however, would be the ability to run the entire pipeline locally. For GitHub actions, it indeed seems like there are viable options to do that.
I do feel like if we’re going the precommit route we need to go much deeper into things like Bazel. Loads of companies are using CI to get a huge test suite to run with a lot of parralelsim for a reason!
Though for me it’s also like… so many people do unspeakable horrors to their machine setups that I like there being a consistent runner in CI.
But “CI is to run tests” in the world of CD is a bit of a simplification anyways.
> Loads of companies are using CI to get a huge test suite to run with a lot of parralelsim for a reason!
Tho technically you can probably have a local runner which either distributes the test suite across a bunch of available matchines (à la pytest-xdist), or one which goes and creates jobs on the CI platform without needing to go through the pretest ceremony (e.g. creating commits, branches, CRs, ...)
Right, bazel lets you do that (it has no notion of version control state), and ships it off remotel if you so choose to have the system work like that.
I stand by this idea that if someone made Bazel, but without all the obtuseness, with strong integration to a hosted CI service, they would quickly overtake a lot of services.
> One thing that would be nice, however, would be the ability to run the entire pipeline locally. For GitHub actions, it indeed seems like there are viable options to do that.
I prefer not having to replicate locally how GitHub runs GitHub actions, but rather just make my GitHub actions run something that I know I can run locally. So all the complicated stuff is contained in the script that I can run locally, and GitHub actions is just a dumb “runner” of this script.
For my local script I prefer using Nix, since it’s the best way I know to guarantee a consistent environment between different machines.
That where I use Makefiles. For GitLab CI, for example, you can have the likes of `make $CI_JOB_NAME` so your CI config can be very short and dry, just a bunch of jobs named after Makefile targets.
git has cli flags that will happily let you bypass pre commit/push checks so it's not even a reliable way to know that tests actually got run. Central CI gives you a traceable test history of every change and keeps us all honest.
This might break the teams assumption that these things pass. After all, centralized CI has, in this narrative, been exchanged for the assumption that everything pushed to git passes all tests.
We use nektos act to run pipelines locally. Works OK, but everytime something fails locally you are left wondering if its act or your pipeline, and some features (like reusable workflows) are not implemented yet.
Maybe the right thing is to just write the workflows in something else, and have the github workflow file be a single call to your script. But
-It would be nice to be able to use github actions others has made (or libraries/abstractions in general)
- I don't see how to easily get parallell execution.
- I love github environments, how do I pass down all my environments and their variables?
> One thing that would be nice, however, would be the ability to run the entire pipeline locally.
This cost me many hours of waiting for the Gitlab CI runner when debugging non-trivial pipelines, when the issue was something that did not have to do with the script steps inside of the jobs but rather how the Gitlab runner handled things.
I've found gitlab-ci-local [1] which actually does run the Gitlab pipeline locally, although I had to write some boilerplaye scripts to set up all the necessary 'CI_FOO_SOMETHING' environment variables before running the tool. (Which sometimes came back to bite me because the issue was actually in the content of some of those environment variables). It's still a very good tool.
It sounds like a problem with the policy, not the tool. Pre-commit hooks is a useful tool but there should be a way to skip them if you really need it.
Per-commit hooks save my time by not allowing to push code which guaranteed to break later in CI.
"One thing that would be nice, however, would be the ability to run the entire pipeline locally."
I'm working on something to do just this. Although I've redefined what "local" means in this context. I'm still using remote servers but everything is happening pre-commit from the terminal. If you are interested check out this demo and let me know if you have any feedback https://brisktest.com/demos
I don’t like the hand-wavey “git is already showing it’s weakness”, then provides no examples.
This is a twitter tier take with no insight to back it up except “things will continue to get better and soon things will be different!” But doesn’t paint a picture of what different might be.
Many integrations between CIs and git frontends (as in showing green checkmarks under your pull request) only work properly if you push. Rebuilding manually in the CI will run the build, but it will no longer be recognized as related to a PR
The only integration I've used like that is github workflows and only because the project owner won't turn on the feature to allow manual re-run . Fortunately transient failures are rare on that project, but I still find it rather odd.
developers become reliant on CI and won't run the tests until merge-time
I think there's a whole lot of truth in this particular statement. I've worked with lots of devs who complete a story, commit it, and then find the tests aren't passing any more and have to do a little more work to fix what they broke. And then tery complain that either tests slow them down or that the estimate for the story was too low. Having a test suite that can be run locally, and teaching the team to use it regularly, even if it's wholly optional rather than running on a hook, improves team velocity significantly.
Why only one commit for a story? Sounds like they should be committing more often.
I often fire off a commit, get notified of a test failure, and fix it in the next commit. Why wait for tests to run locally when some other computer can do it for you?
(Yes, it would be nice if all tests ran in zero time, but back in the real world...)
- trunk-based workflow. Small commits. No feature branches as a rule to be occasionally broken
- unit tests move left - run pre-commit (not necessarily run in CI). Emphasis placed on them as a refactoring tool.
- a critical, small suite of e2e and integration tests that block the pipeline before publication (fast feedback)
- latest build publication being constantly redeployed to production, even if no changes have taken place to exercise the deployment routines
- a larger suite of e2e and integration tests being constantly run against production, to give feedback when something isn't quite right, but it's not a disaster (slow, non-blocking feedback).
In summary, emphasise getting code into production, minimise blocking tests to critical ones, test in production & notify when features are broken.
Why?
- Engineers spend too much time in test environments that give the illusion of the real thing. They lose touch with production as the Dev cycle increases in circumference.
- Enabling tighter feedback cycles by accepting that some features are important and some are not helps put the cost of change into perspective for the entire product team.
- Engineers get used to working in and around production on a daily basis. Production operations and observation of systems (& users) are emphasised - protective bubbles for writing code are minimised.
You're not trying to maximise code output, you're trying to maximise the velocity of safe change, and you do that by understanding the environment (production) through your intelligence (observability of systems and user behaviour), so that you can employ your force (changes, code, features) rapidly and effectively, whilst maintaining the ability to quickly deal with unexpected problems (defects) along the way.
Disclaimer: might not be possible for your specific theatre of war for any number of reasons.
Why bother running a test pipeline at all? Why not have two production environments running in red/green, and always just run your e2e tests then flip.
That's a viable variation on the theme - but there may be some critical sanity tests you want to run before you let code near a production environment - depends on your appetite for risk of course.
Some of this is good, and some of it I feel makes some fundamental misunderstandings. The good: "tight feedback loops are really important". Absolutely. Getting feedback about a change you (as a developer) are making is key, and the longer that takes the more time is spent waiting which is annoying and concentration breaking. This is, in my experience, usually tied to the implementation of tests (read: not bloated and useless) and not really a core tenant of CI.
However, things like:
[...] git is already showing its weaknesses. Fork-and-pull isn't the final form of development workflows.
Show that the author is perhaps conflating GitHub with Git.
>First, CI rarely knows what has changed, so everything needs to be rebuilt and retested.
That's a problem with your building and testing tools. Even without CI you would still need to build everything and test everything if you have no way to do it just for what changed.
>How often have you or a coworker pushed an empty commit to "re-trigger" the CI
Most CI solutions have a button to trigger a new version without a new commit.
>Running tests pre-commit is one answer
Even with CI developers are likely running at least a subset of the test suite while they are developing.
>Yet, there are roadblocks today that need to be fixed, like tests that don't fit on your laptop
Either develop on a server while using your laptop as just an editor or have a test runner on the server.
>Things are shifting left
The problem with this is that at large companies the changes you've already tested will always be rebased onto a newer version of the codebane which it hasn't been tested with. Who runs tests for this newly rebased version? CI. Also for things like come review you can not push it left onto the devs machine. You will want linters and tests to be run for come review.
CI is a collaboration practice. It means to share your work with your team very often, to not develop little parts in silos for weeks or months and then put it all together at the very end.
I think the author means "the end of build servers". You can do CI fine without build servers.
Yeah, it's quite weird to see everyone discussing "CI" as some particular tool or workflow (and not the same one...) and not as a general ideal or practice.
When I read the title I automatically assumed this would be a "...because CI won and is everywhere now" sort of closer. Junior devs' eyes go wide when you tell them builds used to happen overnight, "feature branches" could last months or even years, and merging was someone's full-time job.
"I, for one, hope that we aren't using Jenkins in 2025"
Or at least, if we are, it's a far more robust and capable system that's actually really designed for dealing with all the different things that people need to do while building and testing software.
But surely most IDEs these days make it simple to run all unit tests locally? My usual M.O. is to manually run the tests locally that I believe will demonstrate my changes are "working" and rely on the full suite of CI unit tests to check I haven't broken anything else. What other options are there?
I've moved to develop most of the time on a (time-share) beefy machine on AWS, so my "locally" is always a pretty strong machine.
While I develop I start a `watch` process that keeps running the tests. The watch runs a container with the docker image of the CI, mounting only my development working directory.
Well there are plenty of possible projects for which CI has no value, I don't think that's a controversial statement. But anything of any complexity with a decent size team (where a full CI build is likely to take 20 min+), then I can't imagine being convinced it wasn't needed.
I find it funny how CI is not continuous and not integration.
It’s inherently a batch process, so not continuous.
It’s an automated build and check system. It also runs on merge commits but this “integration” part is really marginal to the concept as represented by e.g. GHA.
I guess it got its name from the contrast to older practices. Software projects used to divide parts up considerably as part of system design and give them to individuals. Separate compilation, like in Ada, was supposed to allow compiling to interfaces without having the implementation available. After the detailed design and implementation was finished, the parts could theoretically be integrated during a period called "integration hell".
I remember the MCSD docs were promoting a "Daily build and smoke test". This was a huge difference in that they promoted at least making sure that thev whole system could be built in some state every day.
CI really appeared in the late 90s when someone had the idea that integration tests could be run for small gains in functionality. Then the software could be automatically tested on every incremental change. I credit C3/XP for popularizing the practice but I'm not a historian. Possibly someone was already at it before.
There have been times where I would not know how to locally test code or did not want to install a bunch of dependencies on my system. I don't think precommit workflows will entirely replace CI, but it's rise will surely ensure that commits and commit history will be higher quality. Testing locally used to require installing additional dependencies and testing suites but this could be simplified with containers.
The first point is not true at all. GitLab CI has a feature when it can run jobs conditionally based on what changed.
We did the same with Jenkins, it's not hard.
237 comments
[ 2.4 ms ] story [ 245 ms ] threadHowever on the topic of real-time integration that is somewhat already here for no-code systems that are already divorced from traditional version control.
When I say tech debt I mean design flaws created by designer who simply could not anticipate the future. I'm not talking about bugs, mistakes or intentional shortcuts.
Whether jenkins is "tech debt" is a different topic. But whether Jenkins is here to stay past 2025 has nothing to do with how well it's designed.
I do worry that a lot of people are missing the first principles where CI is concerned, but a few of us still remember. Whether that’s enough is always a question.
> First, CI rarely knows what has changed, so everything needs to be rebuilt and retested
CI is the impartial judge in your system. That is the ultimate goal and much of the value. Don’t believe me when I say that your check in broke something. Believe the robot. It does not think, it does not interpret errors (which is why intermittent failures poison the well). It just knows that whatever you did was not good.
We make it stupid simple because the more ways you try to interpret things, the higher the likelihood the system will either miss an error or report imaginary ones. And in this case I am very sorry Mr Lasso, but it is in fact the hope that kills. Optimism makes us try things, things others would give up on. That’s what makes a piece of software good. Better. But too much and you start ignoring very real problems and make optimism into a liability. I’ve seen it over and over again, take a build with intermittent failures and people will check in broken code and assume it’s the intermittent problem and not their change.
Ideally, when you start breaking builds you’ve found your limits. But the build has to break when I break the code, and it needs to do it repeatably, because if I can’t reproduce the error I risk telling myself stories, and foisting off debugging to other people, which is a big no no.
If it doesnt build on CI, you broke it, fix it before merging.
Though, I'm still sour from my only experience in "tech" where "works on my machine" meant devs got to demand I fix the CI somehow.
A small but important part of my motivation to build better DevEx is that it puts a squeeze on the foot draggers. If you can’t blame the tools for being shit (and believe me, I know a lot of them are), then the only other explanations are that you are either too stupid to use them or just not a team player. An asshole.
Developers will pick being called an asshole over being seen as stupid any day, but the former eventually gets management involved. It’s the closest I’ll probably ever get to solving a social problem with tech. Make it easy for people to participate, make them feel silly for not, then ramp up the pressure if that doesn’t work, and when it’s clear they only care about themselves you have actionable evidence to involve HR.
Common examples of this are multi-million line C++ codebases (e.g proprietary game engines) and monorepos in any language.
Running tests on my computer uses up valuable CPU cycles that I can use to work on something else while the CI servers are running my tests.
However I can see this working for small to medium sized codebases that really don't need CI for testing.
We have a project where the team split the tests into chunks and eventually I figured out the reason why is because they had coupling between tests, and running them all together ran into problems. I worry about other people opening that door, because it’s damned hard to close again.
How well does it work? Well .... it sorta mostly works. Sometimes a build will fail for inexplicable reasons because Gradle/Kotlin incremental builds don't seem fully reliable. You re-run with a clean checkout and the issue goes away.
How much time does it save? Also hard to say. Most of the time goes into integration tests that by their nature are invalidated by more or less any change in the codebase. That's not exactly incorrect. Some changes in some modules do avoid hitting the integration tests, though.
TC can also do test sharding in the newest versions when you use JUnit. We don't use this yet though.
Capacity planning applies to tests. As your test count goes up your budget per test goes down. Every CI tool should plot build and test duration over time and only a couple do.
There’s a reason most test frameworks can mark slow tests. You need to not only use that but ratchet down over time. Especially when you get new hardware.
I haven’t run benchmarks lately but my old rule of thumb developed over many projects and with several people better at testing than I, was a factor of eight for each layer in the testing pyramid.
That certainly puts a lot of runtime pressure on the top of the pyramid, but that’s by design. You don’t want people racing to the top, because that’s how you get a cone.
In large part, industry could largely use a significantly simpler model than git for VCS since many features are largely moot on a daily basis. Many could probably get away with SVN in large part, for example (although branching isn't nearly as good).
I think you raise a great point in that we need to look at how processes have evolved ontop of version control and look at adapting those to a similar model. In some cases it's just not practical because the way infrastructure testing an deployment works but it's the direction I think we should be going. In fact, the first step would be to make issue systems and project management interfaces provided by efforts like GitLab and GitHub available in a local capable, distributed fashion. Clearly communications often require some degree of centralization or at least peer message propagation but there's no reason that information can't be separated from the infrastructure that displays and interacts with it.
Indeed. Software forges centralized what was a Decentralized VCS.
For one reason: profit.
I agree that centralization is probably not a hard requirement for these properties, but currently no one has come up with decentralized equivalents that provide users what they actually want - and remember, you don't get to tell them what they want!
You just described https://en.wikipedia.org/wiki/Embrace,_extend,_and_extinguis...
We developed distributed VCSes and the ability to do smart things with branches and combining contributions from people all over the world at different times. Then we somehow still ended up with centralised processes where everyone has to merge to trunk frequently and if GitHub goes down then the world stops turning. It turns out that just using git as git wasn't such a bad idea after all.
We were told that dynamic languages like JS and Python are so much more productive and that this was essential for fast-moving startups to be competitive. Today the industry is moving sharply from JS to TypeScript and even Python now has similar tools. It turns out that static typing was better for building robust software at scale than relying entirely on unit tests after all.
Just wait until lots of developers who have only ever worked with JS or Python learn what a compiled executable is. If you tell them that xcopy deployment even to a huge farm of web servers is no big deal when all you're copying is one executable file and maybe a .env with the secrets, it'll blow their minds...
Well, technically it doesn’t have to. It needs to be „a“ CI server that the team agrees upon, not necessarily a GitHub one.
I think the most important factor is to keep in mind that this is something teams decide to do.
You can certainly argue that there are other advantages to using those systems. However it's an inescapable fact that if those teams had been using git-as-git and had a good local environment for each developer then most of the developers would have been able to carry on with most of their work during all of those outages. Sometimes single points of failure fail.
Seriously, a well set up gutlab is pretty much undownable. And even if you do, what really is the impact? If the pipelines don't run you cannot integrate, sure if the core service breaks down you will probably resolve to exchanging patches the old school way. But the good part is, in order to deploy those decentrally sourced changes you still will go through centralized CI and gain its quality assurances by doing so. Where's the drawback?
Given that many companies now tie their entire deployment process into their source control and CI/CD systems that means you really are reduced to exchanging patch files. For any non-trivial change in a large system that quickly becomes impractical and so development slows to a crawl or everyone literally gives up and goes to the pub.
A different view on this is that my CI system is a kind of colleague, responsible for a lot of what testers, ops, and build eng people toiled at before. It is another aspect of the decentralized system where a robot can also check out and work with the code even while humans continue to write more, which is considerably more painful under e.g. the SVN model. My human colleagues and I still share plenty of code directly, via git pulls and other tools.
In theory you could get something reasonably close to this working locally, but the it’s a serial process so that’s pointless.
I’m assuming you are only submitting this as theoretical, because in the real world (where I’ve directly experienced this workflow) it’s a nightmare:
* Someone merges main which deletes working files in your feature branch. Automatic pushing/rebasing of main onto your feature branch creates needless work the moment you try to push upstream when git tries to ascertain state of local main.
* countless times I’ve hit bugs where if I try to merge “patch-a” from a common “feat-1” branch (meaning patch was cut from feat, not from main), but then main is updated by the auto-updater, I then have a messy working directory in which main’s new files are treated as unknown orphans and I have to spend time deleting these by hand.
I’m all for having my feat branch be up to date at merge time. But making it a rolling target is something git (from my perspective) hits the boundary of what git can reasonably do, and creates more pain than any type of positive DX
Git may or may not be part of that process.
>Ideally starting from a blank machine/VM you should be able to run a single script that get’s the latest, builds it locally, runs it successfully, and passes all tests.
This isn't automatic, and is a practice that is fairly standard today (And one I agree with). You have placed a requirement/line in the sand that says, "Only when I have ideal state should this pipeline run in linear time and output the final result, which are the return codes from tests." Your example has the user initiating the update of the upstream main, not some other process that runs git fetch on the Developer's behalf.
Your original comment, as I understand it, is contradictory to this point:
>or code to function on CI machine will also be automatically loaded into production and other developers machines.
All of us (I think) agree that auto-deployment to production is a desirable goal. But we all (I think) know that broken commits are routinely delivered to production, where "production" represents the sum of all production environments in the world. So while we can have a reasonable assumption that "Production is, or should be deployable all the time," that doesn't mean the state that is represented by Production is safe to run locally in my environment, unless I *specifically* request it. Since git doesn't have file-locking, some other team/PM/developer can decide it's time for <MASSIVE REFACTOR> that blows away my work/branch mistakenly (Or maybe even intentionally, especially if I work in an org that is terrible with communication), creating unnecessary merge conflicts/mental load. This happens in short-lived and long-lived feature branches.
In no setup, do I think it's ever safe to take away the developer's agency and let some other process keep my local machine in "sync." There are so many variables to account for that some daemon/service can't be aware of, to allow for automatic updating (and again automatic updating != user running `git fetch`).
That’s not actually involved here. The actual process of coding can take place on a separate standalone project or even a whiteboard. But somehow all that code and everything associated with it needs to be packaged up for the team or it’s never making it to production. Further your process needs to minimally interrupt other team members or team efficiency tanks.
How that’s done is up to the team but it needs to happen somehow and automation avoids headaches. I briefly worked on a project where we kept passing around updates to a VM, slow and bandwidth intensive but it did actually work.
> "Production is, or should be deployable all the time,"
That’s a separate question. I am saying the code should be bundled with any needed environment configuration required to run that code. CI is a direct test of the process.
> In no setup, do I think it's ever safe to take away the developer's agency and let some other process keep my local machine in "sync." There are so many variables to account for that some daemon/service can't be aware of, to allow for automatic updating (and again automatic updating != user running `git fetch`).
Capacity is not a requirement.
For developers it’s about being able to hit a big red button and get your local environment working rather than something you automatically do day to day. Onboarding, or coming back after 3 months on another project, etc shouldn’t involve someone trying to piece together all the little environment changes that people need to apply since the last time someone updated the onboarding document.
In practice you might be reading a diff of some script and just apply that manually. But at least the sharing process is automated.
A red local build probably means a red CI build, but a red CI build doesn’t necessarily mean a red local build. Now you’re fucked because there’s a build failure you can’t reproduce.
Reducing variance helps. Having the person who set up the CI system also copy edit the onboarding docs helps a lot with this.
It’s a matter of scale. As the team grows and particularly as you hit the steep part of the S curve of development, you’re going to have lots of builds and that 1:50 error is going to go from every two weeks to once a day.
Humans interpret every day as “all the time”. Some do this for every week, especially if it coincides with their most important commits. It’s not the ratio of failures that bothers people. It’s the frequency, and the clusters.
CI itself is just continually testing before continually merging. Doesn't matter where.
It’s unlikely that verifying such a hash is possible without rerunning the tests, and not at the same time enabling someone to trivially compute that hash without having run the tests in the first place.
Distributed signing of artifacts is only effective if you've got fully reproducible builds. If you don't, because almost no one does because it is a huge effort, then all you have is attestation. If a broken/malicious artifact gets deployed the damage gets done and you only know who to blame afterwards.
Local tests are useful for not knowingly committing broken code insofar as your tests can determine. Outside of that a full test suite run by a beefy cluster with ready access to assets and network resources is better suited to test for deployment.
On who's system? Your? Mine? Someone else's? The purpose of CI is consistent continuous integration in a same like manner, not relying on developer A, B or C's systems which may vary greatly.
This falls apart for things like integration tests, which may be too large/complex/interconnected to work on a local machine, but most of the time this would be more than sufficient.
In trunk based development you can do a conditional commit, where TC runs your code and only pushes it to trunk if the build is green. This allows you to push something before lunch or a meeting for someone who is blocked without coming back to angry faces because you ding-dong-dashed.
Breaking trunk is fundamentally a problem of response time. If you’re not in the office you can’t fix a red build in a timely fashion. He did this regularly and got put under house arrest.
I started using it on myself so people didn’t have to wait two hours for me to get out of a meeting and fix their api bug.
Jokes aside, one of the reasons why we still need a CI is it proves we can do a full clean build. Optimizations that rely on what actually changed are good for local dev.
One way to truly get rid of CI is to get rid of the need to do a full clean build. If we structure our build to be constructed from a series of immutables input files (easy with version control) with a series of pure (as in no side effects) build steps and hash the output file by its input + builds steps, then the concept of a clean build is meaningless.
Then you could even allow regular dev machines to push the build assets to a centralized build cache. However I would still want an independent "known good" oracle to rebuild and check the hashes. One could call this oracle a CI...
Of course there are many pitfalls. Like with any tool it becomes trivial to use it wrong. Integrating less than once per day and deferring all testing and linting to the CI process is an anti-pattern and the issue is not the CI process but how the team chooses to use it.
For example I worked in a team of 10 and we were doing multiple production deploys per day. This was made possible by a great CI workflow. Everyone was running tests precommit and focusing on keeping the pipeline green. Yes I've seen the opposite as well, but that usually is a symptom of issues with the team not the general concept of CI.
Not saying that there isn't room for improvement but all the real-time collaboration features I used on low-code platforms feel like a step back.
> something even more continuous than continuous integration. It might look like real-time integration of your changes, collaborative editing, or smarter build systems that produce near-live feedback
e.g. https://medium.com/darklang/how-dark-deploys-code-in-50ms-77...
I can have the normal flow, which takes minutes and runs all the tests and other stuff I might have on my pipeline (security, performance, etc). I use this for typical day to day work on features. I don't care if the deploy takes 50ms or 5 minutes.
I can have a fast-track for critical production patches. I skip all the main CI steps and just get my code quickly in production. If done right this takes seconds.
I didn't know about dark but I've seen this type of promise too many times, so I'm pretty sure there are many tradeoffs hidden under the nice shiny exterior. I can't know until I try it, but that kind of complexity doesn't all just disappear, i always has a cost even if it's out of sight.
Perhaps some of those teams and individuals are experts at determining the correct, minimal continuous integration suite to run on a per-commit basis to minimize time and energy expenditure without compromising correctness.
But I can guarantee that not all (in fact, not many) are, and that they pay maintenance and mental overheads to adhere to those practices.
It feels to me like there is a potentially massive opportunity to design an integrated language-and-infrastructure environment that only re-runs necessary checks based on each code diff.
- "Altered a single-line docstring in a Python file? OK, the data-flow from that infers a rebuild of one of our documentation pages is required, let's do that... done in 2ms"
- "Refactored the variable names and formatting within a C++ class without affecting any of the logic or ABI? OK, binary compatibility and test compatibility verified as unchanged, that's a no-op... done in 0ms"
- "Renamed the API spec so that the 'Referer' header is renamed 'Referrer'? OK, that's going to invalidate approximately a million downstream server and client implementations, would you like to proceed?"
(examples are arbitrary and should imply no specific limitations or characteristics of languages or protocols)
Doing this effectively would require fairly tight coupling between the syntax of the language, ability to analyze dataflows relating to variables and function calls, cross-package dependency management, and perhaps other factors.
Those properties can be achieved during design of a programming language, or they can iteratively be retrofitted into existing languages (with varying levels of difficulty).
Bazel[1] attempts to achieve much of this, although to my understanding it offloads a lot of the work of determining (re)build requirements onto the developer - perhaps a necessary migration phase until more languages and environments provide formats that have self-evident out-of-date status and dependency graphs.
We'll get there and the (sometimes uneasy) jokes will be about how often the software industry used to re-run huge continuous integration pipelines.
[1] - https://en.wikipedia.org/wiki/Bazel_(software)
What happens is that people don't understand that most of these higher level solutions are very leaky abstractions, they aren't the silver bullet marketed in medium articles. Yes, they can save you a lot of time and headaches in specific scenarios but when you encounter one of the leaks it can take you weeks to get to the bottom of it.
If the teams doesn't understand what problems the tool is solving and if they have that problem then they might be just cargo-culting. An example of this is kubernetes. I know teams that used kubernetes just because everybody else is using it, they didn't actually need it for their monolithic java spring app. They think they avoided accidental complexity by using kubernetes but in fact they just added accidental complexity to their project. And then they add a new tool that makes it easier to manage the complexity of kubernetes, and so on.
Anyway, I'm probably just a rambling fool and I should appreciate that all this generating and shifting around of accidental complexity will actually mean future job safety for guys like us.
I'm confused as to why this is an anti-pattern? My understanding is that the CI pipeline should run unit tests and linting for every commit. But at the same time, developers should run their tests before pushing code.
It does take some extra work though, because GH and others don't really support this out of the box.
[0]: https://docs.gitlab.com/ee/ci/pipelines/merged_results_pipel...
My issue with this approach is that it becomes tricky to scale since you can only have one job running at a time. Allowing master to potentially break scales better because you can run a job for each commit on master which hasn't been evaluated. Technically you could make that approach work by instead of rebasing onto master rebasing onto the last commit that is being tested, but this adds extra complexity which I don't think standard tooling can easily handle.
https://zuul-ci.org/ and some other systems solve it by optimistic merges. If there's already a merge job running, the next one assumes that will succeed. And tests on merged master + first change + itself. If anything breaks, the optimistic merges are dropped from the queue and everything starts from the second chance only. Openstack uses it and it works pretty well if the merges typically don't fail.
The solution is to require that all PR must be rebased/synced to master before they can be merged. GitHub has an option for enforcing this. The downside is that this often results in lots of re-running of tests.
Of course the CI should always run them, but that should normally be as a confirmation/safeguard.
I've seen too many cases where the devs wouldn't even run the code locally. They would push it and expect the CI to do all the work. That's how you get shitty CI that is always broken.
If you allow devs to push directly on release branch, thus breaking the CI, you’re absolutely doing it wrong.
I agree with you though.
Not that I don't see the value proposed by TBD, but I think you can have >90% of said value and none of the downsides using a well thought out branching strategy.
[0] Trunk-Based Development: https://trunkbaseddevelopment.com/
> Depending on the team size, and the rate of commits, short-lived feature branches are used for code-review and build checking (CI). [...] Very small teams may commit direct to the trunk.
Sometimes TBD is the answer, sometimes you need something else.
What I did notice is that, with time, mature teams end up simplifying processes in order to reduce friction and increase output.
Of course you can do it like you said but that means longer feedback loops in general. If the team wants to integrate more often and reduce feedback loops then that model evolves.
I'll give you an example. In the team I mentioned in my top comment we were initially using a branching model with master, releases/, hotfixes/, dev, features/, which gradually evolved into master, dev, features/, which finally ended up as master, features/*. With the important mention that for small changes/fixes that needed to get deployed quickly nobody would bother with a branch they would just push to master.
This allowed us multiple production deploys per day per developer with no risk. That's why I said I don't get the point of the article, you can absolutely get those short feedback loops and continuous integration if you want it, just need to setup the process that way.
Longer feedback loops =/= long feedback loops. You can definitely wait 5 to 10 minutes if it means doing it right.
> With the important mention that for small changes/fixes that needed to get deployed quickly nobody would bother with a branch they would just push to master.
From my experience, the 1-2 lines fixes are the ones that benefit the most from automated CI because you’re doing it in a rush. In my team just last week a junior dev asked us to review their PR quickly because it was just 2 lines, and it didn’t even compile. We told them to be more careful in the future, but in the end it didn’t impact anything. It couldn’t possibly have impacted anything thanks to CI, it just makes it impossible to fuck up too recklessly.
Lazy devs are going to be lazy no matter what processes their team uses.
It's two different worlds.
: which is pretty much every project that survived the POC phase
Let me assume instead that most software projects happen in the long tail of small companies, low LOC numbers, low developer head counts (even one or zero - consultants working only when needed.) In that world I saw deployments run when tests pass on one PC, deployed from that PC. In the case of scripting languages even with a git pull on the server and a restart. That works surprisingly well until the team gets larger. Then customers often resist the extra costs and burden of CI. Then some of them cave in after something bad happens.
That's why I wrote mazzle and platform-up.
Mazzle is a run server that is configured as a Graphviz file (.dot) and defines an end to end environment. Here's an example graph file of a pipeline:
https://github.com/samsquire/mazzle/blob/master/docs/archite...
It's largely a prototype of an idea. It's infrastructure and pipelines as code but incorporates every tool you use from terraform, chef, ansible, puppet, Kubernetes, packer and shell scripts. My example code spins up a Consul and Kubernetes cluster with hashicorp vault and a Debian repository server, configured SSH keys on every worker, bastion and Java application. And Prometheus exporter and grafana. I haven't got around to adding ELK yet. But it didn't take long to do all these things due to Mazzle meaning it's very easy to test complicated changes together.
https://devops-pipeline.com - Mazzle
Platform up is a pattern for local development that tests your Microservices locally all together. You use vagrant-lxc and ansible together to deploy everything locally. So you can test your secret management locally and deployment process. If your ELK stack is ansible driven you can even run your ELK stack locally as I did on a client project. https://GitHub.com/samsquire/platform-up
Nobody does it because developers can't be trusted to actually run tests.
Perhaps if there was some way to cryptographically prove that you've run tests? That doesn't sound possible though.
1. I still wouldn't put it past developers to lie occasionally.
2. You don't control the test environment. What if the tests pass on the developers machine but not on the CI machines?
3. Difficult to test on multiple platforms. What if the developer uses Mac but you need to support Windows too?
4. Testing can take a long time. Who wants to wait an hour to submit a minor PR? Plus it ties up resources on your machine. CI can automatically scale.
5. There's no way to avoid races between testing and integration (i.e. merge queues).
CI isn't going anywhere. I think the biggest scope for improvement to CI is
1. Only testing things that have changed. Most people don't do that because it requires a build system that properly isolates everything. Basically only Bazel and it's derivatives do this. You can't do it with CMake or NPM or Cargo or ...
2. Make it easier to run CI on your own machine. I don't think there's really any technical barrier to this, it's just people don't usually bother.
https://github.com/nektos/act/issues/235
> Trust in the developer is required ... but it is likely the case that developers are incentivized to illustrate that they are reliable, careful and trustworthy -- and that should strongly encourage accurate test result signing
100% guaranteed you're going to see "oh I just made a stupid typo that broke one test but I can see what is wrong so I'll just fix it and use the previous test results".
Also since it's entirely based on trust anyway I'm not sure what additional benefit signing gets you. I don't think there's really a way to prove you ran the tests.
Either way, all my other points still stand. CI isn't going anywhere.
It's a reputation and trust-building exercise, essentially - and that's one of the reasons that continuous integration is particularly useful. "Entities X, Y, Z all say that commit <ID> looks good".
> Either way, all my other points still stand. CI isn't going anywhere.
Agreed :)
I have worked in a place where they did that, and I think the cons heavily outweighed the pros. I can not push incomplete work to a remote, I can not separate feature development and chores (eg. linting) because I _need_ to fix stuff in order to commit an push, etc.
> Continuous Integration is a model thats tightly tied to how we run version control.
I would say that a pre-commit testing world is much tighter. CI, as many know it, as a passive observer. When stuff fails, it will let you know. You can still merge.
One thing that would be nice, however, would be the ability to run the entire pipeline locally. For GitHub actions, it indeed seems like there are viable options to do that.
Though for me it’s also like… so many people do unspeakable horrors to their machine setups that I like there being a consistent runner in CI.
But “CI is to run tests” in the world of CD is a bit of a simplification anyways.
Tho technically you can probably have a local runner which either distributes the test suite across a bunch of available matchines (à la pytest-xdist), or one which goes and creates jobs on the CI platform without needing to go through the pretest ceremony (e.g. creating commits, branches, CRs, ...)
I stand by this idea that if someone made Bazel, but without all the obtuseness, with strong integration to a hosted CI service, they would quickly overtake a lot of services.
I prefer not having to replicate locally how GitHub runs GitHub actions, but rather just make my GitHub actions run something that I know I can run locally. So all the complicated stuff is contained in the script that I can run locally, and GitHub actions is just a dumb “runner” of this script.
For my local script I prefer using Nix, since it’s the best way I know to guarantee a consistent environment between different machines.
Maybe the right thing is to just write the workflows in something else, and have the github workflow file be a single call to your script. But
-It would be nice to be able to use github actions others has made (or libraries/abstractions in general)
- I don't see how to easily get parallell execution.
- I love github environments, how do I pass down all my environments and their variables?
What if the client is compromised? Are we throwing away reviews? Should the reviewer re-run all the tests?
This cost me many hours of waiting for the Gitlab CI runner when debugging non-trivial pipelines, when the issue was something that did not have to do with the script steps inside of the jobs but rather how the Gitlab runner handled things.
I've found gitlab-ci-local [1] which actually does run the Gitlab pipeline locally, although I had to write some boilerplaye scripts to set up all the necessary 'CI_FOO_SOMETHING' environment variables before running the tool. (Which sometimes came back to bite me because the issue was actually in the content of some of those environment variables). It's still a very good tool.
[1] https://github.com/firecow/gitlab-ci-local
Side note: makes me sad when even open-source tools for Gitlab are hosted... on Github...
Per-commit hooks save my time by not allowing to push code which guaranteed to break later in CI.
Also hate those.
What I do instead is have the tests run as part of the build. If the tests fail, the build fails.
Definitely a good motivator for having fast tests, which is essential. The ones for my projects tend to run in about a second or two.
Also gives you way greater confidence in your tests.
Try it!
https://blog.metaobject.com/2020/05/mpwtest-reducing-test-fr...
https://blog.metaobject.com/2020/06/mpwtest-only-tests-frame...
I'm working on something to do just this. Although I've redefined what "local" means in this context. I'm still using remote servers but everything is happening pre-commit from the terminal. If you are interested check out this demo and let me know if you have any feedback https://brisktest.com/demos
This is a twitter tier take with no insight to back it up except “things will continue to get better and soon things will be different!” But doesn’t paint a picture of what different might be.
I mean, yeah things are going to change, duh
Build can be triggered manually in every CI system I know. Why would I push empty commits?
I think there's a whole lot of truth in this particular statement. I've worked with lots of devs who complete a story, commit it, and then find the tests aren't passing any more and have to do a little more work to fix what they broke. And then tery complain that either tests slow them down or that the estimate for the story was too low. Having a test suite that can be run locally, and teaching the team to use it regularly, even if it's wholly optional rather than running on a hook, improves team velocity significantly.
I often fire off a commit, get notified of a test failure, and fix it in the next commit. Why wait for tests to run locally when some other computer can do it for you?
(Yes, it would be nice if all tests ran in zero time, but back in the real world...)
- trunk-based workflow. Small commits. No feature branches as a rule to be occasionally broken
- unit tests move left - run pre-commit (not necessarily run in CI). Emphasis placed on them as a refactoring tool.
- a critical, small suite of e2e and integration tests that block the pipeline before publication (fast feedback)
- latest build publication being constantly redeployed to production, even if no changes have taken place to exercise the deployment routines
- a larger suite of e2e and integration tests being constantly run against production, to give feedback when something isn't quite right, but it's not a disaster (slow, non-blocking feedback).
In summary, emphasise getting code into production, minimise blocking tests to critical ones, test in production & notify when features are broken.
Why?
- Engineers spend too much time in test environments that give the illusion of the real thing. They lose touch with production as the Dev cycle increases in circumference.
- Enabling tighter feedback cycles by accepting that some features are important and some are not helps put the cost of change into perspective for the entire product team.
- Engineers get used to working in and around production on a daily basis. Production operations and observation of systems (& users) are emphasised - protective bubbles for writing code are minimised.
You're not trying to maximise code output, you're trying to maximise the velocity of safe change, and you do that by understanding the environment (production) through your intelligence (observability of systems and user behaviour), so that you can employ your force (changes, code, features) rapidly and effectively, whilst maintaining the ability to quickly deal with unexpected problems (defects) along the way.
Disclaimer: might not be possible for your specific theatre of war for any number of reasons.
However, things like:
Show that the author is perhaps conflating GitHub with Git.That's a problem with your building and testing tools. Even without CI you would still need to build everything and test everything if you have no way to do it just for what changed.
>How often have you or a coworker pushed an empty commit to "re-trigger" the CI
Most CI solutions have a button to trigger a new version without a new commit.
>Running tests pre-commit is one answer
Even with CI developers are likely running at least a subset of the test suite while they are developing.
>Yet, there are roadblocks today that need to be fixed, like tests that don't fit on your laptop
Either develop on a server while using your laptop as just an editor or have a test runner on the server.
>Things are shifting left
The problem with this is that at large companies the changes you've already tested will always be rebased onto a newer version of the codebane which it hasn't been tested with. Who runs tests for this newly rebased version? CI. Also for things like come review you can not push it left onto the devs machine. You will want linters and tests to be run for come review.
I think the author means "the end of build servers". You can do CI fine without build servers.
When I read the title I automatically assumed this would be a "...because CI won and is everywhere now" sort of closer. Junior devs' eyes go wide when you tell them builds used to happen overnight, "feature branches" could last months or even years, and merging was someone's full-time job.
Or at least, if we are, it's a far more robust and capable system that's actually really designed for dealing with all the different things that people need to do while building and testing software.
But surely most IDEs these days make it simple to run all unit tests locally? My usual M.O. is to manually run the tests locally that I believe will demonstrate my changes are "working" and rely on the full suite of CI unit tests to check I haven't broken anything else. What other options are there?
While I develop I start a `watch` process that keeps running the tests. The watch runs a container with the docker image of the CI, mounting only my development working directory.
With this setup there's no added value to the CI
We looked at doing this, but it seemed to be pretty expensive. Like, it was cheaper to upgrade a developers machine twice a year expensive.
Is that what you see or was it just setup wrong on our side?
It’s inherently a batch process, so not continuous.
It’s an automated build and check system. It also runs on merge commits but this “integration” part is really marginal to the concept as represented by e.g. GHA.
I remember the MCSD docs were promoting a "Daily build and smoke test". This was a huge difference in that they promoted at least making sure that thev whole system could be built in some state every day.
CI really appeared in the late 90s when someone had the idea that integration tests could be run for small gains in functionality. Then the software could be automatically tested on every incremental change. I credit C3/XP for popularizing the practice but I'm not a historian. Possibly someone was already at it before.