Ask HN: How Long Is Your CI Process?

That seems reasonable to me if you require heavy integration test coverage. I work on several applications. The ones that are message driven and don't require a database have test suites that run in a couple minutes. The one that has a large database component takes about 30 minutes to run the tests. This is because we actually run the tests against a real database which requires migrations, data loading, etc.

Depends on the complexity of what you are doing. For web stuff with unit tests I’ve seen it run in a few minutes.

Our current CI takes an hour because it has to build quite a complex app on iOS and Android, this happens in parallel but the Azure build nodes we use are pretty slow. Ideally it would be faster but it’s not too huge an issue in practice, we have the lint/unit tests etc. run first so the build will fail early for any glaring errors.

CI generally is a topic where there is usually lots of potential for optimization, but many things are not easily done with common tooling. Some large companies put serious R&D effort into improving CI with custom tooling.

What is applicable to the specific project depends, same as to what is worth which effort. To a degree, of course throwing more resources at the problem helps - faster build workers, parallelized tests, ... but isn't always easily implemented on a chosen platform and costs money of course.

In projects I worked on, it varied greatly. From just a few minutes to cases where the full process took 6 hours (which then was only done as a nightly job, and individual merge requests only ran a subset of steps). I really would want <15 mins as the normal case, but it's often difficult to get the ability to do so.

You get what you pay for? Cloud CI bottom-feeds unused downmarket capacity like a magazine-stand calling card. And Cloud CI is still "Cloud", i.e., a mass-market, one-size-fits-all solution like the Department of Motor Vehicles.

I don't see how anyone can give you useful information without knowing more about the pipeline and the projects, and as everyone's pipelines/projects are going to work differently (I do web dev work, so pipelines are relatively simple, I can imagine that a game dev team creating Windows/Mac/Linux builds might have multi-hour pipelines though).

Anyway as the question is "How Long Is Your CI Process", here we go!

I have two main types of pipelines, both running on a self-hosted GitLab instance which runs on an 8th-gen i3 Intel NUC. No project is particularly massive.

1. PHP Projects. Run PHPStan + unit tests on each branch. Most projects take 1-5 mins. On master, run PHPStan + unit tests, build a Docker image, and use Helm to deploy to managed Kubernetes on DigitalOcean. This takes 5-10 mins.

2. React Projects, again not massively huge, but sizable. Biggest time is to run ESLint on every branch. About 5 mins (due to very poor caching which I keep meaning to fix). On master, run ESLint, create a Docker image, and deploy to managed Kubernetes. 5-10 mins.

There are opportunities to improve this by fixing/optimising caching. Overall I'm reasonably happy with the pipeline performance. I'm also sure that upgrading the hardware would make a big difference, probably more so than fixing the caching; an i3 isn't really ideal but this machine does well overall for my small team.

2 minutes and 7 seconds https://github.com/jart/cosmopolitan/runs/2482398460

on travis for a repository that builds 14,479 objects, 67 libraries, and 456 static executables, 284 of which are test executables which are run too. If I want to run all the test binaries on freebsd openbsd netbsd rhel7 rhel5 xnu win7 win10 too, then it takes 15 additional seconds. On a real PC, building and testing everything from scratch takes 34 seconds instead of two minutes.

We use CircleCI to deploy PHP apps. It takes about 1-2 minutes to fetch and build the JavaScript and SCSS assets, and 1 minute to install dependencies from Composer. And about 1 minute to rsync the build artifact to the server.

We have a CI pipeline for a cross-platform Rust library, and it currently takes an hour across C, Android, iOS, Java, WASM, etc. and different combinations of cryptographic libraries. This is probably something we’ll tune over this or next quarter, such as by throwing some beefy hardware at it and parallelizing. We also seem to be hitting some GitHub actions limits in terms of storage.

https://github.com/spruceid/didkit/runs/2468746631

Probably your CI could be faster. Just take a look at where the time is going and see if you can speed it up. Here’s some tips that might be useful: https://charity.wtf/2020/12/31/why-are-my-tests-so-slow-a-li...

We need more CI processes using https://bazel.build

Hard to say without knowing /what/ you want to accomplish in your CI process, so maybe some open source examples will help:

* A "complex" library (node-resque). In CI (CircleCI) we install deps, compile Typescript to JS, test on 3 versions of node, and build docs. 4 min w/ some parallelization https://app.circleci.com/pipelines/github/actionhero/node-re...

* A web server framework (actionhero): In CI(Github Actions) we install deps, compile Typescript to JS, test on 3 versions of node, and build docs. 7 min w/ some parallelization https://github.com/actionhero/actionhero/actions/runs/801273...

* A Monorepo (Grouparoo): In CI (CircleCI) we install deps, compile Typescript to JS, run migrations, check licenses, test UIs, CLI tools, Plugins, and try out a few different databases. 5 minutes with rather extreme parallelization https://app.circleci.com/pipelines/github/grouparoo/grouparo...

In my experience, the biggest wins in CI speed improvements come from parallelization. You can parallelize by either running multiple processes/containers or by running tests in parallel on the same container (jest, parallel_tests, etc)

I like how many people responded but didn’t answer your question.

For the kinds of projects you mention (scripting language, small-medium sized) I aim for 1-2 minutes max, which is usually not a problem. This precludes running a lot of integration tests requiring expensive setup/teardown, though the need or value of those greatly depends on the project.

Between 30 seconds and about 45 minutes.

The only times when it was long enough that it was painful it was because there was a stage that couldn't be debugged without running the build. That's invariably what I actually preferred to fix, not the total lead time.

A 45 minute sanity check to verify nothing is fucked before releasing is fine. A 45 minute debugging feedback loop is a nightmare.

Faster CI builds are typically a nice-to-have rather than a critical improvement (& doing too many nice to haves has killed many a project).

At my current job, the full build/test/release cycle is about 45 minutes. There is an effort to begin optimizing it but it is a high risk endeavor that only became worth it once the costs started growing faster than our team size and became unsustainable.

CI tech debt is very difficult to pay down, and imho not worth it unless the dollar costs are becoming excessive and you have a dedicated release or DevOps engineer who can own it as an internal product.

Several hours, which is far too long. It's a massive waste of developer time and money, but try explaining to diverse contributors ("features features features!") that some investment in the testing setup would save money in the long run...

We have a CI pipeline for a containerized python app and a react app. We have a monorepo and only trigger certain jobs depending on code changes. Our CI runs through Gitlab CI on a GKE cluster, which gives us a lot of control over the parallelism and the resources allocated.

Our pipeline typically takes 10-30 minutes, depending on what jobs run and where cache gets used.

The longest job, at a consistent 12 minutes is our backend test job. There’s not a lot we can do to speed this up any further because a lot of the tests run agains a test db so we can’t easily run them in parallel. Perhaps if we wanted to be really clever we could use multiple test dbs.

The build for our containers is usually very quick (a few minutes) unless we modify our package requirements.txt. That happens infrequently but it triggers an install step that will increase the overall time for the job to 10-12 minutes.

The deploy phase is very quick.

We spent a bit of time optimizing this and it came down mostly to:

1. Using cache where we can.

2. Ensuring we had enough resources allocated so that jobs were not waiting or getting slowed down by lack of available cpu.

2. Making sure that each command we run is executing optimally for performance. Some commands have flags that can speed things up, or there alternate utilities that do the same thing faster. One example of the latter is that we were using pytype as our type checker, but it often took about 15 minutes to run. We swapped it out for pyright, which takes under 5.

30 minutes? I don't have exact numbers, but I know ours is under 10 minutes and IMO that is not optimized at all. Maybe 30 minutes is okay for a very large project, but for most applications that seems quite high to me.

Our pipeline runs in Jenkins and builds a docker image that runs composer installs, application copy, and that sort of thing. We also run phpunit, phpstan, phpmd, and phpcs in our pipeline. Finally the image gets pushed up to ECR.

I think that's all pretty standard stuff. TBH I'd like us to move to github actions and optimize for more staged builds in our docker images, but we have higher priorities at the moment.

This is such an open question it is hard to answer. You have to know what runs in the CI as well as size of the project, languages, number of projects, quality steps executed in build, etc. Anyway, to give it a shot:

* multi-million Loc

* number of projects > 50

* languages C#, C, C++, typescript

* Frameworks: .NET Framework, .NET core, .NET standard, Angular, React

* Quality tools in build: TICS, Coverity, Roslyn, custom tools (>10)

* Tests running in build: nunit, msvstestv2, jest, karma

* number of tests running in build > 5000

* package managers used: Nuget, npm

* number of packages (private and public) > 500

Still a lot I forgot now.

It all runs in approximately 45 mins for stage1 builds, stage2-4 run nightly and weekly and take much longer (>2 hours to >24 hours for long duration stage 4). Increasing stages run longer test suites, up to approx 50k or so for stage 3 and 4, more quality checks, etc.

P.s. We spend countless hours reducing our build times. In addition we have setups to split build pipelines for those who do not need the entire archive build for their dev purposes etc. Yet, CI server aways runs single-core and cold builds.

For Flask apps, a little over 2 minutes to git push code and then see passing tests in CI using GitHub Actions.

Most of that time is spent building the Docker image.

The CI pipeline does:

- Build Docker images for the project

- Run the project

- Run Shellcheck on any shell scripts

- Run flake8 to lint the code base

- Run black in check mode to ensure proper formatting

- Reset and initialize the DB

- Run test suite

That's a baseline. At this point any increase to the ~2 min is a result of running more tests but it's usually possible to run about 100 assorted tests in ~10 seconds (testing models, views, etc.).

An example of the above is here: https://github.com/nickjj/docker-flask-example

A similar pipeline with comparable tools for Rails takes ~4-5 minutes and Phoenix takes ~4-5 minutes too. You can replace "flask" with "rails" and "phoenix" in the above URL to see those example apps too, complete with GH Action logs and CI scripts. These mainly take longer due to the build process for installing package dependencies, plus Phoenix has a compile phase too.

Usually it depends on what you need to run. Running some tests on an interpreted language should be done in a few minutes. With a compiled language it takes longer, maybe half an hour. If you have a compiled language that's slow to compile with additional checks, because the language is more footgun than anything (yes, C++) then you might want a standard build, build with various sanitizers, and do some static analysis, and you end up with hours of time spent on building and analysis.

It really depends on what you're trying to achieve and how big the project is.

It’s really down to goals... you should have something that triggers based on open pull requests and does a sanity check and ideally deployed to a test environment... that should be “quick” to give feedback in addition to reviews...

Then in the main build that hopefully is deploying to a qa environment that can do more testing, bundle artifacts for whatever dependencies need them, all that kind of stuff...

That’s how ours is set up... we use Jenkins with parallel parts where possible (like build the ui while tests that hit the db are run) it’s a process that takes time to get right and time to optimize...

We’re at about 5 mins for the quick part and 8 or so for the slower part

Both of those will probably get worse as we are planning to include full ui testing on the deployed environment too

Yes, they probably could be optimized. Backend API tests are usually fast, or can be made so. Webdriver based tests are annoying to write and usually slow, so I don't test frontend code automatically.

Kinda feels like a waste of time, especially if your code is well componentized and there are not many central points of failure. (and those are pretty easy to see with cursory manual testing)

Oh yeah; this was the best part of moving to Erlang/OTP... Test suites are absurdly fast (nearly) all the time. Most test suites I use take less than 10 seconds with anywhere from 100-1000 unit tests. The worst I've seen "monoliths" take at most ~90 seconds to run and that is only ever the case if folks are creating insanely too many objects or needlessly testing the internals of `gen_server`.

Automated tests that my team runs vary from ~seconds to multiple days, depending on what's being tested. Some of the tests involve compiling a multi-billion line repo using over 30+ languages, and doing some analysis on the resulting code graph. So that takes awhile.

30-45 minutes just for a simple test suite, even if it's PHP, Python, and Ruby - that sounds long. But without any details on exactly what's being tested, it's hard to say.