HN Explorer

Ask HN: How can we make the current CI/CD ecosystem better?

10 points by dosisod ↗ HN
Having dealt with lots of CI/CD platforms (GitHub Actions, Gitlab CI, Azure DevOps, etc.), I feel like there is lots of room for improvement:

* There is little to no inter-op between CI providers. Sure there are 3rd party providers, but they only seem to focus on the basics, like running a job when a commit is pushed. But, if you choose something like GitHub Actions, you are now locked into their platform, making switching Git hosting providers very painful.

* YAML. Everything is YAML. Sometimes it is just one big file (Gitlab, Travis, CircleCI, etc), and it becomes very hard to maintain very quickly. Workflows for GitHub Actions can be split up into different files, but at the end of the day, you are still writing YAML, which means no loops, conditionals, functions, and so forth. In addition, you must follow a strict file structure, meaning very little room for customization. In many cases, you need to write a Makefile/Python/shell script for more complex situations.

Does anyone else share these annoyances? Are there any similar sorts of issues that you deal with on a day to day basis that you wish could be improved?

22 comments

[ 12.8 ms ] story [ 992 ms ] thread
TillE
The real meat of what CI/CD does is closely tied to the platform, I don't think there's a great way to improve that. Running on different configurations, uploading artifacts, deployment, etc.

Generally, I do as much as possible in the build system proper. The CI should mostly be running simple commands. Avoid GitHub Actions which can be trivially replaced with one or two shell commands (which is a lot of them).

chillfox
It should be more achievable than it has ever been before with container tech being in a great place, qemu being a thing and the webassembly ecosystem getting better every day.
chillfox
Absolutely, I feel like most CI systems are much harder to configure than necessary.

If I were to build my own CI system, then it would have these features.

- Simple config format, ideally flat and not nested, so maybe toml.

- Ability to run any step, task, job, pipeline locally. It should be possible to do something like `ci run -t build` and have it execute the build task locally. This would greatly help with building CI pipelines and make it possible to just use the CI tool as the local build tool as well instead of having to essentially setup the same thing twice.

- Easy to self host, so not a million micro-services, but just one (or max 3) executable.

- Modular, it should be possible to extend it with plugins using webassembly or containers. Look at how Concourse CI does this for ideas.

Edit: I would pay for a license to something like that, maybe not a lot but I could justify up to $200 as a once off for a self-hosted solution like that if it's well made.

dosisod
I think these are all great points! I have came across Concourse CI, but have not looked too deeply into it yet.

I hadn't thought to use TOML for configuring CI pipelines, I'm curious what that might look like. TOML is indeed very flat, so it would be interesting to see what the equivalent TOML pipeline looks like compared to a YAML one.

chillfox
I run Concourse CI for my personal stuff with the code in Fossil. It works decently enough, I like that it's somewhat extensible, but I have hit use cases that it can't really do like building an Alpine Linux cloud image.

As for YAML vs TOML, I am mostly sick of YAML, and I really like using TOML to configure Caddy, so I thought it might be good enough for CI config. I feel like it would be worth exploring, but maybe something slightly more complex or custom would be required.

xwowsersx
This doesn't have everything you mention, but I recently learned about https://dagger.io/ which allows you to write your pipelines as code (Go, Python, etc) and then run them anywhere in containers. I always hated that in order to test my pipelines, I actually have to trigger it on some special machine (the CI server).
chillfox
Dagger looks interesting, I will check it out.
P5fRxh5kUvp2th
In addition, it would need to try and solve for the 80% witch hatches for those that need the last 20%.

Part of the problem with current CI/CD solutions is trying to solve for everything.

And I think the local thing would be a killer if for no other reason than the ability to debug it locally.

chillfox
That's kinda my idea with plugins, it should be possible to develop a plugin doing your custom thing if you need something in that last 20%.

Concourse CI has a design documented for a better resource/plugin system than what they currently have implemented that's worth checking out. But as far as I am aware they are really far from actually implementing it.

mike_hearn
Sounds a bit like TeamCity, except for the local execution. It has:

- Either GUI based configuration (serialized to XML), or Kotlin. So you can express configuration with a real programming language tracked with version control, or just use the GUI to define it.

- Can self host. There's one program to install for the master, and one for the agents.

- Can be extended with plugins. Not webassembly based, but those are theoretical anyway currently due to lack of better-than-C interfaces. You can write JVM plugins instead and get full GCd OOP.

The main thing it's lacking is the ability to run things locally. But then again, usually a TC step is just invoking a build system or script anyway so it's not that big of a deal.

raptorraver
I've been thinking lately that how I could do some integration testing FOR my CI-pipelines. My client has very complex pipelines which has business logic inside them and every week we run into new bugs and regressions. I can't think any easy way to write tests for them to make it harder to break when developing new features.

Btw. Gitlab allows splitting your pipelines to multiple files using include[1]. Also we have used gitlab-ci-local[2] to run our pipelines locally when developing them.

1: https://docs.gitlab.com/ee/ci/yaml/includes.html 2: https://github.com/firecow/gitlab-ci-local

dosisod
I havn't heard of any products or services that test the actual CI pipelines themselves, which seems like an interesting target market. I can't say how many times I have wanted to test out some CI workflow on some throw-away clone repo without mucking up the live repo.

It's good to know that you can use includes with Gitlab, And, from my quick glance at that second link, the setup process seems a bit involved. I don't like having to set up new user accounts just so that I can run a CI workflow locally.

raptorraver
I've been able to use the tool without setting up any accounts or other settings. Just run `gitlab-ci-local` in a folder where your `.gitlab-ci.yaml` lies.
rpep
I’m not sure the niggles here are platform things - they’re hard lessons that come from experience in using CI for the most part.

Generally you want to aim for writing things in scripts that your CI system calls, so that they can be run locally trivially and so that migrating away from one CI provider (for whatever reason - cost, employer decides to shift, etc.) is not a huge piece of work.

chillfox
So if we have to use a script anyway, then why do we have to write like 20+ lines of YAML to run the script?

Edit: I mean, why is this not the config if we are just doing a script?

  src: github.com/...
  run: test_and_build.sh
rpep
It's not that bad:

  jobs:
    unit-tests:
      runs-on: python:3.8.15-slim-buster
      - name: "Run tests"
          working-directory: ./bin
          run: unit-tests.sh
kanwisher
Make a bash file. Have the platform specific yaml call your bash file. call it a day
MauroIksem
I'm pretty happy with GitHub actions
rurban
With GH actions you got everything. Conditionals, proper expressions, and goto refs, which allows loops.
dosisod
I cannot seem to find any references to "goto refs" in GitHub Actions, where is this documented? I've never seen this before so I am curious