HN Explorer

Ask HN: Team spends all our time fixing CI

4 points by throwaway271723 ↗ HN
I work at a large tech company, I’ve been on the team a few years. We’re building a Kubernetes based service and spend an incredible amount of time on CI related issues. One team member in particular is constantly making tweaks to the pipeline, pushing for improvements and talking about it constantly. The build is broken most of the time. I feel pretty dejected as we don’t seem to deliver much user value. As it’s a big company we just seem to fly under the radar. Anyone else have similar experiences?

11 comments

[ 3.6 ms ] story [ 37.4 ms ] thread
diggs
I am yet to work on a team with a simple, reliable and repeatable build process.

Most build processes (from basic scripts or Makefiles all the way up to CI/CD product-specific pipeline configurations) are complex, brittle, opaque and poorly understood by most of the team. I've seen time and time again one or two people emerge as the "build" people because most team members want to run as far away as they can, and it's easier to let someone else gatekeep the mysterious black box that sometimes does something useful. These processes typically grow organically on an as-needed basis and are rarely designed under a unifying vision.

I think it's a combination of lack of experience on the engineers part, bad off the shelf tooling, lack of recognition of the cost and as a result under investment by the business.

thedevindevops
I take it you can't use gated checkins to prevent someone from checking something in that'll break the build?
throwaway271723
We do but many of the build problems are flaky in nature, eg timeouts
valdask
Meaning you have other issues which have nothing to do with CI. Fix your tests etc.

None of the tests can be 'flaky'. Either remove them, or fix them.

throwaway271723
It’s not that simple, the flakiness is from the internal services we’ve been mandated to use as part of the build process
jimmyvalmer
CI pipelines are a Good Thing. While we didn't get bogged down by them ten years ago, our testing wasn't nearly as formalized nor robust.

If you're using any one of the popular suites (github, travis, circle), you should be able to fork your own little CI fiefdom insulated from the depredations of the incompetent.

One team member in particular is talking about it constantly.

You have to understand only a small fraction of the tech workforce makes a living off code. The rest of us have to do it by talking.

hitsurume
I'm curious to know why you're getting dejected? Is it because you can't get work done and your manager is hounding you, even though its not your fault the builds aren't working? Are you part of DevOps/CI team and are getting blamed for the builds not working? Are you a manager who doesn't know how to deal with a employee that's gone rogue? Do you care too much about productivity as a individual contributor?
throwaway271723
> Do you care too much about productivity as a individual contributor?

This, I hate that we spend so much time on build related stuff. The rogue employee has backup from one of the managers and has spent the last 3 months “upgrading” our pipelines to a new platform (his preferred tool). Things were quite stable before this change now we can’t get much done due to fighting the fall out from the changes

faangiq
Uhhh try fixing your build? Learn to write proper tests.
jimmyvalmer
Rants of twenty-somethings about office politics are fine. Just don't disguise them (weakly) as questions for professional advice.

Then again, nearly all answers on Ask HN are rebuffed by the the poser, so Ask HN is really more akin to craigslist's "rants and raves" or reddit's /r/amianasshole.