50 comments

[ 2.9 ms ] story [ 85.2 ms ] thread
Process gates are for when you know you aren't going to get many iterations. Some things really are one shot or few shot. Example: a set of masks for a silicon chip costs millions of dollars, get it wrong and that's a million dollar coaster. Recognise when you have that problem and when you don't.
Process gates are part of repeated process. Otherwise we'd call them "special one-time checks that we absolutely won't make part of our process."

One-shot checks are a thing and have a place, but that's not what the article is about.

What I mean by one shot is one shot in that product. You only tape a given chip once (unless you screwed up and somehow still have the money for a redo), but you absolutely expect that you are going to tape out more than one chip if you are a chip company. So it is genuinely part of a process.
A process gate is a "bolt-on" solution. Rather than modifying the existing process to improve outcomes, you treat each team's activities as a black box and try to improve outcomes by sticking something in-between their activities. Which, naturally, means that it can be a crutch for management whose expertise is management-per-se and whose domain knowledge is shallow.

Process gates create Waterfall. Every process gate is a new step in the waterfall. If one wanted to create a Waterfall methodology, process gates are literally the thing you would use to accomplish that. Which isn't always a bad thing!

Waterfall isn't a bad thing, but lowering agility and flexibility and adding time and frustration to the cycle by the means of process gates is a bad thing.
Readability at Google suffered from this issue both directly (requiring someone with Readability in a language to review and approve before landing a CL) and indirectly (the backlog of requests to obtain Readability was at one point I think over a year long for some languages).

Contrasted with Facebook where you were expected to commit code to production your first week. I couldn't believe how different the eng cultures were.

(Each has their pros and cons, not saying one's perfect and the other terrible.)

Noogler's first code review is a meme for a reason, but even a decade ago the expectation was that you'd have something shipped in the first month or so after the onboarding sessions, even if it wasn't terribly important. Has the eng culture really atrophied that much?
The onboarding went to shit even in my time there bc nobody cared about anything that didn’t feed directly into promo process
I wish I would have read that 5 years ago. I tried to fix poor coding practices with mandatory review steps, and it just slowed down development and didn't fix anything.

For example, we often forgot to backport fixes. So I added a question to the PR template: "Does this PR need to be backported?"

I thought if people saw that question they would stop and think about it, and wouldn't forget backporting bugfixes.

It turns out they just always typed "No".

I just added friction and the problem persisted. When people don't care, you can't fix the problem with process, because they'll just mindlessly follow the process and make the same mistakes that the process was supposed to prevent in the first place.

People won't do any more work than they have to. If you need the PR to be backported, don't ask a question. Say "step 5. Backport this PR" Don't offer an easy out.

Then if they don't you can ding them for not following the process. You can fix problems with process, but all that does is set a minimum floor of quality, it doesn't guarantee anything higher than that.

In your example, I'd have standards for coding practices, and reviews to be sure that they're met, and merges aren't allowed unless the reviewers all approve.

The first thing I write in a coding guideline, "This is a set of suggestions, not hard and fast rules." So in practice it's harder to figure out if the code is good enough from this standpoint.

What is working, I think, is to have everybody agreed about 1) what actually needs to be done - a good understanding of details of the project - and 2) how it needs to be done - on high level, and on the low level some variations shouldn't matter too much.

I prefer a coding standards document that is divided up into hard and fast rules and guidelines and clearly states which is which. Otherwise, reviews can quickly degrade into arguments.
Can you give an example of hard and fast rule regarding coding standards? I'd assume people who learned programming by books - and even those who learned otherwise - wouldn't name variables after rock stars, but something not too obvious?
Sure. In the codebase for an 8-bit embedded system: recursion is forbidden.
Ah :) even tail recursion?
This is a tough problem to solve. One approach I’ve had success with for this kind of “diffuse accountability” problem is to make this a rotating role that one person does. Eg if you have a release per week, have a BuildMaster hat and part of the job is figuring out which things to backport. The hat rotates so people take turns thinking about the thing a lot, instead of everyone having to spend a couple cycles on it. If someone is wearing the hat and messes up then that is a more focused performance problem you can pinpoint and remedy.

Of course this doesn’t always work for every problem, just an idea that may work for some instances of this kind.

Agree whole-heartedly! Not mentioned in the article:

- Increasing cycle time means you'll end up jamming a larger delivery payload (or more items) into the cycle. This increases risk, which is mitigated via gate-keeping, so cycle times tend to increase even more.

- Gate-keeping is a form of fiefdom-building. Businesses usually don't consider that when implementing them at first.

- Gate-keeping for problem FOO means everyone will treat FOO as someone else's problem; they're free to ignore foo. Quality gates, in my experience, typically make for lower quality products.

Very good read.

Process gates are one of the favorite tools of "successful" managers.

Imagine coming up with action items during post-mortem. Drastic measures are easy to understand and have good optics. Narrow gates look like half-measures.

As a manager, you should be proud of process steps you eliminated, not steps you added.

I swear most of these are quoted elsewhere as good practices and used very well in certain scenarios. I agree with the article to the extent that blindly adding process has costs and if you don't reap the benefits, it's a net negative.

When all of these benefit massively is when there's large information/knowledge asymmetry, but the right questions and review usually doesn't get to the right person. E.g. junior devs architecting things poorly and then wasting time to implement a bad idea. Devs pretending to be designers and coding a UI that's just bad without consulting the designer when they should have.

Does just telling them "you devs to consult the designer more" work? I haven't seen it work. With the right steps to review design, architecture and so you could easily save more time later than you invest upfront.

> Does just telling them "you devs to consult the designer more" work? I haven't seen it work.

Not immediately, but it does. The problem with process gates is that they don't fix the problem, they just hide it. Even worse, they treat problem as unfixable by nature.

If your juniors are architecting, something is wrong with your structure. Juniors should typically execute somebody else's designs.

If your devs don't have a good sense of UX, they should not have the power to make the decision to ship the product where UX is crucial. OTOH if they ship internal dashboards for one user (or for themselves), they don't need a designer approval.

Take a look at this Toyota process map: https://creately.com/diagram/example/hoz4qrcl1/toyota-proces...

There are checks everywhere. Pre-checks at the beginning of the supply process, checks that the supplies are valid, checks during manufacturing, checks during assembly, checks post-production. The value chain must continue to flow freely and continuously, and any bottleneck must be found and dealt with immediately, anywhere in the flow.

During the entire lifecycle of producing the car, there are gates for both quality and waste. You can't get quality if you have waste, because waste leads to defects, which you have to fix, which takes more time and money, and exponentially destroys your productivity & product.

Process gates are necessary to eliminate waste and increase quality. You have to immediately fix the things the gates catch, and continuously improve the process of fixing them, until identifying and fixing problems becomes fast and natural.

So process gates are a good thing - IF you actually follow Lean, and are actively working to improve your process. Most people know jack shit about Lean because the company didn't give a shit about training their employees, so their employees suck, and don't improve the process. Toyota, however, does train their employees, and so the employees follow this process, and the result are the most reliable cars at competitive prices.

If you are resolved to just have shitty people and not train them, then sure you can remove process gates. And you will churn out cars faster. But those cars will be shitty, just like American cars from the 50s-90s. You'll need to spend a lot more to fix all the defects, and customers will be unhappy. If that's fine with you, cool! You're pushing out cars! Progress! But let's not pretend process gates are evil just because we don't want to learn to use them properly.

"Checks" and process gates aren't at all synonymous, though. A process that contains appropriate checks isn't what the author is talking about; I think that's clear enough from the examples, but maybe it's not.
Many of those checks are gates. They will stop the production line when they find a defect, because they know it's a defect that can reoccur, and won't let a car go out with defects. They fix issues on the spot, and immediately incorporate the fixes into the production process.

Of course besides gates (Andon cord, kanban) they have other processes: autonomation, poka-yoke, production leveling, kaizen, 6S, etc. But gates keep production in check, and must be efficient, and part of a continuous improvement cycle.

Yep, Toyota has a pretty amazing manufacturing processes, and as a result their cars are some of the most reliable.

If you don't have process gates, your end result probably sells more due to marketing and hype (looking at you, Ford and GM muscle cars and pickups).

One of the deadly sins of management is treating software engineers as factory workers.

Toyota ships same cars over and over. You can't copy the output of a factory worker, but you can copy the output of a software engineer.

Even if you ship CRUDs all day long, it's still very different from a factory worker.

And one of the deadly sins of software development is pretending that we can't learn new things from other industries.
For this specific context, disagree.
OK. I'd say that it depends on what is meant by "treating software engineers as factory workers."

Elsewhere in the thread I reference Team Software Process (TSP) which is a development methodology that uses feedback to continuously improve the development process in a way that's very similar to manufacturing methods. I've done it; it works.

This is a strawman argument, nobody here advocates against feedback.

And using feedback to improve process is not specific to any industry. You don't need to learn from Toyota to understand that feedback is valuable.

> Toyota ships same cars over and over.

Well, no, they don't, and this is the point: they are constantly refining every car model. They are always fixing bugs in the active production line, working on new features, and every year ship new "major and minor versions" of each model.

Toyota USA ships these models: Corolla, Camry, Avalon, Mirai, GR86, Supra, Sienna, Tacoma, Tundra, Sequoia, Highlander, Venza, C-HR, RAV4, Corolla Cross, 4Runner, bZ4X. For each of those there are 2-6 different trims. Assuming an average of 4 trims, that is 2,380 different configurations of vehicles. Each trim can also have optional components added or removed to customize each vehicle, increasing combinations further. A US Toyota factory worker may support tens of thousands of ways to assemble a car, and those ways change every day.

> Even if you ship CRUDs all day long, it's still very different from a factory worker.

Sure, but even a factory worker is different from a factory worker. The demands are different, the customers, compliance, tolerances. You can't treat them all the same. But some basic principles of managing companies, people, work, etc, can be applied to all of them. Waste is bad. Quality is good. Stopping to fix something once is better than stopping to fix it 20 times.

The impact of a factory issue is often much greater, more expensive, and harder to fix, than that of a software issue. But software workers can learn a lot about quality and efficiency from how well-run factories are managed. That idea gave birth to Lean Software Development (a subset of Agile).

>Well, no, they don't, and this is the point: they are constantly refining every car model.

Do these refinements face process gates though? Because that's a better mapping to software.

And perhaps they do---but I highly doubt it's as many as the cars themselves face.

> no, they don't, and this is the point: they are constantly refining every car model

Do you really believe what I meant is that Toyota ships one model, "The Toyota"?

> They are always fixing bugs in the active production line, working on new features

And people who do that are typically not described as "factory workers".

> Sure, but even a factory worker is different from a factory worker

They are the same in a sense that you can usually add more workers when you need to ship faster. Fred Brooks wrote a book explaining why the same mindset doesn't work in software.

Software programs are not physical goods nor do they always work the same, by letting workers apply the same algorithms.

Programmers are not workers sitting on the belt and assembling the same software over and over, they are creative persons who have to take lots of decisions.

If you find a problem at a process gate (quality gate), then you need to fix the process, as well as the product (Architecture plan/Code in PR/etc). Why did the gate catch a problem? Was it lack of knowledge, failure to follow standards etc?

Management has learned too much and not enough from Lean Manufacturing. I worked in a manufacturing plant. I know why some of these rules exist. We aren't using them correctly in software development.

Quality is about improving process, not just inspection. Inspection is required, but it is treated as an input to process improvement. Are your PRs an input to process improvement, or an excuse to yell at junior devs?

Management techniques in Lean Manufacturing are all about producing consistency and efficiency in human effort. Software Development is not really like this. When you understand an IT process well enough, you automate it, and the software product that performs such automation should be very consistent and efficient.

We want that effort to create automation products to be more efficient and consistent. But that effort is not at all like manufacturing. It is not repeatable. If it was repeatable, someone would have written a tool for it already. And thus, if we are doing our jobs, we only have hard problems.

Managing software developers is more like managing other managers than it is like managing workers on an assembly line.

If you are an executive who oversees managers who are making changes to processes, you need to provide contingent guidance and goals, not dictates. Software developers are making changes to processes.

> If you find a problem at a process gate (quality gate), then you need to fix the process

This is essentially what Team Software Process aims to do. It applies a feedback model to the development process to continuously improve predictability and quality. However, it's time consuming and for it to work, you have to capture a lot of process artifacts like time spent on a task, defects found at every stage of development on that task. They basically try to apply something similar to statistical process control to software. It works (quality definitely improves), but it can be a PITA.

Process gates do slow down progress. But (a wisely chosen and designed) process gate can also prevent issues later that do even more damage than the slow progress does.
How does one not process-gate something like PR reviews?

Example: several times we've had mid-level devs do work in code that could have been handled by a language function, or built out a portion of the system without thinking about how it impacts another portion, or doing work without writing tests... all things that we've attempted to stress to everyone on the team repeatedly. Other mid-level team members who are already performing reviews aren't calling out these things, so the only thing that seems to work is adding a process-gate where some of the more senior team members are required to approve.

I hate it, but nothing else has seemed to work.

At that level, in the immediate, I don't think anything will work as well PR reviews.

However, the long-term, sustainable solution is complexity reduction.

A/B testing shouldn't be just for customers.

We could have gates that have a random bypass. Have a 10% chance that the gate must be bypassed for this iteration.

Did any of the bad things that happened cost more that the gate? No? Gate goes away.

This would miss lower probability but high cost events.
Gates are non-optimal for good workers where the gates are redundant or pedantic, but absolutely essential to leashing poorly behaving people that will literally ride your company off a cliff. If you don't have the later, then process sucks. If you have a bunch of the later, you're stuck. Not everyone can attract effective talent, so the solution is to use common sense to not send your company into ruin.
> If this was a TV show, I’d have scary music play every time you read the word "process gates"

You are entering a realm which is unusual, maybe it's magic or contains some kind of monster... These are just examples, it could also be something much better. Prepare to enter: the process gate.

we solved this by creating a review board to look at proposed new process gates and approve them before they are put into use.
Consulting for a bank some years ago, I found a gate that routinely cost three weeks in time, and no-one could think of a single instance of it resulting in any feedback that made any material impact whatsoever. To their credit it was removed right away, but it’s funny that no-one thought to do it previously.
A fun one from games are process gates between different ‘stages’ of development.

Nominally it makes a lot of sense to make a vertical slice of a game, it means you’ve demonstrated all the core mechanics to a polished state in a production quality part of the game. All you need to do now is go wide in content production and continue to polish the game mechanics.

In reality no one is interested in whether the game is actually ready just that it’s now the arbitrary time to review. Teams end up building the vertical slice to order so they pass the gate review. It’s full of half cobbled together systems and built with half-finished art pipelines. It passes and drags all this crap into production or it fails and you have a demoralised team. Either way you’ve just wasted months of work to fulfil some arbitrary criteria. It goes hand in hand with the seagull school of management. Fly in, shit all over everything and fly away.

It’s more macro than the articles examples so less a death from a thousand cuts but it’s easily one of the most detrimental things I’ve seen cargo culted across game teams.

> What should you do instead of process gates?

The first idea the article suggests is... use a process gate.

Yes, a narrower or short-lived one, but it's quite a mixed message.

I also don't think the death spiral described is inherent to process gates.

If you're going to claim quality gates generally cause worse quality I think you better be prepared to back that up. I don't see it.

The most important process is team-building, broadly construed. Process gates are a smell that indicate the team composition is, or at least was at the time the gate was put up, unsuitable for the task at hand.
I don't have the time or patience to go through hundreds of lines of code from PRs so my code reviews are usually fast and consist in "looks good to me. My colleagues are well versed programmers so I usually assume that there aren't severe mistakes.

Bugs might slip through even with the most severe code reviews and that's why unit tests, integration tests and end to end tests are for. :)

> For example, if you add a QA step before work goes to production, you’re adding a whole set of work that needs to go back and forth before things can be released.

The reason that stuff goes back and forth between engineering and QA is that it didn't work. When choosing between "releases take a little longer", and "releases have bugs", taking the extra time is a no-brainer. Engineers don't like their code getting rejected, so it also teaches them to spend a little extra time not shipping obviously buggy code. Everybody wins.

One thing that is hinted by the article but I think isn't fleshed out well enough, is that "process gates" per se aren't bad. It's the mentality that a process gate is an "add and wait" thing.

Like any intervention that is a "change in process", they should have clear dates and/or criteria under which such a gate either straightforwardly expires, or is at least expected to undergo review to confirm it is still needed, works as intended, and isn't causing more trouble than it is solving.

This isn't just a "project manager" thing. It's a very common mentality found in most professions. Doctors, for instance, are notorious for being very eager to "start" drugs, but very reluctant to "stop" them; especially when "someone else" started it. This leads to an endless cycle of people ending up on bucketfuls of pills every day, many of which interact with each other in adverse ways, and some of which are only there to counter side-effects from drugs that could have simply been stopped a long time ago.