Beyond the IT department everyone else has very to low incentives to adopt "those crazy ideas from computer guys", they care about deadlines, roadmaps, when marketing campaigns can be organised, with what content, what should be on the contracts with business partners,...
Ever since we got those ideas around 2000, the best outcome if agile is adopted at all, is scrumfall as the article describes.
However I have also seen many companies go back to classical requirements documents, and all related processes, as if agile had never been in the building.
The best results I have ever seen have been projects that people on the bottom did themselves, because they needed it without the people on the top ever knowing about it.
The worst results I have ever seen have been projects where the people on the bottom stopped caring because the people on the top tried to make them ignore the laws of nature.
I'm not really sure what you are arguing. Of course if its a one man job (or a couple) and the .. product or whatever isn't system/safety critical even having a PM is probably overkill and the overhead of "any" process likely just wrecks the output.
But if you need to organize 50+ engineers and actually produce a ... product? That people/agencies/customers expect things of you sure as shit is going to need a definable goal, verifiable output and a somewhat chronological way of achieveing that. Waterfall it is (with reasonable iterations).
It's not just about "admitting they don't know." It's about giving up the control that they've used to define their entire professional lives as "managers." Thank you Frederick Winslow Taylor for your outdated ideas.
No one says Agile can't have full blown requirements or the respective documents.
To the contrary, you can't expect your dev to understand what he has to write without giving him a goal he is working on - Agile won't fix that.
Agile just means you build in small steps from what you have and review your steps ASAP, instead of working multiple months based on some detailed documents, derived from other documents, written by some other department months ago, building stuff no one wanted in the first place.
It seems the majority of comments miss what you are saying. We are an Agile Shop. We design things. We have to at scale. We review schema designs, query patterns, error handling strategies, monitoring, etc. Without that review, you get people doing silly things because most devs have less than 5 years experience. Deliver a minimal viable product and use customer feedback to iterate. The goal is to provide customer value as quickly and regularly as possible. This can include all the Excellence in Engineering standards you want.
People tend to go too far in one direction and then later over-correct in the opposite direction. Waterfall model is too much, scrum is too much. The optimal approach is almost always in-between and yet, frustratingly, nobody advocates for balance and nuance.
Software development requires some planing and also some flexibility. One decade, companies think software architecture is the most important technical role in the universe, the next decade, the role barely even exists... What we need more of is common sense.
Honestly I've never seen a waterfall model implemented without iterations. And you can always update the top-spec at later date if needed its just more difficult.. Which it should be.
Yeah - I mean, when I wrote the comment I ignored all the absolute disaster-projects I've seen. But those where doomed on so many levels I don't even wanna blame waterfall or agile or the in-betweens.
Hah yeah i guess you can interpret it like that, but i give the same leeway to agile when they skip the retros and the planning before the sprints etc. You cant blame the process if you dont even make an attempt of adhering to it :)
We can't be sure there wouldn't have been any iterations. In (what's traditionally been called) the waterfall model, there's so long between iterations that the project has time to fail and be scrapped before there ever is one.
I have seen exactly that - no iterations. Mostly it was customer waiting for the finished product and that was the first time they've seen it.
As an improvement, sometimes the project management *forced* the customer to make some checkpoints along the way, but they were a far cry from real iterations.
Everyone is vulnerable to it, but software developers are particularly prone to thinking that one true algorithmic "recipe" can be designed that leads to success if only it could be faithfully executed in the organisational CPU.
I'd just add that scrum and waterfall are both more useful in explaining to management and junior engineers approximately how the team/project works, than in actually running the team/project.
Some people do, but they don't get the views or the support.
People are gregarious, they want to be part of a team, middle ground is not a team, unless you create a new name for it. People tend to give support to whoever hates what they hate, not to those who love what they love. So the middle team shouldn't be sold as "the good parts of A and B" but as "we hate A and B".
For example, in tabs vs spaces there's the "tab for indentation, spaces for alignment" bunch, that (we) usually try to sell it like "we use tabs, so you can set the viewing space as you need, and we use spaces so it's neatly aligned" and people don't listen to us.
What if we called it "tabspacing" and defended it like "spaces for indentation are wrong because there are people with visual impairments that need extra space to see things right, and just tabs are wrong, code looks ugly, also if you use space and tabs interchangeably it's wrong, the people with problems will see the code differently than the people without, there will be errors."? Then people might listen to us if they hate one of these things.
So yeah, that's why there are so many ways to do projects: waterfall, agile, scrum, kanban, scrumban, XP, ... .
Am reminded of similar defences used for Marxist communism. "Oh, it only devolved into totalitarian misery every single time they tried it because they forgot to xyz".
But saying things like "everything is somewhere in the middle" and "use common sense" is just as much marketing speak. It's meaningless to the point it's very successfully used in both marketing and politics (but I repeat myself).
The key thing is you can set the parameters so that whatever you're selling happens to fall squarely in the middle. And of course anything can be common sense so long as you can successfully ridicule the alternatives and describe your solution in a way that sounds plausible(ish).
The reality is that it's not exactly "in the middle" either. It's the simple fact that we must really think and carefully consider these decisions and processes. But the vast majority of people don't want to think. They want recipes to follow.
That's why anything that provides a recipe, like scrum, OKRs and so forth, will get sold as a solution for all of our problems. And inevitably lazy people will follow it.
It ALWAYS depends on the context in which these processes are in, but most don't want to spend real time figuring this out. Just adopt what Google, Spotify (squads anyone?), <insert hyped company here> uses.
Oh I am sure you could come up with myriad processes, rituals and ethos and give it a snazzy name like “zero-sigma” (alluding to being the average way)
Just out of curiosity, are there any non-startup non-tiny companies whose core business is self built software (or a service that runs on said software) that _actually_ and _in practice_ follow the agile manifesto?
I'd say yes - I knew of several in the Germany (Berlin/Hamburg) area a few years ago. In my experience, actually doing agile works better in places with less middle management (just a theory).
Discovery vs design is the real dichotomy. When discovery is needed, move in small steps with lots of discussion. When design is possible, move in big steps with detailed specs
We’re biased to overestimate our understanding of problems, so we should lean towards discovery mode and small steps
Dysfunctional agile has detailed specs and little discussion (no discovery, myopic design)
For people who worship capitalism as much as folks here do, a lot of you sure don't get business processes. Yes, you wind up having to plan things, because the rest of the business world runs on deadlines and contracts and milestones and deliverables. Agile is, God forbid, almost like being an artist or low, disgusted whisper liberal arts major...when done correctly, in that you're iterating and tinkering and shipping and ideally you eventually get to good software sometime.
Only the business world runs on quarterly results and shareholder value and being able to say "yes, we are shipping 10 great new features next quarter, we have a $20 million marketing campaign, new user acquisition campaign, outreach to lapsed users, and PR campaign all ready to go" gets you bonuses and increased stock prices and new investment rounds.
Saying a bunch of Agile gobbledygook gets you...a job as a product manager or lead but you still gotta hit those milestones
The question that you're missing is: Which works better? Which delivers more better software, more reliably?
Granted, the business world does run on quarterly results and shareholder value and being able to say ..., but you can say any damn thing that you like, only promising it doesn't guarantee that it will happen. Software development is always uncertain, the question is how best to tame that - ignore it or lean into it.
All things being equal, of course the the business world would prefer the most detailed predictable plan. But, they just aren't equal. Software development benefits greatly from short feedback loops. In all phases - from debugging to finding out what the users really need. That is antithetical to big upfront plans.
There needs to be a stable narrative to shareholders, anything within your control should be managed.
Moving all software development into some kind of skunkworks part of the business with 0 expected returns and only ever discussing roll outs may be one way to do it. (Shrug)
Its why i think innovation is maybe a bit easier at a privately held or government owned org. They can look at the value more directly rather than through share value impact.
This response is just about how big a deal the shareholder perspectove is, if you can fly under that radar then you stillneed to convince everyone else that you can be trusted to deliver eventually.
> There needs to be a stable narrative to shareholders
I don't disagree with that. In fact, knowing the end goal is even more important when actions are "skunkworks".
> Moving all software development into part of the business with 0 expected returns
Agile's focus on "Working software is the primary measure of progress. ... early and continuous delivery
of valuable software." is IMHO the opposite of that characterisation; as it demonstrates value and solicits feedback as soon as feasible. Note that this has very little to do with "is it following a pre-agreed plan or not?".
The issue is more that too many businesses believe that software development is best done as top-down detailed micromanagement, and the tools at hand (i.e. Jira) lean into this delusion.
Agreed. Software development and basically all efficiency / process improvements are not really top down at all.
I just think that anything not top down doesnt play nice with shareholders. Also doesnt play nice with management that want to say they are the ones that got something delivired.
Seeing projects get rejected as they delivier too much of an improvement (e.g. we only predicted 5% savings this is likely to give 40%) really opened my eyes to how important it is to appear like you are more in control than you are.
On the skunkworks suggestion, i was meaning dont put anything from that team into any forecasts until you have a better measure of the impact.that way you controlthe narative better.
Let's say agile can deliver better software, and maybe even more reliably. The issue is, somewhere up the org chart, there's someone who operates in terms of yearly plans and quarterly results, and there's an impedance mismatch between that layer of the org chart and the agile process below them.
Even if you can really do agile (instead of "Agile"), that mismatch can ruin you. In the long run, it probably will, unless you have someone on the agile side able to talk in terms that the other side can understand. The agile side has to actively manage that connection; the business side isn't going to do it for you.
I have seen the best agile environment I have ever seen destroyed by this mismatch. Upper management killed it because it couldn't understand how to measure and predict the agile development process.
Naturally everyone does waterfall, just at different points. Even if I did the leanest form of Kanban, I have to check the requirements of each ticket, understand them, translate them to maybe a prototype/discovery phase, come up with tests and an implementation strategy and verify the result somehow. That's just waterfall at the finest possible granularity.
Waterfall doesn't mean not to iterate. It also doesn't mean to not build prototypes or MVPs. It just pushes the risky decisions back to people that are actually paid to make these decisions. Scrum is not just a framework for micromanaging, it also conveniently removes any domain responsibility from management. It's always "the team decides". (Of course, just on paper. Try to decide to do tech debt only for a couple of sprints with your team and see how management reacts.)
Besides, waterfall is just the straw man of scrum zealots. I once proposed a new team structure where frontend and backend wouldn't be mixed because I observed that most stories didn't have an overlap but where rather mostly one or the other. The scrum master was shocked that I wanted to get rid of "cross functional" teams: "What do you want to do instead, waterfall?". For them it was inconceivable that cross functionality was not equivalent to scrum.
I don't want to be pedantic, but waterfall does mean exactly that by definition (or at least it used to). There are plenty of variations, that include prototypes or MVPs or iterations (e.g. V-Model, Spiral model).
> waterfall is just the straw man of scrum zealots
The question is whether any of the "classical" examples for waterfall project managements out there didn't iterate at all. Having iterations is not incompatible with waterfall per se. E.g. you can plan to have 3 prototypes and then one final product beforehand. You can also decide to skip project phases if they are not needed.
When people hear waterfall they think of a pure methodology, when in practise the methodology nearly always will be (and was) sacrificed if the need arises. So as we are discussing a thing we do in practise, we shouldn't discuss it as a theoretical framework in the vacuum of space, but as something that will be (a)bused in reality.
As a film guy I am a big fan of clear up-front planing (you probably know some of the things that go into films, treatments, scripts, story-boards, concept art, ...) A film is very waterfall-y in how it is produced.
But good directors maintain their flexibility on set by maintaining a balance in how serious they take the script on set and where other solutions present themselves that are a thousand times better, e.g. because the location/the actors or whatever is more striking than any writer could have imagined etc.
A term has a meaning, and the fact that people don't actually practice it doesn't mean you can just change what it means.
> A film is very waterfall-y in how it is produced. But good directors maintain their flexibility on set by
Right. So they don't stick strictly with waterfall, even though the industry tends to be "waterfall-y".
My understanding is that in some (non-software) industries, waterfall is practiced because you really can't go back and change things. Or maybe you could, but it would mean increased cost and delays to the extent that you might as well have started all over.
There was a time when people thought "software engineering" was really engineering, so you had to follow traditional engineering best practices. eg. Do it right the first time, make plans and stick with them. It just turns out that this doesn't work for software, and as you said, nobody is really stupid enough to do strictly that now.
Technically, that’s exactly what it means. The entire design is signed off and completed before the implementation starts, and there is no feedback loop.
But waterfall itself was a straw man in a paper intended to recommend a much more complex process. It’s more than ironic that it became seen to be a valid process in and of itself.
(Sadly I’ve lost the link to the original paper, but if I find it I’ll update my comment)
> Besides, waterfall is just the straw man of scrum zealots.
The irony is that scrum itself is often just a wrapper for a waterfall process with all the problems it comes with. So many corporate implementations of scrum are AINO - agile in name only. Management that doesn’t understand the complexity of software is doomed to oversimplify it.
That said, if you are working iteratively - one feature at a time, with feedback - then whatever you call it, that’s not waterfall.
> The irony is that scrum itself is often just a wrapper for a waterfall process with all the problems it comes with
I worked as an external employee in an environment that had a very functional scrum implementation. You had groomings, plannings, dailys, retrospectives. Every task was defined, estimated, time on them tracked to get better in estimating. You had scrum masters, capacity plannings. On the surface, it was all very functional and productive.
But the reality is, business analysts and product managers who had little knowledge of how the system works were in charge of talking to clients, writing specifications, getting them approved and ready for development, and very little possibility to iterate if there was a problem in the definition.
It was very efficient at building a bad product. :D
I saw a quote recently from Erik Dietrich that said, "A lot of people mistake activity for productivity” and that describes every scrum implementation I’ve ever seen.
This is it, I distinctly recall a reading a paper defining Waterfall as a thought exercise, with the author coming back years later to clarify that there can never be such a thing in software development. The Wikipedia article for "Waterfall model" mentions something like that as well. As an old teacher used to say, "There's no such thing as a Waterfall model, it was your parents all along. All software development is iterative."
Winston Royce actually did have an iterative process as part of Waterfall, it was in his final model, and the diagram was on the next page of the document that most people fail to recognise.
My recollection from the paper is that he presented what’s typically discussed as waterfall at the start of the paper, and from there developed the iterative process.
Neither process was explicitly called “waterwall” - and the former is what I was taught at software engineering school.
Royce's model wasn't Waterfall. Waterfall is the model he started with, knocked, and then improved on. Idiots in DoD (and from there industry) mistook his first figure as the target, and not the thing to avoid. They codified it in a standard and it has been the cause of much government and government contractor waste ever since.
"The Internet brought agile, what will remote work bring?"
Interesting question. What are the companies doing that have been working remotely already years before the pandemic? How is e.g. Gitlab working? Would love to hear people's experience.
It's not like Agile cannot work with remote collaboration, just that waterfall is more suitable if people have a tendency to isolate and form silos.
Working remote for 10+ years and as a manager of software developers and developing myself, not sure I would see working remote or in house makes any difference on what processes you use.
The processes you use, or try to use, are limited by the culture of management that work above and around you in other departments. Changing that culture to be accepting of the nature of software development is difficult, particularly in my case where we are interacting with non technically-minded people who are prone to make decisions based on their emotions and/or their own motivations regarding office politics.
I disagree. Everyone does iterative development, always, but it comes disguised as waterfall/scrumfall/kanban/whatever.
The product is developed using waterfall and then it gets put in front of a stakeholder, who has changed their mind in the time between the initial spec and the delivery. So another, smaller, waterfall is created. This is iterative development. And it always happens, because no-one gets product development right first time.
We'd actually benefit from realising this. Which was the point of Agile - that you can change course during development and so develop the right thing through iteration. But Agile is (by definition) unpredictable. So you end up with this half-arsed solution that pretends to be predictable (except what it predicts is only the first iteration) and pretends to be Agile (except changing course is impossible) as the article says.
I think this ignores two major constraints - the "waterfall mindset" was about stakeholders (customers) believing they can get a finished product in a fixed timeline for a fixed price.
What happened then was that after the delivery, the stakeholder was wildly unhappy with the result and demanded re-work, but that would be unplanned and un-budgeted. Many ugly arguments and finger pointing would ensue, which could repeat in several "iterations".
The other thing which doesn't align with agile was that the stakeholder usually didn't participate in the development. They weren't reviewing the progress, they were typically waiting for the finished product to be delivered.
1) You can end at the first iteration and call it a day. The only reason nobody does that is because they don't benefit from an unhappy customer
2) If you have experience, nothing stops you from putting some iterations into the waterfall. As you said the need for feedback rounds should be no surprise.
Traditionally the downside to waterfall was that the distance between those iterations was too big. But smaller iterations and partly prototypes is nothing that wouldn't be allowed in waterfall if you'd like to do it. And if you find a PM that doesn't do the reasonable thing "because the methodology won't allow it" they are probably a bad PM.
For me the goal of Agile, especially in giant 100+ yo organization, is to tell people that it's less important to fit the plan than to fit the need of the external stakeholder.
Who cares if 6 months ago some important managing director signed off on a useless feature nobody wants anymore: kill it now and move on, we're agile. This is literally the only way I've seen used agile in big orgs: in words only, and it's liberating: we stop spinning endlessly between useless features as the clients move on faster than us, now we can actually skip a beat and go faster, and screw the paper pushers bemoaning that "everything changes". Because yes, everything changes.
At some level everything is waterfall. If anything is to get done at all the programmer has to make a plan, then put his head down, arse up and implement it.
At another level nothing is waterfall. If the plan is successful it was shipped to the consumer (which may be the programmer himself), and it's very presence changes things for the consumer in ways that they didn't foresee. They then realise they need a new set of changes.
What we call waterfall model is really referring to the scale. If the plan is grand, the specifications for such a grand plan need to be detailed, the implementation long, and the time between putting the head down and evaluating results is large, and we call it waterfall.
But if the strategy is to explore the solution organically, the re-evaluations are frequent, the waterfall periods are short and we call the strategy something else.
So in the end everything uses waterfall. The programmer would not get the long periods of intense focus he need to be productive without it. But also nothing is waterfall, because no plan can foresee everything, it must be continually re-evaluated in the light of unexpected changes it brings as it is implemented.
If you want to get more abstract with it: waterfall is in two dimensions. The process of starting something and working toward completion can be represented as a waterfall (in 2D) but always and by definition must be represented as a line (in 1D) between two points: start and finish.
If you project from 1D to 2D, that linearity will always be there (for completed projects, at least). That extra dimension will introduce additional flavour, and waterfall seems to be a very frequent one.
Agile always was the answer to that nagging project manager asking “When will it be done?”
By engaging with an answer the engineers kept being part of the discussion.
Instead, they should have said: you don’t get to ask that question. That question has no place here. Engage with us in building the product and let us create and enjoy!
Of course, the real world needs deadlines, timelines, roadmaps. And thus, agile is pointless. As it is just waterfall. In a different colour.
And then the project manager fires everyone and hires a team that gives them the answers they want. Even though most of the time the answers were wrong, but by that time the project manager is already neck deep with dependencies around the new team.
That's why we can't have nice things. You depend on having nice project managers, which is rarely the case.
Yup. Every single organisation I’ve worked at just paid lip service to the whole “you need to be involved in the process” observation.
Unfortunately in my career I’ve only ever seen execs recede further and further away, hiring more and more PMs to replace them in that task. The self aggrandisement is getting worse and worse. They go away for retreat camps, and strategy meet-ups etc and pat themselves on the back about doing Very Important Work for weeks after each event
Here it’s a class thing. They see themselves aligned with the capitalist owners more than the “factory workers” (the devs)
Managers should be predicting and controlling the delivery time by clearing organizational roadblocks and shaping the work to be done, not how it is done.
RUP really isn't waterfall anymore. It's not as linear. It's it's own thing, a sort of hybrid model. The nineties already knew the issue with expecting to get everything right the first time.
The problem with it is it's too complicated. Your management with the mental capacity of a five year old just wouldn't understand it.
And RUP had iterations as well, and parallel disciplines. It had a lot of good to the overall idea. Are there modern variants? As a pattern library maybe, take just what you need.
You literally cannot have good technology without enforcing this workflow and properly completing each step. Its a function of the nature of the way the mind works to solve a problem and encode that solution into a workable technology.
This natural law is resisted by those who do not have experience with it, and who also attempt to make their mark on the universe by either re-inventing this natural law, and slapping their own brand on it, or attempting to ignore key parts they don't like, and in which they are not competent, refactoring it over time only to discover .. as it is a natural law - that they end up re-inventing it, as well. And then slap their brand on it.
But I can say with great confidence after 40 years of developing technology, that the more you resist this natural law, the worse your product will be. All failed technology can be evaluated on the basis of how well this workflow was manifested by those involved.
Scrum is just New-Age Waterfall. Agile is just 'officious scrum'. 10x developers are merely individuals who have stopped resisting this natural law, and in so doing have worked out how to apply it productively in almost everything they do ..
Another factor is that Waterfall is iterative. You can do it in grandiose fashion, and you can do it on a granular, microcosm-level. The more you iterate, the more water flows, and the better things get - but people tend to resist revising themselves. The inclination to check oneself before one wrecks oneself is either bred out, culturally, or reinforced through the gauntlet of fire that is the modern industry. The better you get at iterating over the waterfall, the higher quality your product.
I'm glad you brought up iteration in the context of waterfall.
I do agree that the elements of analysis/reqs/spec etc are all vital. What isn't obviously necessary is that they happen in a rigid linear sequence. The point of MVP is to get from implementation back to analysis & requirements.
By iterating through the steps in a non-linear way, you get better results for each step. I'd argue that requirements gathering is the hardest to get right and the part that most frequently fails, probably because workers are not consulted and, even if they are, they don't speak the same language as the analysts.
So I believe that waterfall is still a good methodology, but (in most cases) only if it is done as a directed cyclic graph rather than a strict linear sequence, such that blocks of steps can be repeated.
This breaks estimation, but estimation is broken anyway. It offers that each step is, finally, done right, but recognising that the first attempt at each step is bound to be wrong and must be iterated on.
Well .. yes .. to an extent your point of view does match my view - because I feel that when you reach a step in the workflow that is weak, it indicates a prior step that is also weak. Which is why sometimes you get to Qualification of a feature, for example, and find that it doesn't match expectations - because the Requirements didn't fulfill the conclusions of the Analysis, say. Or because, indeed, of a weak Analysis or crap Implementation.
However, there is a subtlety to this:
>So I believe that waterfall is still a good methodology, but (in most cases) only if it is done as a directed cyclic graph rather than a strict linear sequence, such that blocks of steps can be repeated.
Its both! You go through linearly, and if you're lucky, you ship! But if you don't ship, you go back to the prior weak step until it is strong enough to allow you to continue the linear path - and yes, it is always best done linearly from that point, in my opinion. Don't skip forward steps if you've done review and found a weakness, usually in Qualifications, but also very often in weak Specifications or Requirements - which always indicate a failure to properly Analyse. Instead, go back to the prior incomplete/inaccurate step, and redress the issue - and continue forward with the next step from there. No skipping! Skipping makes the whole thing weak again, polluting the stream. Rigorous linearity holds the order.
For this reason, I have added 'gatekeeper' steps to my workflow, which I didn't want to include in my original comment, because I knew someone would then say "but thats not waterfall!" .. Waterfall needs review steps - navel-gazing - imho, which is why I always include:
[INTAKE] is there to protect Developer sanity. If they intake the Design and find it wanting, that is the point where they can kick it back to the team (or internally, personally) to the prior, weak missing step. Same with [REVIEW] which is there more for managers so that they have an opportunity to indicate the weak/missing steps, prior to Release ...
Without these two review steps, Developers and Managers will rip each other to pieces trying to assign blame/responsibility - but with these review steps, there is an opportunity for everyone to get along. Devs can kick things back if they feel things are weak, and nobody gets offended, because at the end of the line, prior to Release, managers can do it too ..
Note that [INTAKE] allows Developers to protect themselves from issues that will negatively impact their ability to Implement, and [REVIEW] gives Managers a means to allow other factors (market conditions, business cases, etc.) to impact Development, and respond appropriately. These tow actions, done iteratively by well-coordinated Developer/Manager roles, produce great technology.
Estimation: the more rigorous you are about the workflow, the better you get at Estimation, as a Developer and as a Manager. Also, the less rigorous you are about the linearity of the workflow, the worse your estimates - because there will be weak/missing steps which hide costs, time and energy.
The exception to this is of course open source projects operating at a much larger scale than most software companies. But they have come up with highly structured ways to develop that don't involve planning out everything in detail, which is a waterfall thing to do.
Linux project: releases regularly every eight weeks or so. Chrome & Firefox: same thing. In fact most well run larger OSS projects seem to end up using date driven milestones rather than content driven milestones. Inside these projects you of course find people or groups of people that may have their own roadmaps and plans but they keep these separate from the overall plan, which is simply to ship whatever on date x. Either their stuff is ready and it gets merged and released or it stays on a branch until it is ready for a next release.
Branches changed everything for open source developers. Every change is now a mini project. The last step of that project is to merge it, when the project is over. Some branches only last minutes, some last months/years. And those can break down into sub branches/projects. That's actually how the Linux project moves forward. Mostly Linus Torvalds is continuously integrating what is deemed ready.
The mistake many companies make over and over again is managing whole products as one project. It doesn't work. Stop doing that and life gets better. Look at how big companies release big software packages these days. They use calendar driven milestones as well. They have lots of internal projects about things that may or may not end up in there. But the releases mostly go out as planned, i.e. when they are supposed to go out with whatever is in a stable state. The show must go on. That wouldn't work with waterfall. They'd constantly be missing those deadlines. The only reasons for moving deadlines are usually quality related.
I always like this notion of inversion of control. You don't plan what ships, just when you ship and at what level of quality. This is much easier to plan for. Basically it becomes a game of quality control. If it's not ready, don't merge it and it's not on the critical path to anything. The rest just involves looking at the calendar.
Product owners hate this inversion of control because they want to put content on the calendar. A good PO actually understands this and can adapt to that. An inexperienced one will indeed revert to planning lots of silly content milestones in a calendar. And then you are doing waterfall and falling behind the unrealistic schedule. Simple solution: don't do that. A POs job is to prioritize and keep amount of work in progress low to ensure that the right things get done well and as soon as possible. A high amount of work in progress signals slipping deadlines, poorly prioritized work, and lots of people stressing out about everything being super important.
Agile processes tried to fix this. But you still end up with POs in control and sweet talking to senior management. So you end up doing waterfall if they don't know what they are doing (which happens a lot).
Writing this as a CTO who is doubling as the PO (until we can hire someone that is better at that than me, I'm not bored). My approach to this is very simple. I have a hundred things I want to get done. I pick the top two and do those and than we do some more. The hard part is picking the right two things. Not coming up with a hundred more things to do. That requires discipline and a grasp of business and tech. That's my job. A key insight here is that you shouldn't plan out those hundred things in any amount of detail until you are ready to do them. That's waterfall thinking. You are just building an inventory of crap that will never get done and is probably the wrong thing to do by the time you get around to doing those things months/years later. Instead, I set high level mid term goals and plan short term according to those goals. The short term is about what we do right now and making sure that is the best thing to...
I'm doing scrum and waterfall at the moment, customers are saying why is it costing so much? Managers are saying we're agile :-D what fun. We have design docs and requirements docs and epics and scrum masters and devops etc etc
This is the kind if project where agile can truly shine.
“Hey, customer, looks like we burnt through half your budget and it doesn’t look like we’ll be able to complete the full spec on what remains…
I could ask you to increase the budget, but I think I have a better suggestion. You tell me which feature is most important to get in and well focus on that exclusively, and iterate on that until you're either happy with what you’ve got, or the budget runs out. Then lets take a decision on how to proceed from there if still needed”
Look, I paid you your asking price for the /full/ spec; now you're telling me you will only deliver it partially and have the gall to say I get to decide how much money on top I pay you after the deadline for the full spec has passed? Why on earth would I accept this?
Look, we already catered to your "corrections" of the full spec here, here, there and then. surely this is work on our side that comes on top of the work we agreed upon. thing is, we already built the x, y and z features that your current solution is lacking, which already saves you top dollar on your business workflows. that's how good we are.
Everything tends to waterfall because the natural human "thing" is to create a process which can be repeated to ensure success / quality / <thing deemed good>. If Agile thing worked, let's package it up into something repeatable and follow that to the letter... Oh, waterfall.
Agile requires constantly working against that, which is hard, because Managers want and love safe and predictable and measurable against something concrete.
Make a framework rather than a process, but that's a fine line.
Agile ultimately needs trust in the team to be able to think on their feet for every scenario coming into their board. I'm my experience this trust either didn't exist, or is treated as a second class citizen to said safe, predicable, measurable, repeatable process.
And I don't believe necessarily that Agile is the be all and end all.
Waterfall is the default resting state of software development same as how _v3_final_FINAL_newedits_Finalv4_revised.doc is the default resting state of version control. It takes active energy to move away from that state and things inevitably drift back towards it without continued application of discipline.
At its core, the true underlying problem is that you can't actually fully control another person's output while still holding onto any genuine agility. The design is going to leave out some Product features, the code is going to diverge slightly from the design, QA will find some things the code didn't account for, DevOps, Security, and Legal will each chime in with other things that weren't considered.
If the organization requires that everyone is happy, then everyone gets to review the work at every stage (in order to express unhappiness and get it fixed before proceeding). The inevitable result is a Waterfall process to gatekeep how work moves to each stage. The inevitable result is a lack of agility.
But if the organization can lean into the Truth that Nothing Is Perfect, that your product is not perfect, your design is not perfect, your code is not perfect, your scaling, observability, security, and compliance are not perfect, then suddenly, nobody needs anybody's approval. You ship, and you ship quickly. Then you focus on learning and iterative improvement. Make things better over time, but not necessarily right now.
Getting rid of Waterfall is learning to let go and to live with imperfection.
The problem is that some things must be perfect, because they expose the company to significant risk. You can iterate on a feature but not on compliance.
You can iterate on compliance _implementation_ if you keep some tollgate that guarantees all compliance business requirements are eventually met. In my opinion this is not the same as waterfall, and leaves room for experiments and iterative learning.
The (very) tricky part, in my experience, is to have these requirements captured in implementation-agnostic way. Even strict accounting reqs require huge amount of work for someone to deconstruct them down to actual compliance requirement, instead of something solution-specific like "we need to store X in component Y".
Not everyone wants to invest into this just for agility sake.
That might work in some fields, but consider life sciences and pharma where some unmet functional/compliance issue could be injurious to the company, a patient, or both.
Waterfall-ish workflows are hard to avoid in such cases. We learned to simply do iterations of waterfall and speed up the validation via automation as much as possible.
Even if that's true - I'm not sure it is, particularly in very early R&D phases with no external impacts - all that means is that non-waterfall doesn't work absolutely 100% of the time, which nobody is saying it does.
Any time an engineering methodology is discussed, people come out of the woodwork with their specific example of one time where that methodology won't work.
In my experience at pharma there has been an enormous divide between R&D and production for software development. Precisely for the compliance implications. In cases where it hasn't been a perfect black/white differentiation it was always dirty laundry where some subset of people knew they were sitting in a grey area and hoping that compliance officers/auditors wouldn't notice.
I worked in the pharma space building document management systems that supported clinical and pre-clinical processes (mostly clinical, some R&D).
After a while, one starts to understand the need for the rigor. If some cluster of patients suffers some injurious adverse effect pre- or post-market, the FDA and a bunch of lawyers wants to trace the history of development to understand how to assign responsibility. Software systems in this space are validated quite rigorously against their functional and technical design specifications to verify that the records produced are traceable.
Software validation in this space works kind of like GPL licenses: the sponsor (like a Merck or a Pfizer) is responsible for validating software from vendors when it is supporting a GxP process. Those vendors are then responsible for validating not only their systems, but all dependent systems as well. (A lot of it is risk-based and obviously a lot of the methodology, guidance, and interpretation has evolved since the cloud era. But we used to have to take our customers on tours of our data center and their auditors would match the specifications of the hardware!)
I think it's one of the reasons why -- despite this process being extremely high in "schlep" -- there are few startups in this space because the barrier to entry is quite high. Even having 15 years of domain experience and running a team delivering production systems used by pharmas to run their trials, I don't think I'd want to do a startup in this space.
... and leave ops/ devops / are team to pick up the pieces (or put the pieces together) because now your too busy to iterate because your shipping another new thing?
1. If you're agile, you're not too busy to iterate because you're shipping another new thing. You're able to come back and iterate if that's the most valuable thing you can work on.
2. When waterfall delivers the wrong thing because the requirements from two years ago weren't updated to reflect two years of new reality, someone has to pick up the pieces from that, too. It may not be devops - it might be sales instead. But it's still a problem.
The standard fix for this is to put the devs on call.
Recurring operational issues get fixed mighty quickly when J. Dev has to wake up a few times at 02:00 on Saturday.
Heck, design and architectural issues suddenly get a lot of scrutiny at the whiteboard phase and people decide they don't really need Kafka or Kubernetes or Mesos or GenAI anymore.
At a $LASTJOB all the devs had to be on call for some services and literally nothing improved. Still haphazardly throwing code into production, poor monitoring tools, and no time or desire to make stuff better. On-call was just seen as shitwork and devs would find other teams that didn't have on-call to transfer to.
This is why I am a fan of build & run teams instead of having a separate ops org which is responsible for keeping things running. The incentives aren’t aligned and almost always result in silos and excessive outages. The shared responsibility means no one is actually accountable and solvable issues persist for years. Almost every client I work with who has major ops issues have separate teams handling build and run aspects of their projects.
In 15 years of experience here and there, only one has had on-call devs. I’ve come to the conclusion that neither dev nor their management care if someone else’s phone rings at 02:00. I’ve also seen open hostility to 1) the very idea of dev going on-call, and 2) technical recommendations for solutions to recurring issues. So I think it’s not quite “the standard fix”, in fact I think it’s pretty unrealistic.
That's kinda how it works at my current job, where a support rota is manned by members of the various development teams (front end dev, backend dev, app dev, platform ops, etc). You can indeed get woken up at 2am on a Saturday, and expected to fix whatever mission critical issue has popped up at that point.
It doesn't seem to have changed much as far as the architecture process goes though. With a few exceptions, the things that pop up for support to handle are usually either 'something broke on the content side, and the editors can't fix it', or a third party broke (glares at Google and Tag Manager, Musk and Twitter/X, etc)
That thinking is one reason why I left engineering and I don’t see myself coming back to it anytime soon. I don’t like being on-call, I’m not qualified to do Ops-like decisions and I haven’t seen this as the practice in most other industries - Boeing doesn’t have staff flying planes for the airlines, for one.
Compliance is a set of stories/tickets/requirements like any other, with a priority to be assigned and eventually worked and reworked at some point in the process. There’s nothing wrong with not addressing it as the first thing - it just blocks release. With that, it will eventually get worked on, hopefully at the time where the pieces that need it are understood and the work to reach it is understood.
I think this is totally right. But some things really do need to be as close to perfect as you can get them. The things in that category that I've run into most are security, privacy, and legal compliance. But in other applications, safety issues would also go in this bucket.
So those things really do need a proactive gate, or maybe in some cases, at least an extremely fast reactive response capability.
But this just means it is more important to identify the smallest set of considerations like this, in order to give them special treatment, rather than requiring higher scrutiny for everything - if everything is special nothing is special - and to invest in ways for work to be done without implicating those special considerations.
Waterfall is probably the only way of managing large quantities of differently skilled people on a software project and actually getting somewhere though.
True agile is for very small teams of highly competent people only, which are rare.
There are some corrupt organisations out there selling process models which tell you you're getting the latter but actually sell you waterfall again with bits renamed. This drives people away who are sold the agile model at an interview because the people who are invested in it in the organisation don't know either process well enough to rationalise it. So people arrive, realise it's a shit show and leave again quickly. That's reality and it sucks.
I think people need to talk more around the social and political structures of organisations that drive the process model rather than two camps because there's infinite varieties of both ways of doing things.
Waterfall always fails, on every kind of project management.
It's a very recent invention, from the late 20th century, and mostly only "used" in theoretical works until a ridiculous small number of decades ago. It actually being used on practice is very likely the main reason why large projects cost so much nowadays.
It is really not "the only way of managing large number of skilled people".
Nitpick: The RUP is not waterfall. First sentence on Wikipedia: "The rational unified process is an iterative software development process framework created by the Rational Software Corporation, a division of IBM since 2003".
Now, it IS very heavy and bureaucratic, and 99% of corporations applied it in the worst way possible, with heavy tooling and in strict waterfall fashion (even IBM/Rational consultants).
Every project needs some time where you decide what to build, and some time where you build it. The idea of agile and other methods wasn't to eliminate that because you can't. The idea was to reduce the chunk size for each batch (ie flesh out a part of the feature instead of the full product), reduce the detail needed in each chunk, and hence create faster cycle times and more room for feedbaxk, which increases learning, improves speed, introduces autonomy, and decreases risk.
"Waterfall" in the broader sense is inevitable. What we moved away from was "let's spend 6 months writing up a 300 page specifications doc for this new Operating System that will be handed off to 100 engs" to "a PM and designer can mockup this feature in a few days with some input from the person who will implement it, then the person who will implement it will build it with input from them along the way".
No amount methodology is going to make gold come out of a team that doesn't deeply understand the reasons for building what they're building.
I work with a small-a agile team, and the devs who "get it" just make it happen without a waterfall. The ones who don't get it get a super detailed spec to minimize the chances they'll mess it up, and still mess it up. The ones in the middle know when to reach out to ask for an out-of-cycle product clarification before building a useless pile of code.
The main difference is, do they understand why they're building what they're building? Or do they show up, punch through a bunch of tickets, go home? Really hard to fix that with process when that's not working. It starts with good hiring, building a vision together for what we're building, and communicating it repeatedly.
Assuming a team of all at least proficient developers, one who's living a peaceful life outside of work is going to be put into the first category, and people with more chaos in their lives (not by their choice) will fall into the latter category. As their life situation changes they have a bigger chance to lose context, forget who's who or miss out on word of mouth; or conversely, become a sponge for company culture and leave the office with energy for fulfilling projects outside of work and get involved with, for example, open source where they gain insight that they take into work later, for example perspectives on business specific knowledge or some analogues.
I think some developers are just not proficient though. You make a fair point about life circumstances, but not everyone can be prescribed to this track.
Also it's a matter of skill fit. To take extremes I worked with engineers experienced in mobile app game engineering, who were thinking about UI latency in obsessive detail, and engineers skilled in thinking about ML ranking problems, who had deep knowledge about the strengths and weaknesses of various ML approaches. You could've swapped their roles and it would've been a disaster, not because they're not proficient, but because it would've taken a long time to ramp up on what's important in the other's area
If you have a skilled team with deep domain knowledge, you can afford to not be as detailed in the planning process as the team is most likely already aligned.
If you have a skilled team without deep domain knowledge, you need that sense of unity and alignment, that usually comes through more detailed planning.
It's not one's correct or one's wrong, it just depends on the team composition.
If you’ve got a release deadline, and a list of features that need to be ready by the deadline then you’ll end up doing waterfall.
If you have flexibility about when features can be released then you can do agile.
I wish the agile and scrum evangelists were honest about this.
The most agile company I ever worked for never talked about scrum or agile. We just asked customers what they wanted and integrated it into the product. When features were ready we released them. There was no roadmap or timetable. Sometimes customers would ask us when things will be ready, we always pushed back on giving dates.
143 comments
[ 3.0 ms ] story [ 202 ms ] threadBeyond the IT department everyone else has very to low incentives to adopt "those crazy ideas from computer guys", they care about deadlines, roadmaps, when marketing campaigns can be organised, with what content, what should be on the contracts with business partners,...
Ever since we got those ideas around 2000, the best outcome if agile is adopted at all, is scrumfall as the article describes.
However I have also seen many companies go back to classical requirements documents, and all related processes, as if agile had never been in the building.
God intended water to run from the TOP to the BOTTOM. Don't fight gravity.
The worst results I have ever seen have been projects where the people on the bottom stopped caring because the people on the top tried to make them ignore the laws of nature.
Also: God is dead
But if you need to organize 50+ engineers and actually produce a ... product? That people/agencies/customers expect things of you sure as shit is going to need a definable goal, verifiable output and a somewhat chronological way of achieveing that. Waterfall it is (with reasonable iterations).
Waterfall is a pacifier for people who would rather have lies than uncertainty.
But the often used CER -> PDR -> CDR -> TRR etc. is damn useful.
This is the truth. It's painful to see, but some people would prefer to light millions of dollars on fire than admit they don't know.
No one says Agile can't have full blown requirements or the respective documents.
To the contrary, you can't expect your dev to understand what he has to write without giving him a goal he is working on - Agile won't fix that.
Agile just means you build in small steps from what you have and review your steps ASAP, instead of working multiple months based on some detailed documents, derived from other documents, written by some other department months ago, building stuff no one wanted in the first place.
Software development requires some planing and also some flexibility. One decade, companies think software architecture is the most important technical role in the universe, the next decade, the role barely even exists... What we need more of is common sense.
I have. It didn't reach the end.
https://www.praxisframework.org/files/royce1970.pdf
Our industry is well on the path to ignore the obvious truths for 50 years...
As an improvement, sometimes the project management *forced* the customer to make some checkpoints along the way, but they were a far cry from real iterations.
I'd just add that scrum and waterfall are both more useful in explaining to management and junior engineers approximately how the team/project works, than in actually running the team/project.
Some people do, but they don't get the views or the support.
People are gregarious, they want to be part of a team, middle ground is not a team, unless you create a new name for it. People tend to give support to whoever hates what they hate, not to those who love what they love. So the middle team shouldn't be sold as "the good parts of A and B" but as "we hate A and B".
For example, in tabs vs spaces there's the "tab for indentation, spaces for alignment" bunch, that (we) usually try to sell it like "we use tabs, so you can set the viewing space as you need, and we use spaces so it's neatly aligned" and people don't listen to us.
What if we called it "tabspacing" and defended it like "spaces for indentation are wrong because there are people with visual impairments that need extra space to see things right, and just tabs are wrong, code looks ugly, also if you use space and tabs interchangeably it's wrong, the people with problems will see the code differently than the people without, there will be errors."? Then people might listen to us if they hate one of these things.
So yeah, that's why there are so many ways to do projects: waterfall, agile, scrum, kanban, scrumban, XP, ... .
Bit of a weird mixing of levels there: Everything you mention after "agile" is (more or less) "agile".
The most common defence from Certified Scrum Masters TM © is that you haven’t implemented it correctly/fully.
Does YMMV?
I'm feeling grumpy and decaffinated this morning, so I'll just note that you can't sell books and courses and conferences on "balance and nuance".
The key thing is you can set the parameters so that whatever you're selling happens to fall squarely in the middle. And of course anything can be common sense so long as you can successfully ridicule the alternatives and describe your solution in a way that sounds plausible(ish).
That's why anything that provides a recipe, like scrum, OKRs and so forth, will get sold as a solution for all of our problems. And inevitably lazy people will follow it.
It ALWAYS depends on the context in which these processes are in, but most don't want to spend real time figuring this out. Just adopt what Google, Spotify (squads anyone?), <insert hyped company here> uses.
Surely scrum is too little? Most lowercase-a agile people consider it more of a stepping stone into agile.
And no, AIUI most original-ideals-agilists consider (much or most of) Scrum a stepping stone away from "agile".
We’re biased to overestimate our understanding of problems, so we should lean towards discovery mode and small steps
Dysfunctional agile has detailed specs and little discussion (no discovery, myopic design)
Only the business world runs on quarterly results and shareholder value and being able to say "yes, we are shipping 10 great new features next quarter, we have a $20 million marketing campaign, new user acquisition campaign, outreach to lapsed users, and PR campaign all ready to go" gets you bonuses and increased stock prices and new investment rounds.
Saying a bunch of Agile gobbledygook gets you...a job as a product manager or lead but you still gotta hit those milestones
Granted, the business world does run on quarterly results and shareholder value and being able to say ..., but you can say any damn thing that you like, only promising it doesn't guarantee that it will happen. Software development is always uncertain, the question is how best to tame that - ignore it or lean into it.
All things being equal, of course the the business world would prefer the most detailed predictable plan. But, they just aren't equal. Software development benefits greatly from short feedback loops. In all phases - from debugging to finding out what the users really need. That is antithetical to big upfront plans.
Moving all software development into some kind of skunkworks part of the business with 0 expected returns and only ever discussing roll outs may be one way to do it. (Shrug)
Its why i think innovation is maybe a bit easier at a privately held or government owned org. They can look at the value more directly rather than through share value impact.
This response is just about how big a deal the shareholder perspectove is, if you can fly under that radar then you stillneed to convince everyone else that you can be trusted to deliver eventually.
I don't disagree with that. In fact, knowing the end goal is even more important when actions are "skunkworks".
> Moving all software development into part of the business with 0 expected returns
Agile's focus on "Working software is the primary measure of progress. ... early and continuous delivery of valuable software." is IMHO the opposite of that characterisation; as it demonstrates value and solicits feedback as soon as feasible. Note that this has very little to do with "is it following a pre-agreed plan or not?".
The issue is more that too many businesses believe that software development is best done as top-down detailed micromanagement, and the tools at hand (i.e. Jira) lean into this delusion.
I just think that anything not top down doesnt play nice with shareholders. Also doesnt play nice with management that want to say they are the ones that got something delivired.
Seeing projects get rejected as they delivier too much of an improvement (e.g. we only predicted 5% savings this is likely to give 40%) really opened my eyes to how important it is to appear like you are more in control than you are.
On the skunkworks suggestion, i was meaning dont put anything from that team into any forecasts until you have a better measure of the impact.that way you controlthe narative better.
Even if you can really do agile (instead of "Agile"), that mismatch can ruin you. In the long run, it probably will, unless you have someone on the agile side able to talk in terms that the other side can understand. The agile side has to actively manage that connection; the business side isn't going to do it for you.
I have seen the best agile environment I have ever seen destroyed by this mismatch. Upper management killed it because it couldn't understand how to measure and predict the agile development process.
I just don’t see it.
But I see bunch of middle managers who think it will affect their bonus if they push little guys to deliver sooner.
The middle managers definitely win more if they ship more.
Waterfall doesn't mean not to iterate. It also doesn't mean to not build prototypes or MVPs. It just pushes the risky decisions back to people that are actually paid to make these decisions. Scrum is not just a framework for micromanaging, it also conveniently removes any domain responsibility from management. It's always "the team decides". (Of course, just on paper. Try to decide to do tech debt only for a couple of sprints with your team and see how management reacts.)
Besides, waterfall is just the straw man of scrum zealots. I once proposed a new team structure where frontend and backend wouldn't be mixed because I observed that most stories didn't have an overlap but where rather mostly one or the other. The scrum master was shocked that I wanted to get rid of "cross functional" teams: "What do you want to do instead, waterfall?". For them it was inconceivable that cross functionality was not equivalent to scrum.
I don't want to be pedantic, but waterfall does mean exactly that by definition (or at least it used to). There are plenty of variations, that include prototypes or MVPs or iterations (e.g. V-Model, Spiral model).
> waterfall is just the straw man of scrum zealots
You can't just change the definition of a word, and then call people using the original definition "zealots", see https://en.wikipedia.org/wiki/Motte-and-bailey_fallacy
When people hear waterfall they think of a pure methodology, when in practise the methodology nearly always will be (and was) sacrificed if the need arises. So as we are discussing a thing we do in practise, we shouldn't discuss it as a theoretical framework in the vacuum of space, but as something that will be (a)bused in reality.
As a film guy I am a big fan of clear up-front planing (you probably know some of the things that go into films, treatments, scripts, story-boards, concept art, ...) A film is very waterfall-y in how it is produced. But good directors maintain their flexibility on set by maintaining a balance in how serious they take the script on set and where other solutions present themselves that are a thousand times better, e.g. because the location/the actors or whatever is more striking than any writer could have imagined etc.
> A film is very waterfall-y in how it is produced. But good directors maintain their flexibility on set by
Right. So they don't stick strictly with waterfall, even though the industry tends to be "waterfall-y".
My understanding is that in some (non-software) industries, waterfall is practiced because you really can't go back and change things. Or maybe you could, but it would mean increased cost and delays to the extent that you might as well have started all over.
There was a time when people thought "software engineering" was really engineering, so you had to follow traditional engineering best practices. eg. Do it right the first time, make plans and stick with them. It just turns out that this doesn't work for software, and as you said, nobody is really stupid enough to do strictly that now.
Hilariously: https://en.wikipedia.org/wiki/Waterfall_model#Royce's_final_...
No.
This article The Myth of the 'Waterfall' SDLC gives you the details (archive version since the site seems to be down) - https://archive.is/jbBta
Technically, that’s exactly what it means. The entire design is signed off and completed before the implementation starts, and there is no feedback loop.
But waterfall itself was a straw man in a paper intended to recommend a much more complex process. It’s more than ironic that it became seen to be a valid process in and of itself.
(Sadly I’ve lost the link to the original paper, but if I find it I’ll update my comment)
> Besides, waterfall is just the straw man of scrum zealots.
The irony is that scrum itself is often just a wrapper for a waterfall process with all the problems it comes with. So many corporate implementations of scrum are AINO - agile in name only. Management that doesn’t understand the complexity of software is doomed to oversimplify it.
That said, if you are working iteratively - one feature at a time, with feedback - then whatever you call it, that’s not waterfall.
I disagree. I've been in many projects which looked exactly like the stereotypical waterfall, with the expected results (delays, death marches).
Unfortunately the strawman was what started getting traction.
The horror is that it got picked up as a legitimate SDLC. Hence your exposure to it.
It’s almost like they saw the pictures and didn’t read the words.
At the same time, that process he described was the standard procedure for government and any large corporation contracts.
I worked as an external employee in an environment that had a very functional scrum implementation. You had groomings, plannings, dailys, retrospectives. Every task was defined, estimated, time on them tracked to get better in estimating. You had scrum masters, capacity plannings. On the surface, it was all very functional and productive.
But the reality is, business analysts and product managers who had little knowledge of how the system works were in charge of talking to clients, writing specifications, getting them approved and ready for development, and very little possibility to iterate if there was a problem in the definition.
It was very efficient at building a bad product. :D
I saw a quote recently from Erik Dietrich that said, "A lot of people mistake activity for productivity” and that describes every scrum implementation I’ve ever seen.
It’s a cargo cult, no doubt in my mind.
https://upload.wikimedia.org/wikipedia/commons/d/de/1970_Roy...
Neither process was explicitly called “waterwall” - and the former is what I was taught at software engineering school.
Interesting question. What are the companies doing that have been working remotely already years before the pandemic? How is e.g. Gitlab working? Would love to hear people's experience.
It's not like Agile cannot work with remote collaboration, just that waterfall is more suitable if people have a tendency to isolate and form silos.
The processes you use, or try to use, are limited by the culture of management that work above and around you in other departments. Changing that culture to be accepting of the nature of software development is difficult, particularly in my case where we are interacting with non technically-minded people who are prone to make decisions based on their emotions and/or their own motivations regarding office politics.
The product is developed using waterfall and then it gets put in front of a stakeholder, who has changed their mind in the time between the initial spec and the delivery. So another, smaller, waterfall is created. This is iterative development. And it always happens, because no-one gets product development right first time.
We'd actually benefit from realising this. Which was the point of Agile - that you can change course during development and so develop the right thing through iteration. But Agile is (by definition) unpredictable. So you end up with this half-arsed solution that pretends to be predictable (except what it predicts is only the first iteration) and pretends to be Agile (except changing course is impossible) as the article says.
What happened then was that after the delivery, the stakeholder was wildly unhappy with the result and demanded re-work, but that would be unplanned and un-budgeted. Many ugly arguments and finger pointing would ensue, which could repeat in several "iterations".
The other thing which doesn't align with agile was that the stakeholder usually didn't participate in the development. They weren't reviewing the progress, they were typically waiting for the finished product to be delivered.
1) You can end at the first iteration and call it a day. The only reason nobody does that is because they don't benefit from an unhappy customer
2) If you have experience, nothing stops you from putting some iterations into the waterfall. As you said the need for feedback rounds should be no surprise.
Traditionally the downside to waterfall was that the distance between those iterations was too big. But smaller iterations and partly prototypes is nothing that wouldn't be allowed in waterfall if you'd like to do it. And if you find a PM that doesn't do the reasonable thing "because the methodology won't allow it" they are probably a bad PM.
Who cares if 6 months ago some important managing director signed off on a useless feature nobody wants anymore: kill it now and move on, we're agile. This is literally the only way I've seen used agile in big orgs: in words only, and it's liberating: we stop spinning endlessly between useless features as the clients move on faster than us, now we can actually skip a beat and go faster, and screw the paper pushers bemoaning that "everything changes". Because yes, everything changes.
At some level everything is waterfall. If anything is to get done at all the programmer has to make a plan, then put his head down, arse up and implement it.
At another level nothing is waterfall. If the plan is successful it was shipped to the consumer (which may be the programmer himself), and it's very presence changes things for the consumer in ways that they didn't foresee. They then realise they need a new set of changes.
What we call waterfall model is really referring to the scale. If the plan is grand, the specifications for such a grand plan need to be detailed, the implementation long, and the time between putting the head down and evaluating results is large, and we call it waterfall.
But if the strategy is to explore the solution organically, the re-evaluations are frequent, the waterfall periods are short and we call the strategy something else.
So in the end everything uses waterfall. The programmer would not get the long periods of intense focus he need to be productive without it. But also nothing is waterfall, because no plan can foresee everything, it must be continually re-evaluated in the light of unexpected changes it brings as it is implemented.
From https://news.ycombinator.com/item?id=23970156 : by rstuart4133 on July 27, 2020
If you project from 1D to 2D, that linearity will always be there (for completed projects, at least). That extra dimension will introduce additional flavour, and waterfall seems to be a very frequent one.
Can't have product owners without actual ownership - without real equity in the product being developed do they really own anything?
By engaging with an answer the engineers kept being part of the discussion.
Instead, they should have said: you don’t get to ask that question. That question has no place here. Engage with us in building the product and let us create and enjoy!
Of course, the real world needs deadlines, timelines, roadmaps. And thus, agile is pointless. As it is just waterfall. In a different colour.
That's why we can't have nice things. You depend on having nice project managers, which is rarely the case.
Unfortunately in my career I’ve only ever seen execs recede further and further away, hiring more and more PMs to replace them in that task. The self aggrandisement is getting worse and worse. They go away for retreat camps, and strategy meet-ups etc and pat themselves on the back about doing Very Important Work for weeks after each event
Here it’s a class thing. They see themselves aligned with the capitalist owners more than the “factory workers” (the devs)
The problem with it is it's too complicated. Your management with the mental capacity of a five year old just wouldn't understand it.
https://web.archive.org/web/20230322153353/https://sceweb.uh...
Analysis - Requirements - Specification - Design - Implementation - Qualification - Testing - Release.
You literally cannot have good technology without enforcing this workflow and properly completing each step. Its a function of the nature of the way the mind works to solve a problem and encode that solution into a workable technology.
This natural law is resisted by those who do not have experience with it, and who also attempt to make their mark on the universe by either re-inventing this natural law, and slapping their own brand on it, or attempting to ignore key parts they don't like, and in which they are not competent, refactoring it over time only to discover .. as it is a natural law - that they end up re-inventing it, as well. And then slap their brand on it.
But I can say with great confidence after 40 years of developing technology, that the more you resist this natural law, the worse your product will be. All failed technology can be evaluated on the basis of how well this workflow was manifested by those involved.
Scrum is just New-Age Waterfall. Agile is just 'officious scrum'. 10x developers are merely individuals who have stopped resisting this natural law, and in so doing have worked out how to apply it productively in almost everything they do ..
Another factor is that Waterfall is iterative. You can do it in grandiose fashion, and you can do it on a granular, microcosm-level. The more you iterate, the more water flows, and the better things get - but people tend to resist revising themselves. The inclination to check oneself before one wrecks oneself is either bred out, culturally, or reinforced through the gauntlet of fire that is the modern industry. The better you get at iterating over the waterfall, the higher quality your product.
I do agree that the elements of analysis/reqs/spec etc are all vital. What isn't obviously necessary is that they happen in a rigid linear sequence. The point of MVP is to get from implementation back to analysis & requirements.
By iterating through the steps in a non-linear way, you get better results for each step. I'd argue that requirements gathering is the hardest to get right and the part that most frequently fails, probably because workers are not consulted and, even if they are, they don't speak the same language as the analysts.
So I believe that waterfall is still a good methodology, but (in most cases) only if it is done as a directed cyclic graph rather than a strict linear sequence, such that blocks of steps can be repeated.
This breaks estimation, but estimation is broken anyway. It offers that each step is, finally, done right, but recognising that the first attempt at each step is bound to be wrong and must be iterated on.
Does this match your view?
However, there is a subtlety to this:
>So I believe that waterfall is still a good methodology, but (in most cases) only if it is done as a directed cyclic graph rather than a strict linear sequence, such that blocks of steps can be repeated.
Its both! You go through linearly, and if you're lucky, you ship! But if you don't ship, you go back to the prior weak step until it is strong enough to allow you to continue the linear path - and yes, it is always best done linearly from that point, in my opinion. Don't skip forward steps if you've done review and found a weakness, usually in Qualifications, but also very often in weak Specifications or Requirements - which always indicate a failure to properly Analyse. Instead, go back to the prior incomplete/inaccurate step, and redress the issue - and continue forward with the next step from there. No skipping! Skipping makes the whole thing weak again, polluting the stream. Rigorous linearity holds the order.
For this reason, I have added 'gatekeeper' steps to my workflow, which I didn't want to include in my original comment, because I knew someone would then say "but thats not waterfall!" .. Waterfall needs review steps - navel-gazing - imho, which is why I always include:
Analysis - Requirements - Specification - Design - [INTAKE] - Implementation - Qualification - Testing - [REVIEW] - Release.
[INTAKE] is there to protect Developer sanity. If they intake the Design and find it wanting, that is the point where they can kick it back to the team (or internally, personally) to the prior, weak missing step. Same with [REVIEW] which is there more for managers so that they have an opportunity to indicate the weak/missing steps, prior to Release ...
Without these two review steps, Developers and Managers will rip each other to pieces trying to assign blame/responsibility - but with these review steps, there is an opportunity for everyone to get along. Devs can kick things back if they feel things are weak, and nobody gets offended, because at the end of the line, prior to Release, managers can do it too ..
Note that [INTAKE] allows Developers to protect themselves from issues that will negatively impact their ability to Implement, and [REVIEW] gives Managers a means to allow other factors (market conditions, business cases, etc.) to impact Development, and respond appropriately. These tow actions, done iteratively by well-coordinated Developer/Manager roles, produce great technology.
Estimation: the more rigorous you are about the workflow, the better you get at Estimation, as a Developer and as a Manager. Also, the less rigorous you are about the linearity of the workflow, the worse your estimates - because there will be weak/missing steps which hide costs, time and energy.
Linux project: releases regularly every eight weeks or so. Chrome & Firefox: same thing. In fact most well run larger OSS projects seem to end up using date driven milestones rather than content driven milestones. Inside these projects you of course find people or groups of people that may have their own roadmaps and plans but they keep these separate from the overall plan, which is simply to ship whatever on date x. Either their stuff is ready and it gets merged and released or it stays on a branch until it is ready for a next release.
Branches changed everything for open source developers. Every change is now a mini project. The last step of that project is to merge it, when the project is over. Some branches only last minutes, some last months/years. And those can break down into sub branches/projects. That's actually how the Linux project moves forward. Mostly Linus Torvalds is continuously integrating what is deemed ready.
The mistake many companies make over and over again is managing whole products as one project. It doesn't work. Stop doing that and life gets better. Look at how big companies release big software packages these days. They use calendar driven milestones as well. They have lots of internal projects about things that may or may not end up in there. But the releases mostly go out as planned, i.e. when they are supposed to go out with whatever is in a stable state. The show must go on. That wouldn't work with waterfall. They'd constantly be missing those deadlines. The only reasons for moving deadlines are usually quality related.
I always like this notion of inversion of control. You don't plan what ships, just when you ship and at what level of quality. This is much easier to plan for. Basically it becomes a game of quality control. If it's not ready, don't merge it and it's not on the critical path to anything. The rest just involves looking at the calendar.
Product owners hate this inversion of control because they want to put content on the calendar. A good PO actually understands this and can adapt to that. An inexperienced one will indeed revert to planning lots of silly content milestones in a calendar. And then you are doing waterfall and falling behind the unrealistic schedule. Simple solution: don't do that. A POs job is to prioritize and keep amount of work in progress low to ensure that the right things get done well and as soon as possible. A high amount of work in progress signals slipping deadlines, poorly prioritized work, and lots of people stressing out about everything being super important.
Agile processes tried to fix this. But you still end up with POs in control and sweet talking to senior management. So you end up doing waterfall if they don't know what they are doing (which happens a lot).
Writing this as a CTO who is doubling as the PO (until we can hire someone that is better at that than me, I'm not bored). My approach to this is very simple. I have a hundred things I want to get done. I pick the top two and do those and than we do some more. The hard part is picking the right two things. Not coming up with a hundred more things to do. That requires discipline and a grasp of business and tech. That's my job. A key insight here is that you shouldn't plan out those hundred things in any amount of detail until you are ready to do them. That's waterfall thinking. You are just building an inventory of crap that will never get done and is probably the wrong thing to do by the time you get around to doing those things months/years later. Instead, I set high level mid term goals and plan short term according to those goals. The short term is about what we do right now and making sure that is the best thing to...
“Hey, customer, looks like we burnt through half your budget and it doesn’t look like we’ll be able to complete the full spec on what remains…
I could ask you to increase the budget, but I think I have a better suggestion. You tell me which feature is most important to get in and well focus on that exclusively, and iterate on that until you're either happy with what you’ve got, or the budget runs out. Then lets take a decision on how to proceed from there if still needed”
You can choose to accept this or leave the customer to yell at and sue somebody else.
Everything tends to waterfall because the natural human "thing" is to create a process which can be repeated to ensure success / quality / <thing deemed good>. If Agile thing worked, let's package it up into something repeatable and follow that to the letter... Oh, waterfall.
Agile requires constantly working against that, which is hard, because Managers want and love safe and predictable and measurable against something concrete.
Make a framework rather than a process, but that's a fine line.
Agile ultimately needs trust in the team to be able to think on their feet for every scenario coming into their board. I'm my experience this trust either didn't exist, or is treated as a second class citizen to said safe, predicable, measurable, repeatable process.
And I don't believe necessarily that Agile is the be all and end all.
If the organization requires that everyone is happy, then everyone gets to review the work at every stage (in order to express unhappiness and get it fixed before proceeding). The inevitable result is a Waterfall process to gatekeep how work moves to each stage. The inevitable result is a lack of agility.
But if the organization can lean into the Truth that Nothing Is Perfect, that your product is not perfect, your design is not perfect, your code is not perfect, your scaling, observability, security, and compliance are not perfect, then suddenly, nobody needs anybody's approval. You ship, and you ship quickly. Then you focus on learning and iterative improvement. Make things better over time, but not necessarily right now.
Getting rid of Waterfall is learning to let go and to live with imperfection.
The (very) tricky part, in my experience, is to have these requirements captured in implementation-agnostic way. Even strict accounting reqs require huge amount of work for someone to deconstruct them down to actual compliance requirement, instead of something solution-specific like "we need to store X in component Y".
Not everyone wants to invest into this just for agility sake.
It would be a good architectural decision, actually.
That might work in some fields, but consider life sciences and pharma where some unmet functional/compliance issue could be injurious to the company, a patient, or both.
Waterfall-ish workflows are hard to avoid in such cases. We learned to simply do iterations of waterfall and speed up the validation via automation as much as possible.
Any time an engineering methodology is discussed, people come out of the woodwork with their specific example of one time where that methodology won't work.
After a while, one starts to understand the need for the rigor. If some cluster of patients suffers some injurious adverse effect pre- or post-market, the FDA and a bunch of lawyers wants to trace the history of development to understand how to assign responsibility. Software systems in this space are validated quite rigorously against their functional and technical design specifications to verify that the records produced are traceable.
Software validation in this space works kind of like GPL licenses: the sponsor (like a Merck or a Pfizer) is responsible for validating software from vendors when it is supporting a GxP process. Those vendors are then responsible for validating not only their systems, but all dependent systems as well. (A lot of it is risk-based and obviously a lot of the methodology, guidance, and interpretation has evolved since the cloud era. But we used to have to take our customers on tours of our data center and their auditors would match the specifications of the hardware!)
I think it's one of the reasons why -- despite this process being extremely high in "schlep" -- there are few startups in this space because the barrier to entry is quite high. Even having 15 years of domain experience and running a team delivering production systems used by pharmas to run their trials, I don't think I'd want to do a startup in this space.
There's a really great HN thread on this that resonated with my experience that I bookmarked a while back: https://news.ycombinator.com/item?id=33545209
2. When waterfall delivers the wrong thing because the requirements from two years ago weren't updated to reflect two years of new reality, someone has to pick up the pieces from that, too. It may not be devops - it might be sales instead. But it's still a problem.
Recurring operational issues get fixed mighty quickly when J. Dev has to wake up a few times at 02:00 on Saturday.
Heck, design and architectural issues suddenly get a lot of scrutiny at the whiteboard phase and people decide they don't really need Kafka or Kubernetes or Mesos or GenAI anymore.
It doesn't seem to have changed much as far as the architecture process goes though. With a few exceptions, the things that pop up for support to handle are usually either 'something broke on the content side, and the editors can't fix it', or a third party broke (glares at Google and Tag Manager, Musk and Twitter/X, etc)
Compliance is a set of stories/tickets/requirements like any other, with a priority to be assigned and eventually worked and reworked at some point in the process. There’s nothing wrong with not addressing it as the first thing - it just blocks release. With that, it will eventually get worked on, hopefully at the time where the pieces that need it are understood and the work to reach it is understood.
So those things really do need a proactive gate, or maybe in some cases, at least an extremely fast reactive response capability.
But this just means it is more important to identify the smallest set of considerations like this, in order to give them special treatment, rather than requiring higher scrutiny for everything - if everything is special nothing is special - and to invest in ways for work to be done without implicating those special considerations.
To a degree. But I'm pretty sure one of the reasons why software quality is so much lower these days is because many teams take the idea too far.
Waterfall is probably the only way of managing large quantities of differently skilled people on a software project and actually getting somewhere though.
True agile is for very small teams of highly competent people only, which are rare.
There are some corrupt organisations out there selling process models which tell you you're getting the latter but actually sell you waterfall again with bits renamed. This drives people away who are sold the agile model at an interview because the people who are invested in it in the organisation don't know either process well enough to rationalise it. So people arrive, realise it's a shit show and leave again quickly. That's reality and it sucks.
I think people need to talk more around the social and political structures of organisations that drive the process model rather than two camps because there's infinite varieties of both ways of doing things.
It's a very recent invention, from the late 20th century, and mostly only "used" in theoretical works until a ridiculous small number of decades ago. It actually being used on practice is very likely the main reason why large projects cost so much nowadays.
It is really not "the only way of managing large number of skilled people".
Now, it IS very heavy and bureaucratic, and 99% of corporations applied it in the worst way possible, with heavy tooling and in strict waterfall fashion (even IBM/Rational consultants).
And I say that as someone who works in Agile, supports it 100%, and has no time for the "it's a scam" whining.
"Waterfall" in the broader sense is inevitable. What we moved away from was "let's spend 6 months writing up a 300 page specifications doc for this new Operating System that will be handed off to 100 engs" to "a PM and designer can mockup this feature in a few days with some input from the person who will implement it, then the person who will implement it will build it with input from them along the way".
I work with a small-a agile team, and the devs who "get it" just make it happen without a waterfall. The ones who don't get it get a super detailed spec to minimize the chances they'll mess it up, and still mess it up. The ones in the middle know when to reach out to ask for an out-of-cycle product clarification before building a useless pile of code.
The main difference is, do they understand why they're building what they're building? Or do they show up, punch through a bunch of tickets, go home? Really hard to fix that with process when that's not working. It starts with good hiring, building a vision together for what we're building, and communicating it repeatedly.
If you have a skilled team with deep domain knowledge, you can afford to not be as detailed in the planning process as the team is most likely already aligned.
If you have a skilled team without deep domain knowledge, you need that sense of unity and alignment, that usually comes through more detailed planning.
It's not one's correct or one's wrong, it just depends on the team composition.
If you have flexibility about when features can be released then you can do agile.
I wish the agile and scrum evangelists were honest about this.
The most agile company I ever worked for never talked about scrum or agile. We just asked customers what they wanted and integrated it into the product. When features were ready we released them. There was no roadmap or timetable. Sometimes customers would ask us when things will be ready, we always pushed back on giving dates.
There is an optimal amount of planning, but knowing it exactly in advance is suboptimal in terms of planning planning cost.