I still find that these are not easy concepts to explain to people outside of software development. Yes, they start to get it after 1 or 2 projects, but I can't help but wonder that there is something more fundamental here.
Something along the lines of, "I know what the finished product is supposed to be, why does it have to be broken down into incremental deliveries?" I think that non-tech people often see it as a kind of watering down of their vision.
At the software company where I work, I have seen the defects and half-baked implementations go by for years. Meanwhile, it's new features that development talks about the most. Recently it dawned on me that they profess to use the "agile" methodology. So now I wonder... Where does maintenance fall in the agile methodology? Am I mistaken that the balance of it's focus is squarely on development rather than sound design / correct implementation?
Maybe the issue is in use of the word "maintenance"?
Sure, it's fixing things that break as things are used, but it's not like it's replacing something that gets worn out with exactly the same thing, or cleaning, or IDK enough about what actually goes on for physical plant maintenance. Things like that do exist - log rotation, restarting processes / servers because of memory leaks that aren't worth tracking down - but, mostly, seems like software "maintenance" is more like post-release iteration cycles.
Some of these are about the pure tech ("here's a consequence to this design when under use that was not anticipated / handled"), others are product design ("turns out it needed to do something a little bit differently") or additional features necessary to make something useful, or just additional features to make it more useful. Or, you know, it was just built wrong the first time.
Seems like all of those situations are actually just more of the same iteration cycles as during development, we just call them "maintenance" for some reason.
(well, except that maybe they needed information and observation that are only available once it's under use...)
Agile software development frequently focuses on delivering a set of minimal features, i.e. an MVP. Too often that approach results in software that's poorly designed and implemented, immediately needing a huge refactor in order to get anywhere near being maintainable.
Agile is about gradual increases of business value. So, as long as maintenance lowers further costs it is a proper aim. Unless there's something else which can bring more value earlier.
For some businesses maintenance time never happens. For some it is an important stage before moving to the next iteration of new features. For some it is a daily process with dedicated teams. Every case is different. Agile is about practical agility :)
IMHO there should be no distinction between maintenance and new development. In agile you don't really have "green field" development, since development is iterative. Except for the first week, you are always modifying an existing running system.
i'll quote the last three sentences of the article:
"In the agile world, success is all about business value - regardless of what was written in a plan months ago. Plans are made, but updated regularly. They guide decisions on what to do next, but are not used as a success measure."
so, answering your question with that in mind, the focus should be on business value. does maintenance provide business value? do new features?
does maintenance provide business value? do new features?
Yes, and yes. In a mature company, maintenance provides more business value than new features. To list some of the maintenance activities:
- incident response. It is not uncommon for a company to have losses in the tons/hour or millions/hour if the primary process has a malfunction.
- organizational changes. No software exists in isolation; changing requirements in a connected component may require changes in your software component too. Developers may think this kind of work is feature development, but from a business point of view, it's simply maintenance. Because of the dependency by other parts of the company, this kind of work usually has stricter deadlines than "new features".
- Predictive maintenance. Identify weak points in the current setup, and isolate of strengthen them before they cause an incident. This includes evaluating technical debt. The value to the business is in risk minimization, and the rationale for this kind of work is predicated on proper risk accounting.
These requirements and processes are part of any proper engineering discipline; these are not marginal issues.
The trick is tempering those values of continuing to make money vs chasing down growth and new sales. With agile you’re always focused on short term. The next few sprints.
I'm just a support person, but it seems to me that it doesn't. At least, not directly. If it did, then development would be focusing on it.
Why not? Because the customer already paid. Because Dev wants to focus on getting new customers instead of catering to the ones who have already forked over the money.
"What do you mean feature X isn't working for the customer? It's working for us in our lab. Anyway, we're all focusing on getting new feature Y working in our lab now." - This is the attitude I am sensing below the surface. It's frustrating.
I've found that when the term "agile" is tossed around to people who aren't familiar with software development, the expectation is usually that more work can be done in less time. Design and requirements gathering is scrapped, and working software is expected after the first sprint.
> It is possible for a mix of waterfall and iterative where early phases (requirements analysis, high level design) are done in a waterfall style while later phases (detailed design, code, test) are done in an iterative manner.
I prefer this approach where time is explicitly dedicated to planning out the solution, which all stakeholders agree to. It also respects that as you begin implementing the solution you'll learn more about the problem domain and adjust accordingly.
If I remember correctly, the agile manifesto was written, in large part, as a reaction to RUP, Spiral and other heavy weight iterative and incremental processes (not waterfall, btw). Its point was to redirect teams back to the prize. In the 90s, teams were becoming too focused on the process instead of the software and customer / user.
I don’t believe I’ve met anyone who ever said, “Hey, let’s do Waterfall!” More often, processes became waterfall-like when feedback loops would become too long. Sometimes, teams wouldn’t get feedback for months. Interestingly, some of the scrum teams I’ve worked with recently have succumbed to the same thing because product owners aren’t active in the process.
But, you’re right - “agile” is often followed as a way to get more done with fewer people. But, unless you’re starting with a heavy weight process to begin with, you’re not going to see much productivity gain. A process isn’t going to miraculously make your developers write code faster. But, used properly, it can lessen the risk of rework.
> In the 90s, teams were becoming too focused on the process instead of the software and customer / user.
Sadly I see a lot of this today too. I know someone who has worked on an "agile" project with two half hour long standups a day, two hour sprint planning, hour long retrospective, and various other "ceremonies". I'm not sure how much of the sprint comprised of actual work.
> Sometimes, teams wouldn’t get feedback for months.
Feedback is so crucial. It frustrates me that business process and politics sometimes gets in the way.
> unless you’re starting with a heavy weight process to begin with, you’re not going to see much productivity gain
Agree 100%, The scenarios I've mentioned have always been short term (~4 weeks) client engagements where there should've been absolutely no room for needless process. I worked on a project once where we had weekly sprints (!) with a playback with key stakeholders each Friday. Each week the stakeholders would change, so each week we needed to spend at least a day planning for the presentation.
I can't imagine why anybody would see RUP as a bad process to follow. Things I like about it are:
1. It's focus on the User and the business case. In the RUP documents you have to describe what business problem you're trying to solve and make a complete list of stakeholders (this can be a wide variety of people, not just the main app user).
2. During the elaboration phase you have to describe the list of requirements and tie them to a business case - so you're forced to think about the problem you're solving.
3. During design phase you have to tie each class/module back to a specific requirement (this forces developers to focus on the real problem as stated)
4. Use cases are a fantastic tools for describing system interactions with the user, and force designers & developers to think about and provide solutions to the not-happy path in a very quick iterative way. In your use cases you should address the main requirements of your stakeholders and tie them back to the requirements doc.
5. RUP has a specific phase for transition to live, and so forces you to think about deployment, architecture, config and maintenance even before code has been written. Yes, your IT dept is a stakeholder.
6. RUP doesn't care what you use during the construction phase - following an agile methodology would be ideal in fact.
7. RUP doesn't care what you use as a project management process - it will have documents that clearly show the scope of your work and the list of prioritized requirements you have to fill before you're done, which is great for tracking progress.
8. RUP is iterative, so if you discovered something important missing during construction - go back to elaboration and work the new requirements through your design.
9. Creating and keeping up to date a fairly small set of well designed documents will lift the quality of your product.
This is just my personal experience, but I found I learned a lot following the methodology and Rational's standard documents. Not only did I produce better solutions, I also found that the documentation could be shared and collaborated on with other people within the organisation, and so Business Analysts, testers and the system managers were all stakeholders and contributors throughout the whole lifetime of the project. They loved that.
> can't imagine why anybody would see RUP as a bad process to follow.
Are you kidding, the 90s were filled with same for and against arguments of RUP as we have about Agile and Scrum now. No process is right for everything so you will have people misuse a tool and blame the tool. The same orgs that micromanage Agile teams would micromanage RUP teams and people will be just as angry about it. RUP is a tool and is as good or bad as the people using the tool.
That said, I agree with you that RUP has some benefits.
Agile is about trusting people, not process. There is no agile process that scales, because agile is all about trust and Dunbar's number puts a cap on how far trust can scale. Let's face it, every medium to large company process is predicated on distrust. The implicit message is: we don't trust you to do good (or any) work, so we're going to make you waste 5 or more hours a week wrestling with our infuriatingly slow Jira installation so our PMs can complain about "velocity." And so on.
In a high trust environment scrum or xp or any other agile process is great. Of course in a high trust environment any process that isn't actively perverse is great. In a low trust environment no process is great, or even good. The best option is less bad.
It looks like distrust, but in fact it is exponential growth in the need of communication and accounting for the risks related to insufficient communication. People are not bad or stupid - they have their own understanding of requirements and make diverse assumptions about the missing knowledge. Also in large organizations most of the projects are not about software, so waterfall is necessary to control the costs - there’s some space for agility, of course, but it exists only in software engineering and not, for example, in supply chain management.
If you want to see a high-trust environment at scale, look at the development model of the Linux kernel. But this model is also inappropriate for many large organizations, for various reasons - hierarchical structure of power, deadlines as requirements, information hiding. As a result, companies trade off programmer's throughput with these things.
Scrum isn't ritualized micromanagement. Some places call their ritualized micromanagement Scrum. Those same places will have micromanagement no matter what process they use. Micromanagement is a people problem. Scrum is a tool that can be used well or poorly by the people that use it, just like any tool.
I'd distrust any software development methodology which does not allocate time for research and prototyping, unless you're doing cookie cutter work that has been done 10000 times before. I have yet to see a complex software project succeed without those two things.
My experience with waterfall vs agile discipline is all about planning and budgeting. Companies that rely on formal project estimations tend to be extremely rigid thinkers while companies that have learned to “trust” their people (and agile) tend to understand the fluid nature of iterative software development.
Getting the budgeting into the hands of the fluid planners (product owners) is critical to successful iterative outcomes.
The author correctly sites construction as an example of how waterfall may have originated. But it was all of engineering that worked that way. It's called project management. I was an automotive engineer when I graduated in '82. Then a plan was built working backwards from when the first car was due to roll off the line, usually about five years in the future. That would set milestones that would be pushed to all the teams and suppliers. We used critical path analysis to determine what activities had to be completed on time or the project would be delayed. Extra resources would be dedicated to tasks on the critical path. Maybe overtime or more people. So it is not surprising that in the early days software projects were treated the same way.
Whenever I took over a project I would tear up my predecessor's plan and immediately start to build a prototype of the component. The plans would have no wiggle room for discovering a big oops years later. Did we choose the right materials? Will that mechanism work in the cold? I would do the first iteration myself, building the product our of wood or cardboard or plastic. Then I would move to a prototype hand crafted out of steel. Then I would order stamping tools made from Kirksite (zink) and test those versions before finalizing the design and committing to the final tooling. So, yes, iterative design/build works for physical systems too.
There is no reason to avoid iterations. And it is quite obvious that in a highly specialized field you want quick iterations to keep all the expensive specialists occupied (in the OPs example, what do the developers do while the analysts analyse?).
_BUT_ you cannot develop software without following the steps of the waterfall model. You need analysis. You need testing. And development has to happen in between.
And here comes the problem: When you have a very fast iterative process, you have everyone working at high pace. But your quality will invariably suffer. If your analyst needs to come up with a decent distributed numerical solver for differential algebraic equations, they won't be done after two weeks. So their 'increment' is going to be incomplete. And no, Pareto does not apply to software specifications. The same holds for any other complex domain. And if your testers only have time to test for two weeks, they cannot cover as much as in half a year.
So yes, iteration is a necessity and quick iteration has its merits. But it also comes with a cost and choosing the right pace is as difficult as it is important.
We have the same challenge, except that in our case the issue is hardware.
Hardware comes at a very fixed iteration speed, usually around 3 months for something of moderate complexity. That's how long it's going to take to do schematic capture, layout, pcb manufacture, board population and bringup. Even if you can speed up this iteration process, there is still a sizeable financial cost of going around the loop.
This means that at a project level you need to do a certain amount of requirements capture before you start work on electronics. You can then run your software in a fairly agile manner 'inside' the project, hoping that the inevitable changes in requirements can be dealt with in software. Obviously you design your system in such a way as to increase that likelihood.
When I was in college in the 90s, they actually taught waterfall as a baseline to compare other methodologies. Agile hadn't appeared yet, but it was obvious then that waterfall's main issue was the inability to change and adapt to a changing business climate. In other words, once you worked out a 3 year dev plan, it was set in stone, even if the business or industry had changed (and needed the software to change).
The alternative was to make shorter iterations. Our professor called this "Cinnamon Bun". Instead of one big waterfall, covering the entire project, you'd break the project up into chunks, but apply the waterfall steps to each chunk. Planning, Analysis, Design, Implementation, Maintenance were the steps. Each chunk would vary, but we typically broke it up into month or two chunks. Of course you'd need planning and analysis up front to figure out the best way to divide the chunks.
When Agile/Scrum came out, its proponents compared it to the shortcomings of Waterfall, which was disingenuous because pure waterfall had been largely abandoned by at least the mid 90s, and probably sooner.
As an aside, One important step in waterfall was Maintenance. This seems missing in current methodologies as a formal step. If you write successful software, the maintenance phase is 80-90% of the softwares lifecycle. Understanding this equates to the understanding that writing maintainable code is one of the most important things a development team can do.
It's a constant pain of mine to chat with friends that work in Enterprise that deny the possibility that ci/cd & agile mindset could work for them.
Usually this is justified by the idea that their org is immutable and therefore everything has to be planned way in advance and in every possible detail.
With that mindset Agile can't actually work, as it requires reflection in the org structure and decision making process.
39 comments
[ 4.0 ms ] story [ 85.5 ms ] threadSomething along the lines of, "I know what the finished product is supposed to be, why does it have to be broken down into incremental deliveries?" I think that non-tech people often see it as a kind of watering down of their vision.
Sure, it's fixing things that break as things are used, but it's not like it's replacing something that gets worn out with exactly the same thing, or cleaning, or IDK enough about what actually goes on for physical plant maintenance. Things like that do exist - log rotation, restarting processes / servers because of memory leaks that aren't worth tracking down - but, mostly, seems like software "maintenance" is more like post-release iteration cycles.
Some of these are about the pure tech ("here's a consequence to this design when under use that was not anticipated / handled"), others are product design ("turns out it needed to do something a little bit differently") or additional features necessary to make something useful, or just additional features to make it more useful. Or, you know, it was just built wrong the first time.
Seems like all of those situations are actually just more of the same iteration cycles as during development, we just call them "maintenance" for some reason.
(well, except that maybe they needed information and observation that are only available once it's under use...)
For some businesses maintenance time never happens. For some it is an important stage before moving to the next iteration of new features. For some it is a daily process with dedicated teams. Every case is different. Agile is about practical agility :)
"In the agile world, success is all about business value - regardless of what was written in a plan months ago. Plans are made, but updated regularly. They guide decisions on what to do next, but are not used as a success measure."
so, answering your question with that in mind, the focus should be on business value. does maintenance provide business value? do new features?
Yes, and yes. In a mature company, maintenance provides more business value than new features. To list some of the maintenance activities:
- incident response. It is not uncommon for a company to have losses in the tons/hour or millions/hour if the primary process has a malfunction.
- organizational changes. No software exists in isolation; changing requirements in a connected component may require changes in your software component too. Developers may think this kind of work is feature development, but from a business point of view, it's simply maintenance. Because of the dependency by other parts of the company, this kind of work usually has stricter deadlines than "new features".
- Predictive maintenance. Identify weak points in the current setup, and isolate of strengthen them before they cause an incident. This includes evaluating technical debt. The value to the business is in risk minimization, and the rationale for this kind of work is predicated on proper risk accounting.
These requirements and processes are part of any proper engineering discipline; these are not marginal issues.
Why not? Because the customer already paid. Because Dev wants to focus on getting new customers instead of catering to the ones who have already forked over the money.
"What do you mean feature X isn't working for the customer? It's working for us in our lab. Anyway, we're all focusing on getting new feature Y working in our lab now." - This is the attitude I am sensing below the surface. It's frustrating.
> It is possible for a mix of waterfall and iterative where early phases (requirements analysis, high level design) are done in a waterfall style while later phases (detailed design, code, test) are done in an iterative manner.
I prefer this approach where time is explicitly dedicated to planning out the solution, which all stakeholders agree to. It also respects that as you begin implementing the solution you'll learn more about the problem domain and adjust accordingly.
I don’t believe I’ve met anyone who ever said, “Hey, let’s do Waterfall!” More often, processes became waterfall-like when feedback loops would become too long. Sometimes, teams wouldn’t get feedback for months. Interestingly, some of the scrum teams I’ve worked with recently have succumbed to the same thing because product owners aren’t active in the process.
But, you’re right - “agile” is often followed as a way to get more done with fewer people. But, unless you’re starting with a heavy weight process to begin with, you’re not going to see much productivity gain. A process isn’t going to miraculously make your developers write code faster. But, used properly, it can lessen the risk of rework.
Sadly I see a lot of this today too. I know someone who has worked on an "agile" project with two half hour long standups a day, two hour sprint planning, hour long retrospective, and various other "ceremonies". I'm not sure how much of the sprint comprised of actual work.
> Sometimes, teams wouldn’t get feedback for months.
Feedback is so crucial. It frustrates me that business process and politics sometimes gets in the way.
> unless you’re starting with a heavy weight process to begin with, you’re not going to see much productivity gain
Agree 100%, The scenarios I've mentioned have always been short term (~4 weeks) client engagements where there should've been absolutely no room for needless process. I worked on a project once where we had weekly sprints (!) with a playback with key stakeholders each Friday. Each week the stakeholders would change, so each week we needed to spend at least a day planning for the presentation.
1. It's focus on the User and the business case. In the RUP documents you have to describe what business problem you're trying to solve and make a complete list of stakeholders (this can be a wide variety of people, not just the main app user).
2. During the elaboration phase you have to describe the list of requirements and tie them to a business case - so you're forced to think about the problem you're solving.
3. During design phase you have to tie each class/module back to a specific requirement (this forces developers to focus on the real problem as stated)
4. Use cases are a fantastic tools for describing system interactions with the user, and force designers & developers to think about and provide solutions to the not-happy path in a very quick iterative way. In your use cases you should address the main requirements of your stakeholders and tie them back to the requirements doc.
5. RUP has a specific phase for transition to live, and so forces you to think about deployment, architecture, config and maintenance even before code has been written. Yes, your IT dept is a stakeholder.
6. RUP doesn't care what you use during the construction phase - following an agile methodology would be ideal in fact.
7. RUP doesn't care what you use as a project management process - it will have documents that clearly show the scope of your work and the list of prioritized requirements you have to fill before you're done, which is great for tracking progress.
8. RUP is iterative, so if you discovered something important missing during construction - go back to elaboration and work the new requirements through your design.
9. Creating and keeping up to date a fairly small set of well designed documents will lift the quality of your product.
This is just my personal experience, but I found I learned a lot following the methodology and Rational's standard documents. Not only did I produce better solutions, I also found that the documentation could be shared and collaborated on with other people within the organisation, and so Business Analysts, testers and the system managers were all stakeholders and contributors throughout the whole lifetime of the project. They loved that.
Are you kidding, the 90s were filled with same for and against arguments of RUP as we have about Agile and Scrum now. No process is right for everything so you will have people misuse a tool and blame the tool. The same orgs that micromanage Agile teams would micromanage RUP teams and people will be just as angry about it. RUP is a tool and is as good or bad as the people using the tool.
That said, I agree with you that RUP has some benefits.
In a high trust environment scrum or xp or any other agile process is great. Of course in a high trust environment any process that isn't actively perverse is great. In a low trust environment no process is great, or even good. The best option is less bad.
Not according to the highly paid consultants with strange acronyms following their names...
Micromanagement looks a lot different to me.
Developers, the customer, management, they all might give different answers.
And as far as I can tell the only true answer is: good communication.
Agile does this with short waterfalls and stand-ups, but other methods also work well as long as there is communication.
And I believe for most part this is where you can distinguish a good managers.
Good managers feel when the developers need someting and feel when the customer want's to go in another direction.
A good manager is a medium, not someone who dictates what the developers should do.
Getting the budgeting into the hands of the fluid planners (product owners) is critical to successful iterative outcomes.
Whenever I took over a project I would tear up my predecessor's plan and immediately start to build a prototype of the component. The plans would have no wiggle room for discovering a big oops years later. Did we choose the right materials? Will that mechanism work in the cold? I would do the first iteration myself, building the product our of wood or cardboard or plastic. Then I would move to a prototype hand crafted out of steel. Then I would order stamping tools made from Kirksite (zink) and test those versions before finalizing the design and committing to the final tooling. So, yes, iterative design/build works for physical systems too.
_BUT_ you cannot develop software without following the steps of the waterfall model. You need analysis. You need testing. And development has to happen in between.
And here comes the problem: When you have a very fast iterative process, you have everyone working at high pace. But your quality will invariably suffer. If your analyst needs to come up with a decent distributed numerical solver for differential algebraic equations, they won't be done after two weeks. So their 'increment' is going to be incomplete. And no, Pareto does not apply to software specifications. The same holds for any other complex domain. And if your testers only have time to test for two weeks, they cannot cover as much as in half a year.
So yes, iteration is a necessity and quick iteration has its merits. But it also comes with a cost and choosing the right pace is as difficult as it is important.
Hardware comes at a very fixed iteration speed, usually around 3 months for something of moderate complexity. That's how long it's going to take to do schematic capture, layout, pcb manufacture, board population and bringup. Even if you can speed up this iteration process, there is still a sizeable financial cost of going around the loop.
This means that at a project level you need to do a certain amount of requirements capture before you start work on electronics. You can then run your software in a fairly agile manner 'inside' the project, hoping that the inevitable changes in requirements can be dealt with in software. Obviously you design your system in such a way as to increase that likelihood.
The alternative was to make shorter iterations. Our professor called this "Cinnamon Bun". Instead of one big waterfall, covering the entire project, you'd break the project up into chunks, but apply the waterfall steps to each chunk. Planning, Analysis, Design, Implementation, Maintenance were the steps. Each chunk would vary, but we typically broke it up into month or two chunks. Of course you'd need planning and analysis up front to figure out the best way to divide the chunks.
When Agile/Scrum came out, its proponents compared it to the shortcomings of Waterfall, which was disingenuous because pure waterfall had been largely abandoned by at least the mid 90s, and probably sooner.
As an aside, One important step in waterfall was Maintenance. This seems missing in current methodologies as a formal step. If you write successful software, the maintenance phase is 80-90% of the softwares lifecycle. Understanding this equates to the understanding that writing maintainable code is one of the most important things a development team can do.
I believe Conway had it right, back in about oh, '67?
"organizations which design systems ... are constrained to produce designs which are copies of the communication structures of these organizations."
— M. Conway
https://en.wikipedia.org/wiki/Conway%27s_law