Actually, it's not the code that directs us, it's the funding. Work in a department that earns its budget annually or are on a project funded upfront? Then no number of scrum masters, backlogs or showcases will see you evolve from waterfall. Likewise if you are funded incrementally per feature set then your team will gravitate towards leaner methodologies. Software manifestos need to be complemented with finance ones for any methodology to have their intended impact.
This articulates something that's been banging around in my head for a few years now. Agile always boils down to a series of waterfalls crashing down on each other, driven by a pool of tickets with no real consideration that some tasks can't be broken down or take more than 2 weeks to complete.
"I'd like to attempt to modernize of a codebase built up over 6+ years, so we can take advantage of technology which didn't exist then." "Yeah, can you break that down for me into tasks that can be done in a single sprint, preferably in parallel?" "Sure, give me a few months." "Yeah, can you break that into 2 week sprints too?"
Even if given the time to create those tasks, they would be put in the backlog behind a pile of feature requests that nobody actually needs. They would then be culled as "Won't Do", since the backlog has gotten too big.
If you can't do your rewrite in an incremental way, it is almost guaranteed to be a failure[1][2]. If you can make it two weeks without making any visible progress, you project is probably not on track for success. Granted, if your management thinks that they can get an entire multi-month project broken down into stories in advance, they have their heads up their asses.
Coming from a mid-size consulting background, all I can say is that agile is absolutely essential, but it's also absolutely worthless if expectations aren't aligned at the start of a project.
Think back to the project management triangle that was probably drawn up in the 1960s. Between scope, time and quality, you can only constrain 2 out of 3. Waterfall constrains scope and time at the expense of quality. Agile constrains quality and time and leaves scope flexible. That is an incredibly tough pill for most businesses to swallow, but it's absolutely in their own best interests. Agile manifesto lauds "working code" and if you build debt, you're probably not producing working code. Quality-first requires flexible scope. If business accepts flexible scope, then agile is miraculous.
These processes are not for developers, they're for business. To give them visibility and skin in the game.
I think this a well made point that many developers fail to appreciate.
Some processes are there to make people productive, prevent them from distractions and help resolve issues...
...but an equally important set of things you’re doing is providing estimates and feedback to the business, so they can make strategic plans and not look stupid to customers.
A lot of process fails (particularly in agile) seem to stem from failing to understand why they’re being done.
Retrospectives aren’t for planning technical debt sessions; they’re there to explain why you didn’t deliver on time. You made some bad estimates? Well, if you don’t go back and review them, guess what, you’ll never get any better at it.
Were those deadlines arbitrary? Maybe, but aren’t you glad you didn’t have to sit in a meeting explaining why things won’t be delivered when you said they would, and oh yes, how stupid you are and so very sorry, its not good enough is it.
Those are meetings no one wants to have.
If you’re forcing your workmates to have them by half heartedly doing a process, your team isn’t doing their job; even if they’re highly productive and produce high quality output.
Scrums aren’t a tech briefing. No one really cares what you did yesterday. Its your chance to ask for help if you need it, and a chance for the team to make sure everyone is on point and not drifting off into Q-space working on something irrelevant.
Basically, whatever the process, people don’t just magically ‘get it’: you have to actually articulate what you’re doing, and who’s benefit it is for, or its a meaningless mess of going through the motions.
Project management is more of a pyramid than a triangle: risk is the fourth variable. There are sometimes circumstances where you do well on the other three variables by accepting a high risk of outright failure, for example by adopting an untried new technology.
Certainly, but there are different kinds of risk. A quality-first approach reduces operational risk significantly. Agile would also tell you if you made a wrong assumption much more quickly. It can't solve timeline risk, but gives you much better visibility. You also decrease market risk by being able to see your product in partially working state much sooner and getting it in front of users (even just test subjects) much quicker.
Author of O'Reilly books Learning Agile and Head First Agile here. I do have some feedback, since you asked. :)
First of all, very interesting piece. Excellent writing – that's really great to see! I think you touched on a lot of really good points.
I've got some feedback about your section on agile. I think it does a good job of summarizing ineffective agile adoptions. But there are a lot of teams that get extremely good results with agile, and I'm not sure your piece does them justice. I've spent the better part of the last decade – and my last two books – helping teams get from ineffective to effective, and I've seen agile work really, really well on a lot of teams.
I do think this is a very accurate picture of agile on a lot of teams, possibly the majority of them:
> The few that really had read up on it seemed incapable of actually dealing with the very real pressures we faced when delivering software to non-sprint-friendly customers, timescales, and blockers. So we carried on delivering software with our specs, and some sprinkling of agile terminology. Meetings were called ‘scrums’ now, but otherwise it felt very similar to what went on before.
But if you ask someone on an effective agile team, they'll generally have no trouble describing exactly what they do.
One problem that plagues ineffective agile teams is agile zealotry, and I think it's really good that you called it out:
> It’s no coincidence that the Agile paradigm has such a quasi-religious hold over a workforce that is immensely fluid and mobile.
This is a real problem with a lot of newly minted agile evangelists. They're excited that they found a new way of working that goes a long way to help with their headaches, and feel like they've found the silver bullet for every team's problems, but take it too far and demand that everyone around them immediately change to the way they think the project should run. Honestly, I've been there myself. We grow out of it, so be patient with us! :)
So while these what you wrote describes the state of agile on a lot of teams, it's not an accurate picture of agile when it's working well. I think Jenny Greene and I did a reasonably coherent job of giving a big-picture view of agile in the first chapter of Learning Agile – if anyone's curious, you can read that chapter for free on Safari: https://www.safaribooksonline.com/library/view/learning-agil...
A few more things.
I don't think it's fair to use an image of Scaled Agile Framework, or SAFe, (https://en.wikipedia.org/wiki/Scaled_agile_framework) as an example of overly complex, "un-agile" agile. For those who don't know, SAFe is a framework for scaling agile to work with multiple teams across an enterprise. Like all tools, it can be used effectively or ineffectively. It has a lot of moving parts because it needs to work with enterprise PMOs, but it's actually built on sound and simple principles. And it really does work when everyone brings the right mindset. And on a team that's working at a company that uses SAFe and working on an effective agile team, day-to-day life definitely does not look like that picture. It looks a lot more simple, like life on any other effective agile team.
In Learning Agile we caution agile coaches that a common warning sign that people on their team are having trouble when they think that agile is the same thing that they've always done, just with a few minor name changes. Here's what we write on page 376:
> “This is exactly like what we’re already doing, just with a different name.”
> This is one of the most common reasons tha...
I've tried or been used to try a dozen or more methodologies over the years, more than I can easily name. Everything from SSADM and ITIL through RUP, various sorts of Agile... all methodologies are complete nonsense with the exception of Kepner-Tregoe.
Lurker MBA student interested in PM roles here. Would be very interested to hear more about what you liked about Kepner-Tregoe, and how you learned it.
Tangentially related, I recommend learning and getting certified in Lean Six Sigma. It's PM * stats * MBA. The systems analysis / theory of constraints perspective also completely explains why agile falls short in the face of external road blocks.
Based on the description at the link, Kepner-Tregoe analysis looks like a useful and light framework for eliciting and applying critical thinking in project management. But the key words are "critical thinking." The principles of the agile manifesto when put in practice by people who have domain knowledge and who have aptitude for critical thinking confer on a project a good chance of success.
Those kinds of projects and teams have that edge over others because they have critical thinking on their side. That will benefit them first in seeing that they don't need to add a large, trademarked, expensively trained process around agile, and subsequently in remaining focused on people and code rather than process.
I like KT because it rapidly eliminates ambiguity and creates packages of information that can be passed seamlessly between teams or stages in a workflow. There's little or no rework in a KT shop, and few "pipeline stalls" where someone is blocked because a decision hasn't been made or communicated, or because a dependency is discovered too late. I learned it at a company I was at about 10 years ago, there was one guy who had used it in his previous job and he convinced senior management, and so we were all sent to KT for their training course.
Like I say I've tried 'em all, well nearly, KT is the only one that's worth a damn. Even the simple exercises like "is, is-not" make you feel like Poirot.
unified process is not complete non sense. It's the best I found so far and maybe the only methodology that covers projects longer than a few weeks and from start to finish.
The commercial IBM version (RUP: Rational Unified Process) with the associated tool (rationals TM) and consulting service is really horrible though.
I found its practical blending of UP with concepts traditionally associated with other processes quite illuminating.
For example, here is a snippet from section 1.1:
Iterative Development, Agile Modeling, and an Agile UP
Given many possible activities from requirements through to implementation, how should a developer or team proceed? Requirements analysis and OOA/D needs to be presented and practiced in the context of some development process. In this case, an agile (light, flexible) approach to the well-known Unified Process (UP) is used as the sample iterative development process within which these topics are introduced. However, the analysis and design topics that are covered are common to many approaches, and learning them in the context of an agile UP does not invalidate their applicability to other methods, such as Scrum, Feature-Driven Development, Lean Development, Crystal Methods, and so on.
I have seen parts of this in other training (six sigma, tqm, etc). But there is something very important in dedicating the time to think outside the most obvious immediate problem, which we often need to be reminded of by process.This was the link I found, but honestly the images in the Google search were better.
Nice article! I'd like to a software methodology more supportive of what I'd call "Continuous refactoring" or a more "kaizen"-like mindset. "Agile" isn't always as agile as I'd like, but becomes rigid in peoples interpretation of the rules.
It's a constant fight to keep down the number of lines of code and complexity, in large projects. Once a code-base passes beyond a certain point it becomes difficult to see redundancies and patterns. Bloated code breeds bloated code.
If you on the other hand keep trimming and keep refactoring deep and wide as the code and requirements change you can often keep things manageable and understandable. "bad code" get's easier to spot, it's a little bit like the broken windows theory.
Scrum IMO and a lot of the agile methodologies, at least in their implementation, in my experience, tends to treat projects like brick-building and lacks a way to support and enforce constant re-architecturing and removal of code smells.
If code quality and refactoring is not supported by the actual methodology it can become an bureaucratic problem where product owners doesn't see the value and doesn't prioritize it (even though it leads to quicker development time and less bugs in the long run)
Unit testing is supposed to make refactoring easier, but can become an impediment to refactoring as well, as the cost of rewriting tests can be seen as too high.
We're currently experimenting with having one hour meeting each week where you can discuss things like that, and then dedicating some time to refactor, we'll see how it works out :)
That sounds like the 'healthy' team I talk about in the article. The problem with the 'continuous refactoring' (in my experience) is that the people that hold the purse strings cannot grasp the benefit, so cut this early.
Sometimes they're right. I'd prefer a methodology where the people who hold the purse strings have to explicitly declare their current desired emphasis on dev velocity / quality.
At the moment every single methodology requires devs make that decision even though it's really a bizdev decision.
I don't know, when it comes to code quality and things like that I'd rather prefer a methodology that insulates it against bureaucracy rather than involves it. Keep the Dilbertian forces at bay.
Product owners should prioritize feature set, requirements and sometimes bugs, but the less they have to do with the intrinsics of reaching them the better.
I think their involvement should be strictly limited to telling developers what % of their time they're to spend on refactoring/tooling.
This actually avoids dilbertian forces - the PM can't weasel out of responsibility for pressuring you to deliver shit quickly. That % would stick out like a sore thumb if it was permanently under, say, 15%.
Unfortunately IMO that kind of thinking seldom works. It's very managerial to assign specific time and percentages to things but in practice I've never known development to be managed that way. Also you underestimate their weaselness :)
Refactoring needs to be an integral and constant part of the development, in the long run that will decrease time needed for refactoring, and time required to fix bugs. But that time is variable, and might come in large chunks depending on how much technical debt you've accumulated
Precision isn't the issue. Story pointing isn't supposed to be 100% accurate either. It's supposed to shine a light on an otherwise opaque process. It works at that. This is the same.
I could quite easily spend all of my time refactoring and none on features and vice versa. There is no "right" amount as you seem to think, just a whole bunch of trade offs.
I think you underestimate the power of making something measurable and exposing it to upper management. That often changes behaviour in a quite radical way, especially among weasels.
And, you've never known development to be managed in this way because it's a new idea.
Yeah totally, but if was intrinsic to a trendy methodology you might be able to shield it better in the same way you do stuff in the name of being ISO-whatever compliant :)
In my experience, the easiest way to get 'continuous refactoring' done is to build it into all estimates without telling whoever controls the 'purse strings' ....
yeah, that's what's often is being done. But it prevents the methodology to have some useful best practices built into it, and a methodology could be a force in improving code quality and keeping it intact over time, which is from all the projects I've seen severely needed
The problem with refactoring is that developers think it's a task of it's own. They want time to do it, they want special estimate, they want it to become a thing of its own.
Refactoring is low level work that is part of every task, it should not even be mentioned.
When you ask your mechanic to change wheels, you don't expect him to explain that he will have to take off the previous wheel and change the screws, that will be an item on the bill consuming half of the budget.
Some developers think that, and unfortunately a lot of methodology enforces that thinking.
Cleaning up code as you go should be a habit I agree. It's slightly trickier when code reaches a "boiling frog" point and you need to take a look at the bigger picture and perhaps re-architecture. It often easy to keep chugging away at the tree's and not seeing the forest.
The downside with the scope of those kind of structural refactoring is that they touch a lot of code, so they're perceived as "risky".
Also developers get attached to the architecture they have learnt and know, and don't want to re-learn, even if the simplified code would be easier to learn for someone new
Not all refactoring fit neatly alongside existing work.
Sometimes you get a small change request that suggests a broader refactoring. You're pretty sure it will be valuable in future, but it would be a lot more work and isn't that useful in the short term. So if you try and do that refactoring upfront, you get pushback for wasting time on a small change. And then again on the next relevant change.
Developers often want to schedule these kinds of refactorings, because otherwise they get pushback on every chance they might otherwise have to fix these problems.
Not all refactoring fit neatly alongside existing work... because every refactoring fits nearty alongside the new work.
In your example, you don't refactor for fun, you refactor to integrate new things. Integrate the refactoring as part of the new work.
Guess what. The mechanic too has to deal with the old wheels, and don't get him started on the old screws that broke and the assembly that had to be cleaned just to work in decent conditions.
It often is a task of it's own. It's not always "low-level work", sometimes it's directly replacing key components in a huge chain of events that all need to be modified to accept your refactor. It really depends on the codebase, features you're developing, nature of the business engagement and so on. There is also risk in a refactor, especially if it's heavy in business logic.
If I'm doing a bugfix task on a legacy site, I might note that it would be good to spend ~4h on a refactor, when the bug may have taken 0.25h to fix. In that scenario the benefits for refactoring are low, so we don't do it.
Alternatively, if I'm doing a 6h feature, and understanding and working with the bad code would take 6h, versus spending 4-6h on a wholesale rewrite, of course I would just slip the refactor inside the original task.
You have to keep your expectations realistic and your project managers informed. If your refactor suddenly blows out in time because you didn't fully comprehend your problem space, then you both haven't completed the feature or achieved the refactor. Maybe in your business a day extra is fine, in some businesses running over by a couple of hours can mean the ROI doesn't make sense on that task anymore.
The key is to be sensible and pragmatic. There is no one solution to all thinking. If there's anything any of these methodologies are trying to do, it's provide a framework for quickly communicating and reacting to the nuance of software development.
>>> It often is a task of it's own. It's not always "low-level work", sometimes it's directly replacing key components in a huge chain of events that all need to be modified to accept your refactor. It really depends on the codebase, features you're developing, nature of the business engagement and so on. There is also risk in a refactor, especially if it's heavy in business logic.
And that's why refactoring has a terrible reputation and developers can't be trusted. Now you're talking about replacing key components and altering the business logic under the pretext of refactoring.
This should NOT be called refactoring, this should be called replacing key components and revisiting the business logic.
Of course, management doesn't want that to be done without justification.
Refactoring may change how business logic works internally without changing anything in input/output or features. If you need to change how key components are implemented, without changing what they do, then you are refactoring them. Changes may be big internally, but it is still called refactoring if features don't change.
If your code developed a "god class" that does everything and should be ripped apart there isn't much you can do about it except changing core parts of your project.
I don't mean changing business logic, refactors have to retain business logic still, and the risk is that the developer gets that wrong.
It sounds like you're not getting good communication from your developers or you're misunderstanding them when they say they need to refactor. Refactor is still the right word for it, but you need to find a way to communicate the extent of the refactor.
" The problem with refactoring is that developers think it's a task of it's own. They want time to do it, they want special estimate, they want it to become a thing of its own."
I always tell people to constantly refactor on a smaller scale and not ask management. It's just part of the job. A lot of the "big" refactors end up with just another set of problems but don't really make things better.
If you limit yourself to refactoring within the scope of the task at hand, then you end up with a mess after a certain size of software project. In my opinion, that is why software leads will normally at some point steer the success of future development by refactoring project wide, when and where his/her team can. It's also necessary for performance reasons sometimes.
The Agile methodology is to argue about the Agile methodology.
PMI style critical path, planning backwards from the release, is the only approach I’ve ever seen that was clear, concrete, actionable, high functioning.
Didn't see it that time. Found it interesting this time.
Hacker News Guidelines What to Submit
On-Topic: Anything that good hackers would find interesting. That includes more than hacking and startups. If you had to reduce it to a sentence, the answer might be: anything that gratifies one's intellectual curiosity.
We treat stories as dupes if they've had significant attention in the last year or so. If not, reposts are ok. This is in the FAQ: https://news.ycombinator.com/newsfaq.html.
Some such practice follows from the core value of the site, because intellectual curiosity withers under repetition. On the internet, most attention tends to converge, power-law style, on the same small number of hot topics. On HN we actively try to break that by welcoming the long tail. Because there are so few slots on the front page we tend to be diligent about marking dupes, even though they're often excellent stories that much of the community hasn't seen.
We don't, though, mark a story as a dupe if it was posted before but didn't get much attention. It's important to let good stories have multiple cracks at the bat, in order to mitigate the randomness of what gets noticed here.
And there you have pretty much the whole of our thinking about dupes.
It's important for me to separate the principles behind these methodologies, from the industries that spawn up around them to sell guides and certifications and books and conference tickets.
Adhering to the ideas of clear goal-setting, team communication, autonomy, and iterative development have never had bad outcomes for me. But formalising them into a ruleset for any arbitrary team doesn't capture everything.
I do agree with you, but saying "I adhere to the ideas of clear goal-setting, team communication, autonomy, and iterative development" doesn't lead you anywhere: you need tools and actual processes to convert this adhesion to an idea into the materialization of this idea.
And that's where those companies intervene: they build an implementation of those ideas. The implementation is not adapted for any project and team and it is lossy regarding the original principles, but it has the advantage of having being seriously thought about, tested and scalable.
Companies really really like the words PMO and portfolio management. That's why Scaled Agile Framework(SAFe) is that nasty thing you see in that picture.
"People will benefit from SAFe, but not nearly as much as they would from a true agile approach"
IMHO all those methodologies that try to somehow shoehorn classical project management into new things are at least to some extent the acknowledgement that things aren't working as expected.
In all the years I've been doing this the only times I've felt things really gelled and were firing on all cylinders had little to do with methodology and everything to do with culture and motivation. When people felt cohesive, and they were all working towards the same goal, and that their personal priorities and the project priorities could align.
I do think that the 'agile' methodology focus on short achievable iterations and daily stand up meetings yielded good results _in that context_. Disconnected from a healthy organization or transplanted into a bad one, it just yields epic fail and falls apart.
Another thing that I saw work well was embedding product management, testing, and operations people into small per-project teams with developers, which I found reduced conflict from territoriality and reduced "throwing over the wall" syndrome.
In my soon-to-be 15-year career as a professional software engineer (and several more as an amateur) I've noticed that the common trait of the high functioning teams I have had the distinct pleasure to work with was always the same thing – a great leader. (In one case, there were two, working sort of in tandem.)
Trying to sum up these leaders' qualities is difficult, but if I try to distill it I think it comes down to primarily these two aspects:
1. Strong vision and ability to clearly communicate direction
2. Mutual trust
They wouldn't tell people what to do, and certainly not how to do it, but they'd always – every single day – point in the direction they wanted you to go and course correct where necessary. It wasn't like it was an all-hands meeting every morning either, it was more like gentle nudging or little reminders, but not unsolicited. It's difficult to explain frankly, but the result was that we as a team never felt unsure as to where we were heading.
The most important thing however certainly was the mutual trust. You could trust these leaders to have your backs, no matter what, and these people would trust you to get things done. That sense of trust created an immensely palpable sense of loyalty, you really did feel like your input and efforts mattered, and you always wanted to bring your a-game for the team. Again, difficult to explain, but you know when you're in it.
Not one of these teams worked like the other, in terms of methodology, yet things worked and they worked very well. We had issues, for sure, but I've also worked in dysfunctional teams and the same kind of issues in those teams were amplified by terrible leadership, leading to constantly worsening morale and in some unfortunate cases abject failure to deliver.
I find it somewhat amusing that this article doesn't pick up on the fact that it is furthering the root cause of this very issue in the act of basing it on the book Sapiens. A core problem with these mythological trends is that they reduce world views down to easy-to-communicate little stories. This is useful for propagating ideas from person to person, but it necessarily has to be pithy and avoid too much detail. What we aught to do is to notice this historical pattern, and to take a stand against it. We should try to not be too persuaded by the book, or methodology, of the month (Sapiens right now) but to try to go back to the hard, tough, classics. We should try to build a foundation based on actual psycological and sociological research, instead of being steered too much by the, understandably attractive, mimetic pulls of water cooler philosophy.
I used to think in terms of stories, but then I heard Tyler Cowen tell me a story about the dangers of stories, and now I don't think in terms of stories so much. (My story of personal growth.)
"they reduce world views down to easy-to-communicate little stories."
Yes, exactly. That's what you want. Make it too complicated and you lose people in the details. The content of the fiction you choose to operate under isn't as important as the collaborative power it unleashes.
I'm not sure that trying to fight human nature is going to be a winning strategy. Stories may be oversimplifying, but it is exactly that strategic oversimplification that is important to many endeavors.
I am ambivalent about "the difficulty lies not in defining [useful software methodologies], but in convincing others to follow [them]."
Currently, I see too much faith being put in methods of dubious utility. I see techniques, that may be effective when applied judiciously in some circumstances, being promoted to universal principles. In particular, I see the failure of waterfall being taken to mean that any attempt to think ahead is counterproductive.
On the other hand, ten years before the scope of this article, Barry Boehm took on the waterfall model with an iterative one that emphasized frequent reevaluation of where you are, where you are going, and what impediments you face [1]. The fact that this article begins with waterfall supports the notion that adoption is the problem. Perhaps Boehm's model was largely overlooked precisely because it was not dogmatic enough to promote zealotry?
Very true. I've found at every company that I've worked at that a "bad" process followed consistently by everyone always outperforms the "perfect" process followed haphazardly.
This reminds me of the recent threads about people using Excel instead of "real" software. At the end of the day, you are trying to add value. As long as you aren't adding technical debt or other risks, the how matters much less.
Methodologies are just virtue signals. Project management is just the self soothing that people do to pretend we’re not all just a bunch of howler monkeys fighting over whatever kibble is left over after the job creators take their share.
‘The test of a first-rate intelligence is the ability to hold two opposed ideas in mind at the same time and still retain the ability to function.’ F. Scott Fitzgerald
I know I am taking this out of context, but this is best thing I read on the internet since a while.
My approach is "Risk Driven Development" - find the most likely causes of failure for your project, and structure your methodology around reducing those.
The standard lean startup risk is building something the customer doesn't want, so it makes sense that the way to mitigate that is by getting customer feedback quickly.
But other projects can have substantially different risks - for example, one project I worked on involved building a pipeline to process a large amount of data every day. There was no real customer risk, the spec was pretty much "do this thing that's valuable on some data, but for 1,000x more." Instead, the major uncertainty was whether the libraries we considered would scale up. To mitigate this, we built small prototype projects with each library on production workloads, and found something like 7 out of 12 of them would work with minor adjustments, while 5 had to be discarded pretty much entirely. We then wrote the project with alternatives for those 5, or writing replacements for the functionality we needed from scratch.
As teams grow, coordination becomes a bigger risk. One way to reduce this is by having interfaces between modules, and enforcing those interfaces in code. My preferred method is static types and functions, but of course there are others like JSON APIs.
Many companies I know try to use Agile/Scrum for data pipelines, and focus heavily on getting a customer-ready prototype. This is then often really difficult to scale, because bad architectural assumptions have been baked into the project.
In general I see most companies/teams picking on methodology and staying with it, rather than adapting to the project.
Okay. I'm not a zealot about these things, but I think the author underappreciates the nicer aspects of agile. My favorite book on agile looks at it from a mathematical / risk management / process model standpoint.
The concepts used in the book - batch size, uncertainty, feedback, control, queue size and latency - these really are useful things to think about when scheduling work.
I second your recommendation. Any product manager or engineering manager who hasn't read that book is probably working on the basis of many flawed assumptions and optimizing for the wrong metrics.
He freely admits he has never read anything about any of this or made a serious study of it, or had to implement any of it. It is more of a perspective or list of observations from life in the trenches than a serious study or even a very informed position. The whole list of methodologies even mentioned in the article is just three: waterfall, hacking, agile.
That said, it seems the author is making a larger point than the efficacy of any individual methodology: Convincing people to co-operate within any reasonable methodology/framework has the larger effect than effects of different methodologies. That having a religion/philosophy is more important than which religion/philosophy in terms of national co-operation, etcetera.
If at any job you are using some methodology "exactly as it was written" (orthodox-style), without adapting it to the needs and vagaries of your company, your team/org/company is being run by someone who has probably never done the job themselves. I would advise polishing the resume and shopping it around.
If you use a software development methodology for something that isn't software development, you should again probably find a different company to work for, because things aren't going to end well there.
If every single project and task you work on uses only one methodology, the people in charge are either morons or full of shit. (Or you are actually a robot in a factory, in which case, hello, I appreciate your intelligence, please do not harm me)
Perhaps, but on the other hand too many managers wrongly think that their organizations are unique and special snowflakes. Sometimes plain vanilla is the best flavor, and many organizations would get better results by implementing a modern methodology exactly as written rather than wasting a bunch of time customizing it.
The whole point of project management is using the right tool for the job. Sometimes that’s waterfall, sometimes it’s agile but most of the times it’s a mix.
There is a tendency in Software Development, that terms lose their meaning the more they are used. I think it is partly due to software people not talking concrete, physical objects that much, but more about abstract ideas with more or less relevance to reality.
This can lead to interesting effects. For example, I have heard "DRY - Don't repeat yourself" most times I encountered it as a reasoning for obsessive refactorings, where functions where extracted from code without any regards to abstraction layer just because a few lines of code looked the same (or could be rewritten to look the same). But originally the term was coined to represent the principle of "Every piece of knowledge must have a single, unambiguous, authoritative representation within a system". "DRY" was however so catchy, that the slogan propagates without the explanation of the principle.
I think agile suffers a similar fate. Countless consultants, managers and engineers have rendered the word almost meaningless. Even with the Agile manifesto at hand, it is sometimes hard to free agile from the multiple layers of paint spread over it.
DevOps suffers the same fate, and while I haven't researched this, I assume that even waterfall is probably a gross misrepresentation of the original ideas.
I think DRY is actually an axiomatic principle of good code. However, there are several axioms and they all conflict, so maximizing one at the expense of the others means you are writing bad code.
Do we have the same understanding of code being DRY? Because it is not an axiom. It is a guideline on reducing errors, that are enabled by writing the same information in various places of your code. As opposed to "reduce code duplication at all costs" which does emphasize textual/structural equivalence.
When I said there are several axiomatic principles and they conflict that's ^^^ precisely what I was talking about. Reducing duplication at all costs is harmful.
No, I do mean is that properties of good code can all be boiled down to a set of fundamental principles.
I think one of the defining features of all of those principles is that they directly conflict with one another. Good code means achieving a local minima, because up to a point, maximizing any one of these principles (e.g
loose coupling) will inevitably violate one or more of the other fundamental principles.
I.e. maximizing loose coupling often has the side effect of violating DRY or ramping up language complexity (another fundamental principle).
I don't think you really can derive DRY from other principles, though I'd be curious to hear what you think it could be derived from.
Nice summary especially about the real intent of DRY and how people implement it. I think what it comes down to is that software development is very complex so any attempt to produce cookie cutter rules will fail eventually. OOP is a good idea, Agile is a good idea, DRY is a good idea but you still need the judgement to see how it applies to your specific situation.
I started doing this in the 90s and I feel that there is a growing trend to replace judgement with process and strict rules.
> I started doing this in the 90s and I feel that there is a growing trend to replace judgement with process and strict rules.
Hmm, I am not quite sure whether I share your sentiment on process and strict rules. I don't mind process if it is actually one that is sensible and leaves freedom to the individual.
For example, in my experience as a developer, Scrum can drastrically improve developer-management communication. Story point estimation with estimation cards has always revealed stories with unclear scope or requirements. This requires however, that both sides actively try to improve that process.
I have often heard of people who said "Scrum doesn't work" who actually were really running a very weird subset of it (for example burndown charts and velocities, but no proper story point estimation), etc.
A key point comes from the linked post entitled, "Why don't software methodologies work?" People are the greatest factor in a software project's success, not tools or methodology. Good tools and methodology are part of success, but they aren't sufficient for it.
109 comments
[ 3.3 ms ] story [ 177 ms ] threadPersonally, I did not know this manifesto before.
Nowadays I always refer to agile as mini-waterfall, because that is how it gets actually done in most companies, with the same kind of SLAs.
This articulates something that's been banging around in my head for a few years now. Agile always boils down to a series of waterfalls crashing down on each other, driven by a pool of tickets with no real consideration that some tasks can't be broken down or take more than 2 weeks to complete.
"I'd like to attempt to modernize of a codebase built up over 6+ years, so we can take advantage of technology which didn't exist then." "Yeah, can you break that down for me into tasks that can be done in a single sprint, preferably in parallel?" "Sure, give me a few months." "Yeah, can you break that into 2 week sprints too?"
Even if given the time to create those tasks, they would be put in the backlog behind a pile of feature requests that nobody actually needs. They would then be culled as "Won't Do", since the backlog has gotten too big.
</rant>
[1] https://www.joelonsoftware.com/2000/04/06/things-you-should-...
[2] https://codeahoy.com/2016/04/21/when-to-rewrite-from-scratch...
Think back to the project management triangle that was probably drawn up in the 1960s. Between scope, time and quality, you can only constrain 2 out of 3. Waterfall constrains scope and time at the expense of quality. Agile constrains quality and time and leaves scope flexible. That is an incredibly tough pill for most businesses to swallow, but it's absolutely in their own best interests. Agile manifesto lauds "working code" and if you build debt, you're probably not producing working code. Quality-first requires flexible scope. If business accepts flexible scope, then agile is miraculous.
These processes are not for developers, they're for business. To give them visibility and skin in the game.
Some processes are there to make people productive, prevent them from distractions and help resolve issues...
...but an equally important set of things you’re doing is providing estimates and feedback to the business, so they can make strategic plans and not look stupid to customers.
A lot of process fails (particularly in agile) seem to stem from failing to understand why they’re being done.
Retrospectives aren’t for planning technical debt sessions; they’re there to explain why you didn’t deliver on time. You made some bad estimates? Well, if you don’t go back and review them, guess what, you’ll never get any better at it.
Were those deadlines arbitrary? Maybe, but aren’t you glad you didn’t have to sit in a meeting explaining why things won’t be delivered when you said they would, and oh yes, how stupid you are and so very sorry, its not good enough is it.
Those are meetings no one wants to have.
If you’re forcing your workmates to have them by half heartedly doing a process, your team isn’t doing their job; even if they’re highly productive and produce high quality output.
Scrums aren’t a tech briefing. No one really cares what you did yesterday. Its your chance to ask for help if you need it, and a chance for the team to make sure everyone is on point and not drifting off into Q-space working on something irrelevant.
Basically, whatever the process, people don’t just magically ‘get it’: you have to actually articulate what you’re doing, and who’s benefit it is for, or its a meaningless mess of going through the motions.
First of all, very interesting piece. Excellent writing – that's really great to see! I think you touched on a lot of really good points.
I've got some feedback about your section on agile. I think it does a good job of summarizing ineffective agile adoptions. But there are a lot of teams that get extremely good results with agile, and I'm not sure your piece does them justice. I've spent the better part of the last decade – and my last two books – helping teams get from ineffective to effective, and I've seen agile work really, really well on a lot of teams.
I do think this is a very accurate picture of agile on a lot of teams, possibly the majority of them:
> The few that really had read up on it seemed incapable of actually dealing with the very real pressures we faced when delivering software to non-sprint-friendly customers, timescales, and blockers. So we carried on delivering software with our specs, and some sprinkling of agile terminology. Meetings were called ‘scrums’ now, but otherwise it felt very similar to what went on before.
But if you ask someone on an effective agile team, they'll generally have no trouble describing exactly what they do.
One problem that plagues ineffective agile teams is agile zealotry, and I think it's really good that you called it out:
> It’s no coincidence that the Agile paradigm has such a quasi-religious hold over a workforce that is immensely fluid and mobile.
This is a real problem with a lot of newly minted agile evangelists. They're excited that they found a new way of working that goes a long way to help with their headaches, and feel like they've found the silver bullet for every team's problems, but take it too far and demand that everyone around them immediately change to the way they think the project should run. Honestly, I've been there myself. We grow out of it, so be patient with us! :)
So while these what you wrote describes the state of agile on a lot of teams, it's not an accurate picture of agile when it's working well. I think Jenny Greene and I did a reasonably coherent job of giving a big-picture view of agile in the first chapter of Learning Agile – if anyone's curious, you can read that chapter for free on Safari: https://www.safaribooksonline.com/library/view/learning-agil...
A few more things.
I don't think it's fair to use an image of Scaled Agile Framework, or SAFe, (https://en.wikipedia.org/wiki/Scaled_agile_framework) as an example of overly complex, "un-agile" agile. For those who don't know, SAFe is a framework for scaling agile to work with multiple teams across an enterprise. Like all tools, it can be used effectively or ineffectively. It has a lot of moving parts because it needs to work with enterprise PMOs, but it's actually built on sound and simple principles. And it really does work when everyone brings the right mindset. And on a team that's working at a company that uses SAFe and working on an effective agile team, day-to-day life definitely does not look like that picture. It looks a lot more simple, like life on any other effective agile team.
In Learning Agile we caution agile coaches that a common warning sign that people on their team are having trouble when they think that agile is the same thing that they've always done, just with a few minor name changes. Here's what we write on page 376:
> “This is exactly like what we’re already doing, just with a different name.” > This is one of the most common reasons tha...
I have no affiliation with Kepner-Tregoe the company, nor is that an affiliate link...
Those kinds of projects and teams have that edge over others because they have critical thinking on their side. That will benefit them first in seeing that they don't need to add a large, trademarked, expensively trained process around agile, and subsequently in remaining focused on people and code rather than process.
If you want to get a flavour of it the original book is pretty cheap now https://www.amazon.co.uk/Rational-Manager-Charles-Higgins-Ke...
Like I say I've tried 'em all, well nearly, KT is the only one that's worth a damn. Even the simple exercises like "is, is-not" make you feel like Poirot.
The commercial IBM version (RUP: Rational Unified Process) with the associated tool (rationals TM) and consulting service is really horrible though.
http://www.craiglarman.com/wiki/index.php?title=Book_Applyin...
I found its practical blending of UP with concepts traditionally associated with other processes quite illuminating.
For example, here is a snippet from section 1.1:
Iterative Development, Agile Modeling, and an Agile UP
Given many possible activities from requirements through to implementation, how should a developer or team proceed? Requirements analysis and OOA/D needs to be presented and practiced in the context of some development process. In this case, an agile (light, flexible) approach to the well-known Unified Process (UP) is used as the sample iterative development process within which these topics are introduced. However, the analysis and design topics that are covered are common to many approaches, and learning them in the context of an agile UP does not invalidate their applicability to other methods, such as Scrum, Feature-Driven Development, Lean Development, Crystal Methods, and so on.
http://begrip.be/kepner-tregoe-analysis/
It's a constant fight to keep down the number of lines of code and complexity, in large projects. Once a code-base passes beyond a certain point it becomes difficult to see redundancies and patterns. Bloated code breeds bloated code.
If you on the other hand keep trimming and keep refactoring deep and wide as the code and requirements change you can often keep things manageable and understandable. "bad code" get's easier to spot, it's a little bit like the broken windows theory.
Scrum IMO and a lot of the agile methodologies, at least in their implementation, in my experience, tends to treat projects like brick-building and lacks a way to support and enforce constant re-architecturing and removal of code smells.
If code quality and refactoring is not supported by the actual methodology it can become an bureaucratic problem where product owners doesn't see the value and doesn't prioritize it (even though it leads to quicker development time and less bugs in the long run)
Unit testing is supposed to make refactoring easier, but can become an impediment to refactoring as well, as the cost of rewriting tests can be seen as too high.
We're currently experimenting with having one hour meeting each week where you can discuss things like that, and then dedicating some time to refactor, we'll see how it works out :)
At the moment every single methodology requires devs make that decision even though it's really a bizdev decision.
Product owners should prioritize feature set, requirements and sometimes bugs, but the less they have to do with the intrinsics of reaching them the better.
This actually avoids dilbertian forces - the PM can't weasel out of responsibility for pressuring you to deliver shit quickly. That % would stick out like a sore thumb if it was permanently under, say, 15%.
Refactoring needs to be an integral and constant part of the development, in the long run that will decrease time needed for refactoring, and time required to fix bugs. But that time is variable, and might come in large chunks depending on how much technical debt you've accumulated
I could quite easily spend all of my time refactoring and none on features and vice versa. There is no "right" amount as you seem to think, just a whole bunch of trade offs.
I think you underestimate the power of making something measurable and exposing it to upper management. That often changes behaviour in a quite radical way, especially among weasels.
And, you've never known development to be managed in this way because it's a new idea.
[1] http://www.extremeprogramming.org/rules/refactor.html
Refactoring is low level work that is part of every task, it should not even be mentioned.
When you ask your mechanic to change wheels, you don't expect him to explain that he will have to take off the previous wheel and change the screws, that will be an item on the bill consuming half of the budget.
Cleaning up code as you go should be a habit I agree. It's slightly trickier when code reaches a "boiling frog" point and you need to take a look at the bigger picture and perhaps re-architecture. It often easy to keep chugging away at the tree's and not seeing the forest.
The downside with the scope of those kind of structural refactoring is that they touch a lot of code, so they're perceived as "risky".
Also developers get attached to the architecture they have learnt and know, and don't want to re-learn, even if the simplified code would be easier to learn for someone new
Sometimes you get a small change request that suggests a broader refactoring. You're pretty sure it will be valuable in future, but it would be a lot more work and isn't that useful in the short term. So if you try and do that refactoring upfront, you get pushback for wasting time on a small change. And then again on the next relevant change.
Developers often want to schedule these kinds of refactorings, because otherwise they get pushback on every chance they might otherwise have to fix these problems.
In your example, you don't refactor for fun, you refactor to integrate new things. Integrate the refactoring as part of the new work.
Guess what. The mechanic too has to deal with the old wheels, and don't get him started on the old screws that broke and the assembly that had to be cleaned just to work in decent conditions.
If I'm doing a bugfix task on a legacy site, I might note that it would be good to spend ~4h on a refactor, when the bug may have taken 0.25h to fix. In that scenario the benefits for refactoring are low, so we don't do it.
Alternatively, if I'm doing a 6h feature, and understanding and working with the bad code would take 6h, versus spending 4-6h on a wholesale rewrite, of course I would just slip the refactor inside the original task.
You have to keep your expectations realistic and your project managers informed. If your refactor suddenly blows out in time because you didn't fully comprehend your problem space, then you both haven't completed the feature or achieved the refactor. Maybe in your business a day extra is fine, in some businesses running over by a couple of hours can mean the ROI doesn't make sense on that task anymore.
The key is to be sensible and pragmatic. There is no one solution to all thinking. If there's anything any of these methodologies are trying to do, it's provide a framework for quickly communicating and reacting to the nuance of software development.
And that's why refactoring has a terrible reputation and developers can't be trusted. Now you're talking about replacing key components and altering the business logic under the pretext of refactoring.
This should NOT be called refactoring, this should be called replacing key components and revisiting the business logic.
Of course, management doesn't want that to be done without justification.
It sounds like you're not getting good communication from your developers or you're misunderstanding them when they say they need to refactor. Refactor is still the right word for it, but you need to find a way to communicate the extent of the refactor.
I always tell people to constantly refactor on a smaller scale and not ask management. It's just part of the job. A lot of the "big" refactors end up with just another set of problems but don't really make things better.
But I could be wrong too. I’m starting to think “devops” just means tech support with continuous rework.
Well, I think it's nothing more than giving the ops access and responsibilities to the same devs that created the mess to start with.
I think a lot of this is because, if you don't enforce the rules rigidly, they tend to get cast by the wayside, especially by management.
PMI style critical path, planning backwards from the release, is the only approach I’ve ever seen that was clear, concrete, actionable, high functioning.
Hacker News Guidelines What to Submit
On-Topic: Anything that good hackers would find interesting. That includes more than hacking and startups. If you had to reduce it to a sentence, the answer might be: anything that gratifies one's intellectual curiosity.
Some such practice follows from the core value of the site, because intellectual curiosity withers under repetition. On the internet, most attention tends to converge, power-law style, on the same small number of hot topics. On HN we actively try to break that by welcoming the long tail. Because there are so few slots on the front page we tend to be diligent about marking dupes, even though they're often excellent stories that much of the community hasn't seen.
We don't, though, mark a story as a dupe if it was posted before but didn't get much attention. It's important to let good stories have multiple cracks at the bat, in order to mitigate the randomness of what gets noticed here.
And there you have pretty much the whole of our thinking about dupes.
Adhering to the ideas of clear goal-setting, team communication, autonomy, and iterative development have never had bad outcomes for me. But formalising them into a ruleset for any arbitrary team doesn't capture everything.
And that's where those companies intervene: they build an implementation of those ideas. The implementation is not adapted for any project and team and it is lossy regarding the original principles, but it has the advantage of having being seriously thought about, tested and scalable.
"People will benefit from SAFe, but not nearly as much as they would from a true agile approach"
https://twitter.com/martinfowler/status/440874515199844352
IMHO all those methodologies that try to somehow shoehorn classical project management into new things are at least to some extent the acknowledgement that things aren't working as expected.
I do think that the 'agile' methodology focus on short achievable iterations and daily stand up meetings yielded good results _in that context_. Disconnected from a healthy organization or transplanted into a bad one, it just yields epic fail and falls apart.
Another thing that I saw work well was embedding product management, testing, and operations people into small per-project teams with developers, which I found reduced conflict from territoriality and reduced "throwing over the wall" syndrome.
Trying to sum up these leaders' qualities is difficult, but if I try to distill it I think it comes down to primarily these two aspects:
1. Strong vision and ability to clearly communicate direction 2. Mutual trust
They wouldn't tell people what to do, and certainly not how to do it, but they'd always – every single day – point in the direction they wanted you to go and course correct where necessary. It wasn't like it was an all-hands meeting every morning either, it was more like gentle nudging or little reminders, but not unsolicited. It's difficult to explain frankly, but the result was that we as a team never felt unsure as to where we were heading.
The most important thing however certainly was the mutual trust. You could trust these leaders to have your backs, no matter what, and these people would trust you to get things done. That sense of trust created an immensely palpable sense of loyalty, you really did feel like your input and efforts mattered, and you always wanted to bring your a-game for the team. Again, difficult to explain, but you know when you're in it.
Not one of these teams worked like the other, in terms of methodology, yet things worked and they worked very well. We had issues, for sure, but I've also worked in dysfunctional teams and the same kind of issues in those teams were amplified by terrible leadership, leading to constantly worsening morale and in some unfortunate cases abject failure to deliver.
https://www.ted.com/talks/tyler_cowen_be_suspicious_of_stori...
(PS: my comment is actually just a joke from the talk, but in any case I really enjoyed and talk and would recommend it.)
Yes, exactly. That's what you want. Make it too complicated and you lose people in the details. The content of the fiction you choose to operate under isn't as important as the collaborative power it unleashes.
Currently, I see too much faith being put in methods of dubious utility. I see techniques, that may be effective when applied judiciously in some circumstances, being promoted to universal principles. In particular, I see the failure of waterfall being taken to mean that any attempt to think ahead is counterproductive.
On the other hand, ten years before the scope of this article, Barry Boehm took on the waterfall model with an iterative one that emphasized frequent reevaluation of where you are, where you are going, and what impediments you face [1]. The fact that this article begins with waterfall supports the notion that adoption is the problem. Perhaps Boehm's model was largely overlooked precisely because it was not dogmatic enough to promote zealotry?
[1] https://en.wikipedia.org/wiki/Spiral_model
This reminds me of the recent threads about people using Excel instead of "real" software. At the end of the day, you are trying to add value. As long as you aren't adding technical debt or other risks, the how matters much less.
I know I am taking this out of context, but this is best thing I read on the internet since a while.
'So I’m cool with it. Lean, Agile, Waterfall, whatever, the fact is we need some kind of common ideology to co-operate in large numbers. '
The standard lean startup risk is building something the customer doesn't want, so it makes sense that the way to mitigate that is by getting customer feedback quickly.
But other projects can have substantially different risks - for example, one project I worked on involved building a pipeline to process a large amount of data every day. There was no real customer risk, the spec was pretty much "do this thing that's valuable on some data, but for 1,000x more." Instead, the major uncertainty was whether the libraries we considered would scale up. To mitigate this, we built small prototype projects with each library on production workloads, and found something like 7 out of 12 of them would work with minor adjustments, while 5 had to be discarded pretty much entirely. We then wrote the project with alternatives for those 5, or writing replacements for the functionality we needed from scratch.
As teams grow, coordination becomes a bigger risk. One way to reduce this is by having interfaces between modules, and enforcing those interfaces in code. My preferred method is static types and functions, but of course there are others like JSON APIs.
In general I see most companies/teams picking on methodology and staying with it, rather than adapting to the project.
https://books.google.com/books/about/The_Principles_of_Produ...
The concepts used in the book - batch size, uncertainty, feedback, control, queue size and latency - these really are useful things to think about when scheduling work.
If you use a software development methodology for something that isn't software development, you should again probably find a different company to work for, because things aren't going to end well there.
If every single project and task you work on uses only one methodology, the people in charge are either morons or full of shit. (Or you are actually a robot in a factory, in which case, hello, I appreciate your intelligence, please do not harm me)
This can lead to interesting effects. For example, I have heard "DRY - Don't repeat yourself" most times I encountered it as a reasoning for obsessive refactorings, where functions where extracted from code without any regards to abstraction layer just because a few lines of code looked the same (or could be rewritten to look the same). But originally the term was coined to represent the principle of "Every piece of knowledge must have a single, unambiguous, authoritative representation within a system". "DRY" was however so catchy, that the slogan propagates without the explanation of the principle.
I think agile suffers a similar fate. Countless consultants, managers and engineers have rendered the word almost meaningless. Even with the Agile manifesto at hand, it is sometimes hard to free agile from the multiple layers of paint spread over it.
DevOps suffers the same fate, and while I haven't researched this, I assume that even waterfall is probably a gross misrepresentation of the original ideas.
https://twitter.com/BonzoESC/status/442003113910603776/photo...
DRY-as-in-the-original-sense is not an axiom because you can actually give good and simple reasons from which you can deduct the principle.
You are definitely are right in the sense, that treating soft truths as "hard facts" is a problem in software.
I think one of the defining features of all of those principles is that they directly conflict with one another. Good code means achieving a local minima, because up to a point, maximizing any one of these principles (e.g loose coupling) will inevitably violate one or more of the other fundamental principles.
I.e. maximizing loose coupling often has the side effect of violating DRY or ramping up language complexity (another fundamental principle).
I don't think you really can derive DRY from other principles, though I'd be curious to hear what you think it could be derived from.
I started doing this in the 90s and I feel that there is a growing trend to replace judgement with process and strict rules.
> I started doing this in the 90s and I feel that there is a growing trend to replace judgement with process and strict rules.
Hmm, I am not quite sure whether I share your sentiment on process and strict rules. I don't mind process if it is actually one that is sensible and leaves freedom to the individual.
For example, in my experience as a developer, Scrum can drastrically improve developer-management communication. Story point estimation with estimation cards has always revealed stories with unclear scope or requirements. This requires however, that both sides actively try to improve that process.
I have often heard of people who said "Scrum doesn't work" who actually were really running a very weird subset of it (for example burndown charts and velocities, but no proper story point estimation), etc.
Definitely read that post.