Ask HN: Is there a way to get Agile right?

Agile kind of has to mean "willing to hack your methodology". ("Let us be agile by following this rigid methodology" is self-contradictory nonsense.)

So, try something. If it makes things worse, stop and try something else. If it makes things better, keep that. Repeat. (Note, however, that things that work may not work forever.)

It's really hard to lose by following that approach. The only way you lose is when someone has a rigid approach that must be followed, in spite of obvious evidence that it isn't working.

> So, my question: Is there a way to get Agile right? or are we doomed to fail in the same ways that has been described by countless articles discussing the downfalls of Agile?

Yes. Don't be dogmatic. Pick an initial set of processes/practices that match more or less what you already do, want to do (based on a reasonable belief of utility and practicality), or have done (with success) in the past. As you move forward, periodically reexamine what you do and change when something isn't working well (in Scrum this is the Sprint retrospective, which seems to be the first thing cut from every Scrum-derived process I've ever been part of or near to "save time"). Experiment not just with your code and systems, but with your processes and organizational structures themselves. If you only apply things like the ideas of Extreme Programming to your programs, you're only doing half (actually less) of what Agile aims for.

Sterman in his Business Dynamics book has a nice set of diagrams starting on page 15, talking about feedback loops/systems for learning.

Attempting to draw them as ASCII art, not that skilled at this:

Simplest feedback model:

         Real world
         ^      \
        /        \
       /          v
   Decisions<--Information feedback

This is a very common thing, people make a decision, try it, get info, and make new decisions. What they don't do is extend this feedback to their decision making model. He extends this model on page 16 with:

         Real world
         ^      \
        /        \
       /          v
   Decisions<--Information feedback
      ^
       \
        \
     Strategy, Structure,
       Decision Rules
           ^
            \
             \
          Mental Models
            of Real World

Again, note that the feedback does not go back to the mental models which feed into the decision making strategies and, ultimately, the decisions. This is the dogmatic approach to any process. You select it, and you run it regardless of its real applicability or utility but based on a belief that you don't update (because it's dogmatic). Doesn't matter if this is Agile (big-A) or big design up front (BDUF) or anything else, if you don't reconsider your models then you will not be agile, you will be stuck. This doesn't mean things won't work, you may have settled into a local optimum (on accident or on purpose). Maybe things are done this way because they've always been done this way, but 30 years ago someone did think about the business and selected something that worked (deliberately) and things haven't changed much so it happens to still work well. But once it fails, you have no deliberate way of adapting to the new reality (which is a key to being agile, responsiveness).

The final model he shows on page 19 brings that feedback to not just the decisions but also the models (this is the hard one to put into ASCII art):

         Real world
         ^        \
        /          \
       /            v
   Decisions<----Information feedback
      ^                       ^|
      |                       ||
      |                       |v
  Strategy, Structure,<----Mental Models
    Decision Rules         of Real World

This, to me, is the critical thing to agility and if you focus just on the manifesto and not the Agile Industrial Complex (and fuck SAFe and others like them), you will see that this is present (though less explicitly). Paying attention to feedback, trying new things, being adaptive. That is the essence of agility and Agility.

=======================

Regarding using a shared spread sheet or something else for managing your work, that's all about scale. A 4-person team can use a spreadsheet for status management, it may not be the best, but it will totally work. Scale up to 4 teams, each with 10+...

(1) There has to be trust between management and people who do the work. You can get things done without trust but it is going to be expensive, late, broken, nerve-wracking, etc. You aren't going to paint process onto a low trust situation and see improvement unless the trust improves.

I'm going to go out on a limb and say that close to 100% of the time when people are in agony over agile there is a lack of trust.

(2) If you want to have good results you have to have a technical framework such that changes that management thinks are "a small change" are really a small change. Some people think agile means "not thinking ahead" -- you certainly shouldn't be borrowing trouble but you should be planning for sustained productivity which means that tomorrow's "small change" is not a 6 month project.

This is not to advocate for some particular, language, framework, database, whatever but to say that, from a system level, you are designing on the expectation that you're going to be doing "sprints" on this thing for a long time and that you shouldn't need major re-engineering.

(3) You've got to get realistic expectations about details, the "definition of done", etc. For instance does "finishing the sprint" mean that the developers throw a feature over the wall to testers? Or does that mean the testers say the feature is ready to go? If it's the latter, the testers will need to see the product long before the sprint is "over". Then what do the testers do before that? What do the coders do when the testers are testing?

I'm not advocating for a particular right answer but that there has to be agreement on the team for something real. I have been on some teams where we often finish 2/3 of what is supposed to be on the sprint but we do a good job of not letting things get stuck so we solidly deliver features. There are other people who would blow a fuse if there were 15 items on the sprint and only 14 got done.

(4) When it comes to very high velocity I think fixed length sprints are a problem instead of a solution. Sometimes there is a feature you could put in front of customers in two days but they have to wait two weeks because of the process. If there are multiple teams that have to line up with their own sprints a small delay can snowball into a big delay.

In teams without trust people really stick to the sprint structure rigidly because it is all they have. The most effective teams will figure out how to to bend the sprint structure to overcome its limitations.

Two rules to get agile right:

1. Make things that are easy to change

2. Change things

The first line of the Manifesto for Agile Software Development:

Individuals and interactions over processes and tools

Don't try to spend time deciding on spreadsheets versus Trello versus Slack versus etc for managing the project, instead focus on the problems your team has.

Simple example: let's say your project management spreadsheets are not working well because people keep duplicating work that someone already finished, so you decide to start using Trello instead. Turns out, your teammates had not been regularly updating their status in the spreadsheet, and you still have the same problem in Trello- people are not keeping their status up-to-date, so the problem is not solved, and you have a bunch of teammates angry because the problem of duplicated work has not been solved AND they had to learn a new tool.

> tracking work efforts

Why are you doing this? How does it help improve delivery of value to those who are paying for the product/service?

Agile was intended to have people discover better ways to deliver something of value.

This is why the Agile Manifesto starts with People (vs. some unchangeable process handed down from above) and Working (Valuable) Software, instead of a book full of "requirements". And so on.

If you haven't read the Agile Principles, you're unlikely to be Agile: https://agilemanifesto.org/principles.html

The goal is: provide value, get external feedback on how it could be more valuable, as well as internal feedback on how people can deliver that value better.

That's it.

Everything else needs to either support that, or is just someone trying to micromanage you (and don't think that's not done on purpose).

As someone who has worked in 5 different companies using some form of Agile, I have the following take on this (below). I know you said you're not strictly a development team, but that's where my experience is and the comments below are focussed. That said hopefully some of the points are transferable to your domain.

- Agile is about flexibility, but it can* come with a much bigger price tag if you don't do most of your analysis/design up front for big/technical projects. *Sometimes not always. More on this below.

- It's great having conversations with product owners and business representatives at various times as you progress to other features, but you can't have multiple developers going back and forth for this (as you loose continuity) and if you choose your lead developer or solution architect (if you have one), then they'll spend more time on admin than on actually designing and building things. Therefore it's good to have a business analyst or dev manager help do most of this, with the option for your tech guys to get some face time with product owners if they want/need it.

- I found that none of the business analysts charged with leading/facilitating the Agile requirement gathering really knew how to write good epics / user stories where I worked, and this can make life difficult / waste lots of time during refinement sessions and during development.

- The biggest challenge I found with Agile previously is how to incorporate design tasks (more on this below).

- Some of the business analysts, dev managers, project managers, portfolio managers that I worked with thought that once all the epics and stories were defined we'd just be able to build everything with zero design. Er, no. The lack of a definite 'design phase' like in waterfall doesn't mean no design is required, it just means that your design execution is more flexible and incremental if need be. You still should go through the design process for your separate features (as they get prioritised) because a) Like TDD for units of code, it forces you to think about the gotchas/edge cases/touch points/cross cutting concerns BEFORE you implement something. b) You have the design to refer to in follow up discussions with product owners / BAs / dev managers and your development team which could consist of several developers.

- By design above I mean a design document/wiki page that covers conceptual, logical, and physical aspects of the solution that will be built. The design might also include and overview of discounted designs initially considered but then not chosen. You want your conceptual/high level design section to be just that, with all the really tech stuff in your logical design section e.g. most your UML stuff and other tech points. But back on Agile, by design I ALSO mean Spikes and Proof of Concepts that you sometimes need to do to validate design assumptions.

- Even with an 'Agile' approach being dictated from above, you might still decide / realise that you need a waterfall approach of sorts, at least initially for the analysis and design (or enough of the design to de-risk things). For example, in massive or extremely technical projects if you don't consider the full requirements or full design up front, then you can have these incredible costs of re-work. E.g. If features 1 and 2 need to be re-written because their implementation is not compatible with features 5 and 6 which were always needed, but were just further down the backlog and not considered in the initial design work.

- This leads me to what I see as the the delayed cost and increased cost of Agile, for large projects with set business requirements. By that I mean again, if you have a large/very technical project with clear defined goals (example in next point), then you probably want to do your full design up front or at least 80%-90% of it. Definitely all of your conceptual design and most of your logical/detailed d...

Check early ID software (from 91-97) and you will get a clear picture of how to do high quality, very efficient development:

- Small team in which everyone knows everyone else and the chemistry is right

- Clear objective, all team members own something, could be share or part of project, whoever falls behind got maybe one or two chances and then gets cut ruthlessly

- Team is working on challenging projects

Most teams simply do not satisfy even one.