Ask HN: How do you do estimates in 2021?
For context, I am a manager at a medium sized enterprise software company, I worked as an engineer for 10+ years and took over managing the team. We have 80+ engineers in the entire org. broken down into smaller teams of 5-10. My team specifically has about 15 engineers broken down into teams of 3-5.
We have a very challenging roadmap and often we end up delivering 20-30% of what's planned for the year. One thing that's often asked is how do we estimate, how do we predict when some feature will be done.
We are close to 20+ years into the usage of Agile methods, there is the school of thought who prefer to use time based estimates, some try story points and then there is the No Estimates movement.
I am trying to see what's considered as a best practice to start something for the team in 2021.
101 comments
[ 0.32 ms ] story [ 188 ms ] threadI've ask because I've seen large orgs have accurate estimates, but lay those estimates out incorrectly on the roadmap because the product managers overestimate what percentage of time is actually spent working on the products.
From my perspective it involves two things.
First, it is about applying the experience of similar situations to the future. I can't very well tell my coworkers "I don't know" because the work that they do depends on me getting my stuff done, especially in a timely manner. Thus, I have to draw on previous experience to say, "Well, this (or something like this) took me two weeks to get done last time so it will likely take me a similar amount of time this go round."
The second part of that is being open with expectations. Being up front with my coworkers involves me telling them that while I think this will take X amount of time, these are the complications I am facing that could have an impact on my ability to deliver in the provided time frame.
Ultimately, by applying a combination of those two things I have been able to build a good relationship with those I work with when it comes to providing estimates and expectations for delivery. Honestly, not sure how this will apply to software engineering or if it is even translatable at all, but that's my two cents.
Quite a few software projects though are one off custom jobs where the requirements have never been implemented before (sometimes anywhere) and it does make estimating based on past performance pretty difficult. It's also notoriously hard to predict all the problems you might come across ahead of time, especially if you aren't familiar with the specifics of the code you will need to integrate with (maybe you could consider that like quoting to refurbish an old house before you've looked inside it and seen the damp patch and the hole in the roof)
It does get easier with experience though, and much more so if you are regularly doing the same kind of software projects.
(1) Individual sprints more-or-less hit their goals. Maybe you do 80% of what you expected to do consistently, but there never seems to be a last sprint. (e.g. new requirements keep coming up, new problems get discovered, etc.)
(2) Each sprint is a disaster. You deliver 20 or 30% of what you expected in the sprint.
If you ask the people in the team and other stakeholders you might even find that some believe (1) is the case, others believe (2) is the case.
I would look the following mismatch: The conventional sprint planning process assumes the work is a big bucket of punchclock time where there are no dependency orders, one team member can do the work of another team member, etc.
In some cases this is close to the truth, in other cases it is nowhere near the truth.
For instance if you plan to have work implemented and tested within the boundary of one sprint there is a point at which the work is sent over the wall to the tester. I worked on one project for which each iteration contained a machine learning model that took two days to train (most of this process happened outside "punchclock time") If everything went right you could start two days before the end of the sprint and have a model, but often things didn't go right and if you really wanted the sprint to succeed you would want to start training the model as early as you can, maybe even over the first weekend.
If wallclock time and temporal dependencies are the real issue you have to address that.
- XS = +- 1 day of work
- S = +- a few days of work
- M = less than 1 sprint (= 2 weeks)
- L = a few sprints of work
- XL = months of work
That is amazingly bad estimation! You must have some big systemic issues if you can repeat such a failure even twice and still do estimates. Why bother if you're that far off?
If the goal was a genuine estimate, the next year you would surely multiply your estimate by 3 and get closer, maybe overestimate that year
For level of urgency, I have another system for that, using Cold stone creamery sizes: Gotta have it (must ship this sprint), love it (should ship this sprint), like it (stretch goal). Anything that doesn't get done gets bumped to the next level of urgency for the next sprint.
https://en.wikipedia.org/wiki/MoSCoW_method
"When we plotted the data, in all cases, the actual time was very accurately fit by a lognormal whose scale parameter was precisely the predicted completion time."
[1] https://news.ycombinator.com/item?id=26366112
I use that method for all my consulting estimates, and it has been very reliable for me; no crunch time, and I often come in under time. I've also made a tool that does the math so my clients and other people can continue to use the same process without me.
[1] https://quotes.vistimo.com/
https://writemoretests.com/2012/02/estimating-like-an-adult-...
https://fogbugz.com/Evidence-Based-Scheduling/
https://blog.fogbugz.com/evidence-based-scheduling
https://support.fogbugz.com/hc/en-us/articles/360011258994-E...
If it's like anything that most managers do, you have the team come up with estimates, have some meetings to see if you missed anything, discover that you did, then create a schedule, remembering to add slack time for unforeseen interruptions due to people getting sick, customer issues, etc.
Then you see that the schedule is too long. Slowly but surely browbeat the team into saying "yes" to the question "do you think this can be done faster" on every task. Do this some more when someone from sales asks if something can be delivered in a particular quarter.
Get very stressed out when the real project deliverable dates align much more closely with the original estimate than the unrealistic one. Micromanage your people and stress them out. Keep freaking out as you miss every deadline in the "pipe dream" plan.
Deliver the project with a mild slip compared to the original plan (because you missed something major in the original planning as it is impossible to foresee everything). Use the twice-too-short plan as your metric though. Still, the product is awesome, so give your people a pat on the back.
Hold a "lessons learned" presentation swearing to never do this again. Speak at length about how critical good estimation is. Go on to repeat the very same exercise for your next project.
Just had an idea: maybe keep the version of the timeline from before you haggle down the dates and keep checking which one matches reality better? You don't have to tell the developers that you are doing it if you believe us programmers need some flogging to keep us coding.
If I had a bad case of managers, I would probably multiply by four.
My favorite might be 31. (Mo's Law of Evolutionary Development) You can't get to the moon by climbing successively taller trees.
What happens is that the difference between the estimate that management will psychologically accept to begin the project (and then commit to due to sunk cost fallacy) and the actual schedule is about a factor of 3.
https://pbs.twimg.com/media/EyNRiYQXMAIshEj.jpg
If we had to absolutely deliver by a fixed date - a rarity - we broke the task down in parts that could be estimated, identified the dependencies between tasks, and kept management informed if those dates could slip.
This lightweight process worked pretty well if we had to implement it. There was a lot of trust put in the engineers.
As long as your team manages to finish sprints, it doesn’t matter how you are estimating (or not) the tasks inside of a sprint.
Sprint tasks are for internal (team) use only, including the management inside the team. But people outside (and up) the team just need your deadline commitment.
They have to convince themselves.
We planned out all the epics (create promotion flow, promotion landing page, redeem flow, etc) and sprints. I think it was six 2 week sprints, or about 3 months.
Then the scope creep happened. We agreed on just a "send now" button in the designs, but marketing decided that a scheduling feature was necessary. Somehow we got bogged down building the scheduler and doing timezone calculations (e.g. what if a business located in Pacific time has an owner in Eastern Time who is traveling to Central time, when should a promotion scheduled for 6pm actually get posted?).
In the end we ended up taking twice the estimated time for a feature that was only used by 0.2% of paying customers (it turns out that businesses either already have a coupon system or they don't want one at all).
We naturally learned to slice « too big » tickets and we roughly define « too big » by « I’ll probably struggle on this ».
It’s far from perfect, not really better but absolutely not worse than when we did estimates and we are able to achieve our goals, which is what matters.
I was briefly informed about it during a scrum event and didn't saw it mentioned here.
And it's about estimating project tasks. So relevant to the question.
My guess is that you or the down voters don't know it. Correct me if I'm wrong.
On some teams, we're trending towards "story-count" rather than points. These teams are typically executing extremely well and don't need the (small) overhead of points.
-----
John Cutler has some good thinking on this issue: https://medium.com/hackernoon/work-small-even-if-it-makes-no...
If you're a company that plans far in advance, the same is true. You'll demand the work you wanted is done at a given date, again regardless of how difficult it was.
At my company, we have a well maintained backlog of work. We pick dates in the future to check in on what got done in between check ins. The product manager can (re)prioritize work as needed. We pull from the top. If there's a surprise deadline it gets brought up to everyone and then put at the top of the queue, sometimes over currently in-progress tasks.
You cannot bend the realities of time and complexity. If something is hard, an estimate doesn't make it easier. A hard delivery date also doesn't bring predictability. Ultimately, with or without estimates, you'll get what you get. If you want to get _more_, ask a team what's making them slow and then prioritize fixing the things they bring up. If every team is working optimally (they're not), hire.
I'll also add, the time it takes to prepare an estimate of any value is probably 10x what anybody asking you to estimate something is willing to provide. You're being asked to spit out a number to fit an existing narrative. If you wanted to estimate a unit of work with any amount of legitimacy, you'd need hours/days/weeks (depending on SOW). These companies scheduling weekly estimation meetings that last an hour and are bullshit scrum cards don't matter and aren't interested in being even close to correct.
Sometimes implementing what should be a simple feature requires a month of refactoring some old technical debt. And sometimes what seems like a very complex feature is actually just a matter of enabling a flag on some third party component you're already using.
All is us have likely experienced "why is x taking so long?? Had we known, we wouldn't have done it now!" and/or "we didn't realize y was so simple, otherwise we would have done it months ago and probably won a few more deals!" How do you avoid that?
Why does the size of the work that needs to get done matter? If you need it you need it. If you don't, you don't. If you want a fast fix, note it. Speak in outcomes. "The lowest amount of work possible to get us ________"
> All is us have likely experienced "why is x taking so long??
I wouldn't work with a PM that disrespected me like that.
> Had we known, we wouldn't have done it now!" and/or "we didn't realize y was so simple, otherwise we would have done it months ago and probably won a few more deals!" How do you avoid that?
This isn't prioritization, it's a PM making ill-informed judgements on dev work. Don't have your PMs do that. Product people can define priority, scope, etc. If your PM isn't prioritizing something because they think it takes a long time, they do not understand their role.
Generally, I would agree, but I've seen cases where it mattered, because the value of a feature would become 0 after a set date. For instance, we needed a specific feature to fulfill the requirements for a (massive) contract, and you could only apply for the contract until a specific date. The value of this feature outside this scope was pretty much zero.
As I said, you put it at the top of the backlog and alert the team of a new deadline. You work on it until the deadline. If you make it, great, if not, you move on. Asking for an estimate wouldn't have made you make it, it might've only prevented you from trying.
Except for when potential clients ask your company: "How much implementing a system to do X would cost?"
If your company attempts to calculate this based on how many people would be needed to cover the scope and what the technical complexity of the implementation would be like, then you need to give an answer as a developer, so that the sales department can do some ballpark calculations and give a response to the clients. Especially when numerous other companies within the industry are also attempting to answer that same question.
The processes and methods vary, of course, some use historical data from other projects, some use methodologies like COCOMO, others don't even ask their technical people and try to squeeze as much money from these potential clients as possible, but in the end, someone somewhere cares about the total time the project could take, ahead of time.
It doesn't matter that it's almost impossible to give accurate estimates due to the nature of development (e.g. an everchanging environment with bunches of different technologies that evolve and die, as opposed to a production line of widgets) and it doesn't matter that these requirements are probably inaccurate, that they will change, that there will be scope creep and numerous other difficulties (various development decisions that will impact the project long term, many restrictions and requirements that are dependent on the environments that the clients have).
In the end, i dislike estimation, clients probably don't care about any of the above and want them anyways, which leads to the development methodologies remaining "agile" in name only and estimates end up being expressed in days rather than an abstract number representation of complexity when compared to other similar tasks in that particular project.
And what do estimates provide here, that sales just picking a number doesn't? Estimates are made up numbers. They're estimates.
Also this line that you omitted answers your question more directly:
> If you're a sales driven/feature factory company, the estimates won't matter anyway as you'll demand to meet your obligations regardless of estimates.
Example #1:
Example #2: Example #3: In the example #1, the ballpark figures were accurate enough for the project to be done in a profitable manner. In example #2, there were factors that weren't considered and would result in either contractual penalties, the clients deciding to break the contract because you can't deliver on time, or to take you to court. In example #3, past data was used in an inaccurate way due to not being applicable to the constraints at hand (or lack thereof).Of course, the above happens when you're in a market that requires estimates as a part of BOMs, which is a lot like bidding on projects and is just a race to the bottom for the most part.
Yeah this, to me, totally invalidated the rest of your example. Pick a low number to win the bid, blow the timeline/estimate, require more money to do anything at all. It is not clear to me that any "estimation" changed this calculus at all.
Except that there are penalty mechanisms in place to prevent this from happening.
So your company was slow to deliver the product and needs more time? You better be prepared to work for your own resources, otherwise the escrow won't get released, as per the contract.
Or perhaps you cannot finish it at all? Get ready to either not receive a large portion of the money at all, or to be sued outright. For an example, see here: https://www.consulting.us/news/2197/accenture-sued-for-32-mi...
Disclaimer: i have nothing to do with Accenture or consulting in the US, though consulting, government contracts etc. all share certain similarities and bureaucratic processes in most countries that don't always align with the realities of actually developing software.
If you can, steer clear of all of that and instead work for a technology oriented company that develops a product that they also sell themselves. One, in which the developers are viewed as a profit center, instead of a cog in a ROI generation machine. Ideally, an engineering led company.
Deliver working software? Perhaps.
Deliver quality software? Almost certainly not.
Now, whether people actually want quality software, given that its development would take a lot of time, is debatable. Many times being the first to market is good enough, though saying that leaves a sour taste in my mouth as an engineer.
This only works for some projects. For example if your MVP or go/no go prototype costs $5MM (which a hardware or life science product easily could) you really need to know if it will be 5 or 10.
I ran a company which always quoted 3X fixed price what we thought the project would really cost, even if things turned out to go wrong. Typically we made the (promised) schedule, and when not (luckily didn’t happen too often) the customer would be able to tell pretty much as soon as we could tell, and together we dealt with it.
But as for the cost: most projects were extremely profitable but sometimes we would lose hundreds of thousands on them. That’s why we charged such a premium: we absorbed the financial risk (not that we ever told the customers our internal cost estimate — none of their business!)
Also, how frequently do hardware and life science projects go far over budget and under scope? Do their estimates matter?
As I said, if you have $5MM, you'll get what you get for $5MM. If you want to spend a boatload of time researching, your estimate might be closer, but that research isn't free.
Most of the time, in my experience. At a bigger company that may not matter so much but at a startup it can be fatal.
Knowing how far you'll get might affect what you prioritize, no?
Exactly, the answers might surprise you