What the article suggests is basically Kanban. It's the most effective SW development method, and similar scheduling system (dispatch queue) is used by operating systems in computers. However, management doesn't want Kanban, because they want to promise things to customers.
You can make good estimates, but it takes extra time researching and planning. So you will spend cycles estimating instead of maximizing throughput, and to reduce risk, plan is usually padded up so you lose extra time there according to the Parkinson's law. IME a (big) SW company prefers to spend all these cycles, even though technically it is irrational (that's why we don't do it in the operating systems).
This is why I push for Kanban whenever I am a PO. If we can ballpark an estimate, I can prioritize it. If we cannot ballpark an estimate, I can prioritize the research to clear out some of the unknowns. But most importantly, we set an expectation of rolling feature rollouts, not inflexible release dates. We communicate both internally and externally the next few things we are working on, but no hard dates. The article correctly identifies that hard release dates communicated to customers are the root cause of problems, so I simply don't give such things out.
A couple of decades back PMs used to look at historical data to guide the estimates for a new project. If a similar coding work took 2 weeks on average in the past, that gives some basis.
So, I think the issue is about whether it is a routine workflow work which has well-tested historical timelines or not.
Nevertheless, estimates are needed at some granularity level. When you order something on Amazon, would like an estimate on when the item would be delivered to you?
Even if coding work can't be estimated, the overall project requires estimation. Someone need to commit to timelines and come under pressure. Distribution of that pressure is only fair.
When teams don't need strong estimates, then Kanban works well.
When teams do need strong estimates, then the best way I know is doing a project management ROPE estimate, which uses multiple perspectives to improve the planning.
The unique thing about estimates in software engineering is that if you do it right, projects should be impossible to estimate!
Tasks that are easiest to estimate are tasks that are predictable, and repetitive. If I ask you how long it'll take to add a new database field, and you've added a new database field 100s of times in the past and each time they take 1 day, your estimate for it is going to be very spot-on.
But in the software world, predictable and repetitive tasks are also the kinds of tasks that are most easily automated, which means the time it takes to perform those tasks should asymptotically approach 0.
But if the predictable tasks take 0 time, how long a project takes will be dominated by the novel, unpredictable parts.
That's why software estimates are very hard to do.
I call it the "Persistent Incompetence of Software Development", which is another perspective on estimation, focused more on expertise. A chef that cooks pizzas, cooks the same pizza over and over again and becomes amazing at it. If you are a developer that writes the same code over and over, you are terrible at software development. A good software developer should always be solving new problems, as by the nature of software, once they solve a problem, they never solve that (exact) problem again. So we are persistently incompenent.
Which is why software development can't be estimated, as well. Because it is all, as you say, novel. With infinite error bars.
At this point, I can't take anyone seriously that believes software dev can be estimated.
One thing that changed my way of thinking about estimates is reading that 86% of engineering projects, regardless of what kind of engineering (chemical, infrastructure, industrial, etc) go over budget (in time or money).
Missing estimates isn't unique to software, but it's common across all engineering fields.
The thing that sucks is that when I avoid giving estimates, I'm not trying to be difficult, I'm being honest about the unknowns of the project and the inherent uncertainties and messiness of software development. I'm helping protect myself and the rest of the team from making plans based off of bad estimates.
But I get all this pushback when I do that, such that the path of least resistance is to give some bullshit estimate anyway. Or I get asked to make a "rough guesstimate", which inevitably turns itself into some sort of deadline anyway.
Garbage in, garbage out. Inaccurate estimates, unreasonable timelines, stressed devs and upset PMs.
It's also difficult for LLMs it seems. If I forget to add instructions to skip resource estimates, Claude will estimate a week or two, then bang it out in under an hour.
As a largely solo dev I found I can't estimate well unless it's a common task, and it's easy to find tasks grow exponentially if it touches too many layers.
Asking "how long do you want me to spend on this?" got better results, because I got more idea how important tasks were to the business and can usually tell if something is going to take longer than they want. (Or know when we need to discuss scoping it back, or just abandoning the feature)
30 years ago my boss at a large defense/aviation contractor told me estimating software projects was a very valuable skill, but all estimates were always wrong because they are simplifications and to keep that in mind -- his words.
Mainly they are useful to build belief and keep a direction towards the goal.
Models of any kind in whatever domain are necessarily always something less than reality. That is both their value and weakness.
So estimates are models. Less than reality. Therefore we should not expect them to be useful beyond 'plans are useless, but planning is indispensable' -- I think thats' Eisenhower.
There’s a well-established Agile technique that in my experience actually succeeds at producing usable estimates.
The PM and team lead write a description of a task, and the whole team reads it together, thinks about it privately, and then votes on its complexity simultaneously using a unitless Fibonacci scale: 1,2,3,5,8,13,21... There's also a 0.5 used for the complexity of literally just fixing a typo.
Because nobody reveals their number until everyone is ready, there's little anchoring, adjustment or conformity bias which are terribly detrimental to estimations.
If the votes cluster tightly, the team settles on the convergent value. If there’s a large spread, the people at the extremes explain their thinking. That’s the real value of the exercise: the outliers surface hidden assumptions, unknowns, and risks. The junior dev might be seeing something the rest of the team missed. That's's great. The team revisits the task with that new information and votes again. The cycle repeats until there’s genuine agreement.
This process works because it forces independent judgment, exposes the model-gap between team members, and prevents anchoring. It’s the only estimation approach I’ve seen that reliably produces numbers the team can stand behind.
It's important that the scores be unitless estimates of complexity, not time. How complex is this task? not How long will this task take?
One team had a rule that if a task had complexity 21, it should be broken down into smaller tasks. And that 8 meant roughly implementing a REST API endpoint of complexity.
A PM can use these complexity estimations + historical team performance to estimate time. The team is happy because they are not responsible for the PM's bad time estimation, and the PM is happy because the numbers are more accurate.
One thing frustrating for me is when folk leave $BigCo, with it's methods (ie: estimate time to complete, sprint planning) and try to apply those same methods at a very early company.
Estimates don't work there at all - everything is new.
So, flip it. Use known values to prioritize work. That is: client demand and (potential) revenue. Then allocate known time/budget to the necessary project, see how far you get, iterate. Team can move faster. Looks chaotic.
At some (uncomfortable) point however, need to rotate into the "standard" process.
I wonder how different the perception of these projects being late or (massively) over budget would be if we used different words. Bear with me here...
Words mean things. Estimate carries a certain weight. It's almost scientific sounding. Instead, we should use the word "guess".
It's exactly equivalent, but imagine the outcome if everyone in the chain, from the very serious people involved in thinking up the project, to funding the project, to prioritising and then delivering the project, all used the word "guess"
Now, when the project is millions of dollars over budget and many months/years late, no one is under any pretence that it was going to be anything else.
I tried this once. It turns out serious people don't like the idea of spending millions of dollars based on "guessing", or even letting developers "play" in order to better understand the guesses they are forced to make, even when it turns un-educated guesses into educated guesses.
Of course, none of this would improve the outcome, but at least it sets expectations appropriately.
In the 70's a Brazilian programmer told me his method was to make his best guess how long something would take, double it, then promise that amount plus-or-minus 50%.
My experience is that I have to basically always overestimate if I can get away with it because otherwise if something goes wrong, I will pushed to do free overtime to complete all the work assigned in a given sprint.
I think that executives requiring estimates of time from product owners (PMs, Engineering Managers) is an instrument for putting them into de-facto 'debt' servitude, and provides a constant stream of justification for dismissal with cause. As others have commented, if the ability to time perfectly was there, it would no longer have been an innovative product. Same with requiring sales forecasts from salespeople. There's no way for the salesperson to know, so they are constantly on the chopping block for falling short of forecasts they are forced to generate.
I imagine above is more or less tacitly acknowledged in tip-sharing conversations between & among execs & their investors.
From both the developer and manager side of things, I've found that the most important attribute of estimates is frequently the least paid attention to: that they be kept up to date.
When you discover more work hidden under that "simple" pile of code, you absolutely HAVE to update your estimate. Add more points, add more tickets, whatever. But then your various managers have the ammunition to decide what to do next - allocate more resources to the project, descope the project, push back the release date, etc.
Far too frequently, the estimate is set in stone at the start of the project and used as a deadline that is blown past, with everyone going into crisis mode at that point. The earlier the estimate is updated, the calmer and more comprehensive action everyone responsible can take.
59 comments
[ 3.5 ms ] story [ 56.8 ms ] threadhttps://blog.pragmaticengineer.com/yes-you-should-estimate/ > https://news.ycombinator.com/item?id=27006853
https://josephmate.github.io/PowersOf2/ Complexity Estimator
https://earthly.dev/blog/thought-leaders/ > https://news.ycombinator.com/item?id=27467999
https://jacobian.org/2021/may/20/estimation/ > https://news.ycombinator.com/item?id=27687265
https://tomrussell.co.uk/writing/2021/07/19/estimating-large... > https://news.ycombinator.com/item?id=27906886
https://www.scalablepath.com/blog/software-project-estimatio...
https://estinator.dk/ > https://news.ycombinator.com/item?id=28104934
https://news.ycombinator.com/item?id=28662856 How do you do estimates in 2021?
https://web.archive.org/web/20170603123809/http://www.tuicoo... Always Multiply Your Estimates by π > https://news.ycombinator.com/item?id=28667174
https://lucasfcosta.com/2021/09/20/monte-carlo-forecasts.htm... > https://news.ycombinator.com/item?id=28769331
https://tinkeredthinking.com/index.php?id=833 > https://news.ycombinator.com/item?id=28955154
https://blog.abhi.se/on-impact-effort-prioritization > https://news.ycombinator.com/item?id=28979210
https://www.shubhro.com/2022/01/30/hacks-engineering-estimat...
But, if you are enough experienced, with some fermi-style math, you will do good enough estimates most of the time.
Another way to say this is that an estimate becomes a commitment to not learn.
Re-planning is seen as failure by [management].
Re-planning is what happens when you learn that a previous assumption was not correct.
You should be encouraging learning, as this is THE cornerstone of software development.
You can make good estimates, but it takes extra time researching and planning. So you will spend cycles estimating instead of maximizing throughput, and to reduce risk, plan is usually padded up so you lose extra time there according to the Parkinson's law. IME a (big) SW company prefers to spend all these cycles, even though technically it is irrational (that's why we don't do it in the operating systems).
So, I think the issue is about whether it is a routine workflow work which has well-tested historical timelines or not.
Nevertheless, estimates are needed at some granularity level. When you order something on Amazon, would like an estimate on when the item would be delivered to you?
Even if coding work can't be estimated, the overall project requires estimation. Someone need to commit to timelines and come under pressure. Distribution of that pressure is only fair.
When teams do need strong estimates, then the best way I know is doing a project management ROPE estimate, which uses multiple perspectives to improve the planning.
https://github.com/SixArm/project-management-rope-estimate
R = Realistic estimate. This is based on work being typical, reasonable, plausible, and usual.
O = Optimistic estimate. This is based on work turning out to be notably easy, or fast, or lucky.
P = Pessimistic estimate. This is based on work turning out to be notably hard, or slow, or unlucky.
E = Equilibristic estimate. This is based on success as 50% likely such as for critical chains and simulations.
Tasks that are easiest to estimate are tasks that are predictable, and repetitive. If I ask you how long it'll take to add a new database field, and you've added a new database field 100s of times in the past and each time they take 1 day, your estimate for it is going to be very spot-on.
But in the software world, predictable and repetitive tasks are also the kinds of tasks that are most easily automated, which means the time it takes to perform those tasks should asymptotically approach 0.
But if the predictable tasks take 0 time, how long a project takes will be dominated by the novel, unpredictable parts.
That's why software estimates are very hard to do.
Which is why software development can't be estimated, as well. Because it is all, as you say, novel. With infinite error bars.
At this point, I can't take anyone seriously that believes software dev can be estimated.
Missing estimates isn't unique to software, but it's common across all engineering fields.
But I get all this pushback when I do that, such that the path of least resistance is to give some bullshit estimate anyway. Or I get asked to make a "rough guesstimate", which inevitably turns itself into some sort of deadline anyway.
Garbage in, garbage out. Inaccurate estimates, unreasonable timelines, stressed devs and upset PMs.
I'm so over working on software teams.
For humans, 2x the original estimate.
Asking "how long do you want me to spend on this?" got better results, because I got more idea how important tasks were to the business and can usually tell if something is going to take longer than they want. (Or know when we need to discuss scoping it back, or just abandoning the feature)
Mainly they are useful to build belief and keep a direction towards the goal.
Models of any kind in whatever domain are necessarily always something less than reality. That is both their value and weakness.
So estimates are models. Less than reality. Therefore we should not expect them to be useful beyond 'plans are useless, but planning is indispensable' -- I think thats' Eisenhower.
The PM and team lead write a description of a task, and the whole team reads it together, thinks about it privately, and then votes on its complexity simultaneously using a unitless Fibonacci scale: 1,2,3,5,8,13,21... There's also a 0.5 used for the complexity of literally just fixing a typo.
Because nobody reveals their number until everyone is ready, there's little anchoring, adjustment or conformity bias which are terribly detrimental to estimations.
If the votes cluster tightly, the team settles on the convergent value. If there’s a large spread, the people at the extremes explain their thinking. That’s the real value of the exercise: the outliers surface hidden assumptions, unknowns, and risks. The junior dev might be seeing something the rest of the team missed. That's's great. The team revisits the task with that new information and votes again. The cycle repeats until there’s genuine agreement.
This process works because it forces independent judgment, exposes the model-gap between team members, and prevents anchoring. It’s the only estimation approach I’ve seen that reliably produces numbers the team can stand behind.
It's important that the scores be unitless estimates of complexity, not time. How complex is this task? not How long will this task take?
One team had a rule that if a task had complexity 21, it should be broken down into smaller tasks. And that 8 meant roughly implementing a REST API endpoint of complexity.
A PM can use these complexity estimations + historical team performance to estimate time. The team is happy because they are not responsible for the PM's bad time estimation, and the PM is happy because the numbers are more accurate.
A clear description with background appears in Mike Cohn’s original writeup on Planning Poker: https://www.mountaingoatsoftware.com/agile/planning-poker
Estimates don't work there at all - everything is new.
So, flip it. Use known values to prioritize work. That is: client demand and (potential) revenue. Then allocate known time/budget to the necessary project, see how far you get, iterate. Team can move faster. Looks chaotic.
At some (uncomfortable) point however, need to rotate into the "standard" process.
Words mean things. Estimate carries a certain weight. It's almost scientific sounding. Instead, we should use the word "guess".
It's exactly equivalent, but imagine the outcome if everyone in the chain, from the very serious people involved in thinking up the project, to funding the project, to prioritising and then delivering the project, all used the word "guess"
Now, when the project is millions of dollars over budget and many months/years late, no one is under any pretence that it was going to be anything else.
I tried this once. It turns out serious people don't like the idea of spending millions of dollars based on "guessing", or even letting developers "play" in order to better understand the guesses they are forced to make, even when it turns un-educated guesses into educated guesses.
Of course, none of this would improve the outcome, but at least it sets expectations appropriately.
These are usually companies that are led by and perform engineering work.
Software developers aren’t engineers.
Project managers have no authoritative training, certification or skills to manage software development projects.
When you discover more work hidden under that "simple" pile of code, you absolutely HAVE to update your estimate. Add more points, add more tickets, whatever. But then your various managers have the ammunition to decide what to do next - allocate more resources to the project, descope the project, push back the release date, etc.
Far too frequently, the estimate is set in stone at the start of the project and used as a deadline that is blown past, with everyone going into crisis mode at that point. The earlier the estimate is updated, the calmer and more comprehensive action everyone responsible can take.