the top voted answer lectures OP about ackshually having an X/Y problem, coupled with with a cheeky boomer pull-yourself-up-by-your-bootstraps “you're part of the problem; be part of the solution” to downplay any responsibility on the part of management
this is such hilariously distilled peak stackoverflow that it belongs in the hall of fame
This didn’t read cheeky to me; how I understood it is: it’s pointless to convince the management to prioritize dev needs in this scenario, either throw the management some bone or find a better workplace.
I agree that the top-voted answer is obnoxious, and right off the bat:
"Most of your question is really a rant about how things work at your workplace. Discussions about toxic workplace practices per se are out of scope for PMSE."
Not a friendly, charitable answer, and it is right at the top.
As to the question itself, crowd-source the estimation, is the best approach, in my experience. Get the team together, each member write down an estimate, and reveal all the estimates at once. If there are outliers, have a discussion, otherwise round up and move on to the next estimation. If you're in a scrum shop, you can abstract the estimation by one level and call them "stories" and you're estimating "complication" not "time", which is even better.
I agree with you on this being peak stackoverflow. It's a shame how these people install themselves as mods and just run amok. What a shame. The first year or two of stackoverflow was the best times. You could actually have great conversations with professionals.
There is some reason why so many people upvoted this comment and not yours to the top. Many readers here are highly experienced developers and know that there is always some responsibility on the part of management but this question also strongly tells a lacking responsibility on the developer side.
You’re all acting like he does know, he’s just not telling you. Step back for a moment and consider how things change if he actually doesn’t know and not even a water boarding session will get him to tell you how long it’s going to take. Now what? You either live with the uncertainty or fire him and replace him with somebody who can conjure up estimates - which, presumably was what you were trying to do when you hired him in the first place.
Yeah I’d hate to be working with this guy. Your coworkers care about when things get done, too. Nothing more frustrating than to sit on your hands because of a dependency on some other team.
Agreed. Having human dependencies who have competing priorities or unknown competencies on the critical path is such a big risk on timelines.
In some cases I’ve got agreements up front that if the dependencies aren’t done or being worked on by particular dates my team gets access to the codebase or systems and can take over. That type of consequence/potential embarrassment can set up potential adversarial positions but it has helped a few times.
You've identified the problem but missed the solution: don't bet your business on a dependency that doesn't exist yet.
Be useful enough to work on a different project, or take a furlough until you are needed.
And yet at most companies an army of "facilitators" exist, with access to extensive tracking metrics and tools. They should be able to at a minimum ask the right questions (i.e. not "how long will x take") to create and maintain the estimates.
I don't think there's any question that people always want this estimate. The problem with most software is that there really isn't one, since it's usually an open-ended search for a solution.
Also, a job on his home is exactly what it should not be compared with, since that's the sort of repeatable job that can be estimated.
No, just all software that provides any sort of tangible benefit to anybody. I’m sure you could estimate fairly accurately how long it would take to implement quicksort - but nobody’s going to ask you that.
Frederick Brooks' "Mythical Man-Month" book is the classic answer. For my own evidence I base it on 50 years experience. I just wrote a piece of code that I had all working, but then I discovered a new method (requiring rewriting it all) that greatly improved it in every respect. So was I 90% done before that? Coding is just a very non-linear and never-ending process.
You were 100% done. And then you discovered an opportunity for new work.
I know many developers enjoy the art of software development and want to continually improve it, but that isn’t always the task you are being paid to do.
There are many ways to remodel a bathroom. Some more optimal than others.
Now imagine your plumber tells you he discovered a better layout for the plumbing just after finishing his work, then he enthusiastically asks if he could tear everything out and start over (and pay him for it).
That same argument can be applied to home renovations then. The carpenter has build many cabinets before, but perhaps not YOUR exact cabinet. So then no estimate is expected in that case?
The carpenter rarely discovers a few days in that the screwdriver they have to use on this project only works every other week. Or that literally every measurement on every component is off by 20% because "that's how it's done" with whatever garbage you stuck them with.
Vendors and even open-source libraries are often liars or just missing what seems like it ought to be obvious functionality. Sometimes tools are just broken. I once was assigned a Java AWS Lambda project, which, great, supported, right, says so right on the website? Except half the CLI tool functionality just didn't work, at the time, for Java projects, and I mean the official tools. It was alpha-tier supported at best. Usable, sorta, kinda, but you're gonna lose some time un-fucking the situation, and the whole thing would be slower to develop than it needed to be. You'd not know that unless you went and did a focused search on their issue tracker(s), which isn't really reasonable to do for all the stuff you might use in a typical project (some of which you might not even know you need until you've started work). "Great, so now you know that for estimating next time, if you get assigned a task exactly like that one" well, no, not really, because that was like four years ago and the situation's probably totally different now (maybe better, maybe worse, who knows?)
That's why I think giving 4-8 weeks of development time with some basic t-shirt sizing or pointing for tasks and just seeing how it's going is better, before estimating. I think if you can't afford that, before getting the estimate, you can't afford a half-decent estimate at all and just need to deal with the fact you're going to get a terrible estimate with an absolutely huge range ("six to eighteen months"). Which, fine to ask for that up front then refine it after some dev time! It just needs to be clear and expected that those early estimates are going to have some wild variance—but they can definitely tell you "is this more a 1-year project, or more a 5-year project?" which does have some real value.
Commercial software isn't generally an artisanal crafting process. If you're a programmer who solves variations of a problem, analogous to a carpenter building varieties of cabinets, then you have parameterized tools. When the client asks for maple instead of oak, you don't pull out a maple board and start sawing. You'd just go to your cabinet_template.yaml file and change material=oak to material=maple. You don't have to estimate how long it will take to build the cabinet because it's already done, you just ran build_cabinet.sh.
If it's not you copy & modify a snippet from StackOverflow, or call a library.
It's not strictly correct that most software is open-ended research. Most economically valuable software is open-ended research, because otherwise a solution already exists and you don't need to pay a software developer to write it.
Software is a somewhat unique industry in having zero cost of replication. The only other industry like this is creative & artistic pursuits (writing, artwork, music, film, etc.). That industry is also notoriously difficult to estimate and tends to be an area where the the greatest works of art are done when they're done. (How long have people been waiting for George R. R. Martin to finish A Song of Ice and Fire?)
Even if the job was incredibly small and predictable, my point is that he would likely not be prepared to sign off an open ended project with an uncapped budget.
I find it kind of funny how both sides take an "it's just not realistic" approach to each other's complaints and then the compromise is usually "You pretend you can make an estimate and I'll pretend to believe it".
>since that's the sort of repeatable job that can be estimated
To some degree and depends on the job. Start opening up walls in an old house and tell me how cookie cutter it is. When I used to run shipyard jobs on offshore drilling rigs, I can tell you that every single job ran into all sorts of problems that were individually not predictable.
It's also just nonsense to think that most software is this uniquely open-ended snowflake. Most engineering jobs require problem-solving that can be affected by all sorts of things in the physical world--plus you sometimes just have to build and try out designs and maybe they work, maybe they need to be refined a bit, and sometimes you have to toss them out and start again from scratch.
I think that's the parent comment's point -- there are rabbit holes and yak shaving in old houses, but trades still estimate work on them all the time (and sometimes take the risk of binding quotes).
You’re right, there is a high variability in the complexity in physical tasks just based on factors you cannot predict. The quality of things that impact physical work versus software is fundamentally different.
I see people on here post construction projects as the poster child for things that can be estimated with traditional project management techniques. This is wrong because most construction projects come in over cost and over schedule.
This can be because materials costs change, labor costs change, the soil at your site may not be what you expected, weather impacts, and stakeholders may demand something different after the project is estimated.
Although these are unknown quantities, these factors existence are known. Whereas in software, you’re more likely to encounter unknown unknowns. These are factors that nobody thinks are even possibilities until you run into them because nobody has done anything similar before. An example from my experience is when I was working on implementing a marching cubes algorithm on a GPU, we did research to make sure that you could do it. But when it came time to implement, we found that the cube data was expressed as a set of faces, and not a set of vertexes and no efficient algorithm existed for that expression of the data that could be mapped to a single instruction multiple data paradigm. We could not preprocess the data to convert it to vertexes because it would not offer any speed up if we did. To implement the acceleration we wanted, you’d have to change how the data was stored in the application. Which is a very different project with very different timescales. We ended up abandoning the effort and moved to other work. This happened because we made a discovery that no one could have predicted, in the implementation, that had an unknown impact.
In physical projects, you might find there are a lot of unknown things that can happen, that you may not predict. You can certainly reduce this by bringing in experts to a project, people who have seen much more than you. Certainly there is still potential for black swan events. But these are rare. Where as in software, the experts you may bring on may not necessarily know more than you, you are frequently your own expert. Black swan events in software projects are common.
But they were surely predictable in aggregate, no? There's only so much volume in the house and a volume will take a bounded amount of time to deal with, even if the bound is very large. An oceanic vessel has a maximum amount of mass that can be added or removed, and every unit mass will take a bounded amount of time.
This isn't the case for development. There is a bound to how much code can possibly go in a project, but there is no bound for how much development work goes into any unit of code.
>But they were surely predictable in aggregate, no?
Only in the sense of jobs seem to take at least twice as long as they "ought" to. The volume of the vessel has very little to do with it. In a refit, you know some things going in--which may indeed be relatively straightforward (so long as the right parts arrive on time)--but for bigger jobs lots of things crop up including resource conflicts and parts that need to be ordered.
If software projects added more slack, they'd be more likely to be completed on time too. There's nothing magical about software. And, in fact, software doesn't need to deal with a lot of the quirks of physical systems.
Basically the author is lamenting that estimations are NEVER accurate enough. I've not known a dev who made accurate enough estimations to avoid a crunch.
It does seem like the responders have a little bit too much sunk cost in the whole famously fraught agile thing to understand critique of it unbiased. It's fine in theory but does have flaws.
One flavor of agile (XP) says that estimates longer than 3 weeks diverge from reality too much. If you want to be able to estimate accurately, you must break it down until every task's estimate is less than 3 weeks.
That doesn't give you an accurate estimate, but it does usually give you a less inaccurate one.
I'd probably argue that any accurate estimate almost has to have some slack built in. For any task of any complexity, especially one requiring coordination, things rarely go more easily and quickly than you expect. Far more likely, you run into problems or you really need to talk to a person about something and they're on vacation.
> I've not known a dev who made accurate enough estimations to avoid a crunch.
And yet after three years at my first job, I could easily do a quick breakdown on a project, assign a three level complexity to each step (easy, medium, hard), put an amount of days corresponding to the selected complexity (to be reviewed after each project), add 25% for unforeseen events and contingency and my estimates were accurate enough most of the time.
It is not difficult to properly do rough estimates. Most developers just refuse to invest the time necessary to do it properly and sadly for some as the answers here already illustrate I do believe it’s because they think of themselves as special snowflake rather than team player.
> If developers have to be the ones carefully calculating estimates and managing expectations then there's really no purpose to middle-management.
Op is not saying that having estimates is useless, but rather it should be a management task rather than a developers burden.
To me, that sounds even worse, but I do think devs are expected to do wider and wider tasks across the business. Their post reeks of resentment against management and PMs and I don't blame them. Especially in my experience, I have worked with PMs that I have no idea what they do as devs need to step in to do all project management related tasks themselves
Just to be clear, most of the time “management” comes to developers with a proposed timeline, there’s significantly more complaints than if the engineers are asked to estimate.
Asking the builders to estimate the time they need is a good sign of high trust in that team. It’s healthy. We should want more of it. The business’ side of the commitment is to understand that it is an estimate and should be treated accordingly.
Exactly this! Most managers are not expecting the air-tight rocket science that most software devs seem to insist is absolutely necessary.
I was tasked with creating an entire email campaign system (like mailchimp). But I had only 3 weeks to do it. I made the deadline no problem, but the system had numerous obvious issues.
My boss didn't care, because the task was not about delivering a perfect email system. The task was getting something useful in the hands of customers rapidly.
If a feature starts getting heavy use, then more investment is justified.
>> it is an estimate and should be treated accordingly.
That's where it all breaks down. What does "treated accordingly" mean? In reality it means the estimates can't be used for anything because they are always unreliable. There is no such thing as a reliable software estimate, ever, unless what you're doing is so repetitive that you're about to be replaced by AI.
But that's not why people ask. Instead, half the time you give an estimate it gets immediately turned into a deadline, even if you were promised it wouldn't.
The top comment on this thread is naive. The reason devs hate giving estimates or point blank refuse is because it is meaningless, that's not how the software business works. Analogies to plumbers and builders just reinforce the naivety. Guess what, blue collar work is only predictable for as long as it's highly repetitive which being physical in nature, it often is. The moment what you're asking for is "build me an underground railway using the latest technology" it turns out construction estimates are worthless too (see: Crossrail).
One reason tech firms crush their competition so reliably is they don't have an estimates-deadline culture, because they're run from the top by programmers who understand intuitively why they're pointless. Instead developers are incentivized by equity, bonuses etc to do the job as fast as possible.
Estimates are about relative comparison, not specific accuracy. If you tell me one item will take a day and another will take 3 months, you can be wrong by several multiples and the information is still useful.
It’s less that I now have a specific endpoint, but rather I have a target (or multiple targets) at which I can decide if we’re on track and, if not, how I may want to change previously made prioritization choices.
If you tell me a day, then 5 days later if it’s still not done I may not rip you for being wrong but I may decide it’s time to pull the plug on that initiative since it no longer meets the criteria under which we agreed to do it.
I’m all ears for another proposal, but “just keep building stuff with no regard for how long any of it takes” is not much of a strategy and doesn’t reflect the way that decisions have to get made in the real world with limited resources.
Estimates for relative comparisons are more justifiable but rarely used, in my experience. And you can't bend on that stuff. The moment you say (to non developers), OK, I'll estimate X for task 1 and Y for task 2 they think, ah ha, if you can do estimates for this you can do estimates for all tasks.
Also and I cannot stress this enough, if you can make any estimates at all you're playing on easy mode. If the stuff you're programming isn't entirely rote boilerplate then you cannot estimate anything, ever, because your error rate will be measured in orders of magnitude not multiples.
Classic scenario: "how long will task X take?" ... "hmm well there's an open source library that claims to do X here, looks pretty easy to use, so maybe two days?". Some time later: "why is X taking 2 months and not 2 days?" ... "it turned out the library has some non obvious architectural limits we can't fix and which kill performance for our use case, so we have to write our own from scratch".
That sort of thing - you thought you could rely on another component and then you can't - can happen at any time, to anyone, and renders all estimates based on the assumed viability of that component meaningless. But you're always building on third party work. If you have absolute confidence in all your dependencies it, again, means you are doing repetitive work that isn't much different from your last task. Such works exists, I guess a lot of frontend dev falls into that category, but it's also highly susceptible to being automated away by better frameworks or AI.
>> it doesn’t reflect the way that decisions have to get made in the real world with limited resources.
My experience was the opposite actually, the big successful companies do exactly that. They may cancel projects if they seem to be no hopers, but they don't try and estimate up front how much a project will take because they don't think in terms of projects at all. If something is important, it gets staffed until it either stops being important or the team is beaten, at which point it gets unstaffed. If progress isn't fast enough they add more headcount. Everything is thought of in terms of relative resource flows, not timed Gantt charts.
> In reality it means the estimates can't be used for anything because they are always unreliable.
This seems like throwing out the baby with the bathwater. An unknown thing can still have bounds on it: I don't know exactly how far it is to the nearest supermarket, but it's further than 50m away and closer than 1km away. That's useful information: I know I'll be able to walk that distance, I know I can probably carry a reasonably heavy load back, and I can roughly estimate how long I'll be gone.
Fwiw, I agree with you that estimates are too often used badly by management to set up deadlines rather than genuinely make planning decisions, but I also think developers often fail to communicate their confidence levels when giving estimates. There is a material difference between "I know exactly what the problem is and what needs to be done, this will take two days" and "this looks like a pretty minor issue that will probably take two days, but it's touching a system I'm less familiar with, so it could end up going on a lot longer if strange things are happening there".
If you've worked in the industry long enough you work with a PM who will ignore everything you say and ask for a delivery date, and then ignore everything you say until you give the date that the management or client wants. What they won't do is give you the resources and support you need to hit the date.
One thing I find frustrating is that non-engineering people in tech companies don’t get held to as high standards of productivity because it can be harder for especially engineering founders to evaluate whether they are doing a good job. This can lead to people thinking those roles are “worthless” and then Continuing to have low standards for those roles.
The best combo project/product manager I had didn’t have either official title, but we were on an internal team and somehow she always managed to make sure people around the company knew what we were doing and delivering, pester people so that no tasks were dropped, etc.
Interestingly, she did most of this by putting tickets for projects into Google docs, and otherwise didn’t buy into any of the standard project management dogma. She had an ear/nose for who might want to use our stuff and was relentless at getting us on internal presentations of any teams/groups that would be interested.
That sounds great and all, except that estimates are usually wrong. It’s a real problem when estimates are treated as deadlines, which is exactly what you’re describing.
this assumes that there’s a buyer of software, and that the buyer decides which software to build. In those cases estimates might make sense
But in many other instances (product/engineering led companies, start-ups) software isn’t built for a buyer, it’s build for a user - and a user doesn’t care if it took 5 days or 15 weeks to build their favourite feature. What they care about is that you build features those users actually want.
In those cases estimates form part of a whole basket of mechanisms and rules designed to ‘shield’ developers from users and vice versa, which imo is among the biggest mistakes one can make in such software companies.
Give them a break, they're venting but they're raising valid problems within the workplace (don't go by the headline alone). I've personally been in this spot and had to seek outside help to identify issues within my workplace!
They key is negotiations, and both parties being able to balance a moving target of requirements and scope. Large software development is not a determinable thing as much as we would like. Perhaps ChatGPT will solve this?
>> I wonder if the person asking the question would be happy to let someone do a job in his home with an uncapped budget and timeline?
Even this is not always accurately determinable! If you've had work done in your home, or heard anecdotes from others, you would realize this.
Not surprised this discussion devolved into this bizarro world point so quickly. The developer says “there are too many unknowns to provide any sort of estimate”. The manager retorts “but we need estimates!” As if they expect the developer to say “oh, you need them? You didn’t say that. In that case, the unknowns are now known and it will take four days, three if you remove this inconsequential detail that nobody cares about”.
Unfortunately this bit of illogic is so common that I know it’s a waste of time to argue, so I’ve learned how to figure out what they want the estimates to be, give them that, miss them (just like everybody else does), and then get to work on actually providing the feature.
For these I like to say "by the time I can give you an accurate estimate, I will be done with the work."
Seriously though. It's always a time tradeoff. How much time do you want an expensive developer investigating how long this problem will take. Will they learn something they can use during development.
I always follow up my above quote with the unknowns and how hard I think they really are; what are the possible risks; _and_ I include the experience of the developers working on the problem; our ability to bring in other devs who understand this better etc etc. I like to make sure whoever is quoting this problem out understand everything as well as we can in the given amount of time.
Is this not what estimating and refinement means to developers? I’m always pulling a number out of my ass, but that number is generally informed by current and past work, unknowns, and a “gut feel”.
If that number is too high, then that’s a sign to management that the work needs to be changed. The only way to determine that is to estimate it then make that call.
> and only for "body sbop" work that is the same as previous work.
That’s absolutely preposterous. As a senior engineer, I have seen enough that I have at least a high-level idea of what most of the dev tasks and stories I’m estimating are. I don’t always have the specific details that may change the estimate, but I also have a whole team of other people who have seen even more things and can inform that opinion.
The point of refinement sessions is to share this information with everyone else so an estimate can be created and if needed, tasks can be better defined, split up or merged.
> And only works if management wants an estimate. More commonly, management wants a lie they can tell the client.
Wat? Estimates is used for plenty of reasons that have nothing to do with external clients.
How do you think your manager, their manager, etc knows what is going on and approximately how long it’s going to take? They’re going to be far more disconnected from the work than the people doing it day to day, so they would generally have no clue how long something is estimated to take without someone telling them.
> The developer says “there are too many unknowns to provide any sort of estimate”.
You are misunderstanding your
job. You are a subject expert and paid accordingly. Answer with what the unknowns are, an optimistic and pessimistic take on them, what you could do and how long it would take to lift them.
If you don’t, your boss is going to ask you for exactly that in a roundabout way and micromanage you because you are forcing them to do your job.
It still baffle me than approximately half of developers don’t understand things which are obvious to any seasoned blue collar worker.
Blue collar vocational training is more mature than software development vocational training. It's typically taught by people who have professional experience, and there may even be an apprenticeship period where people are explicitly learning from an expert in their profession while on the job.
In software, on the other hand, we often have most training being taught by professional academics who do not understand and are not qualified to teach the parts of the job that don't have a straightforward academic analogue. Combine that with a business environment where things like budgets and cost control are often only theoretical, and all but the most gross liability for defects due to poor workmanship has been waved away by EULAs, let it marinate for several decades, and... yeah. Not only do people not know these things, but I'm not entirely clear on how or why they should be expected to learn them, either.
I mostly agree, but if it's a technology I've never worked with and I'm asked to do something I've never done before, I'm not an expert. That doesn't stop them asking for estimates, which turn into deadlines.
I just say that to my boss: "this would involve using a library/tool I've not yet used, so I cannot give a good estimate until I've spent some time digging into it. Right now I don't know if it will be easy or very difficult."
If he really needs an estimate he'll tell me to start digging into it right away, and I'll let him know after a bit of time how it's going. If there are still large uncertainties and my boss still wants this completed by some specific date, we might try to explore alternative solutions.
Yes it boils down to, if the person asking for estimates understands the problems with them, it's safe to try. With bad managers it's better not to give them as they will be abused.
In the above case, I didn't get that time I would need, they wanted the estimate to include that time I would need to dig into it. But that's above my estimation powers. I didn't even know if what they asked was possible at all.
They went and asked someone else who did guess some number of weeks, but the work never got priority so we still don't know.
1) Without knowing further best case is it takes N time, worst case M time, and if there are certain obstacles it could be completely unfeasible.
2) To get more accurate time and to understand whether it's feasible I would need O time to figure that out first, and I can give you the more accurate estimate then.
This seems unexpectedly optimistic, to me. I'd think: best case = it's already done; worst case = it's impossible, but we don't know that until we've spent too much time and money on it; the "certain obstacles" are almost always uncertain or completely unknown; and the "O time to figure out [how much time it will take] is also unknowable until the "figuring" is already complete. Now, if this were manufacturing with well-known processes, then I feel I could use your formula.
As a manager that's all I ask. Give me an estimate of the knowns and an estimate on how's long you need to figure out what is realistic from the unknowns. Just let me know what your next move is.
The problem is when we assume everything is a known. I try not to do that and I try to set that expectation with my manager. If everything had an easy answer we wouldn't need highly paid problem solvers.
Can you do it? Provide an estimate that's accurate and stick to it with the routinized scheduling of a blue collar job?
It's not an apples to oranges comparison. You give me an architect's blueprint and guys who build to the legal building code, and I could probably get a house built. Expensive and painfully, but it can be done.
I don't even know what the coding analog to an architect's blueprint is. They are fundamentally different jobs.
You just made two contradictory claims: (a) that your job is to produce estimates; (b) that the developers who you work with are the ones who need to "do" the estimates.
I've met your sort before. You're a net waste of resources in an org: instead of developers getting things done, they're dealing with you asking them to do your job for you, i.e. produce estimates for the things they'll never actually get around to doing (because of producing estimates).
Your username would be accurate if s/My/Everyone's/
I produce estimates and coordinate eight concurrent projects involving ~70 people in cross functional teams. At some point, the actual domain experts have to give me their inputs so I can consolidate and prioritise things. Nothing contradictory here.
If it helps you sleep at night, sure, assume I’m a net waste of ressource. Good luck fighting with the other seven teams for some time with the shared senior UX expert.
Are you sure you’re not mistaking guess=deadline mode for actual estimating? And selecting among those who can meet deadlines.
If you rarely eat your time buffers, great. If you do that often, maybe you just coinflip-fix deadlines this way and maneuver between them under the guise of successful estimation.
It's more likely than not that everyone is giving the wrong estimates i.e. just claim things take 3x as long, skip out on anything too hard and produce lower quality.
You always get what you ask for. Like sure I can estimate. If you ask for a web server you'll just have that. It won't be secure. It won't be fast. It may work. It will certainly pass the tests you think of.
Is that actually good? No. The real estimates get rejected. Everyone cuts corners. When something does happen e.g. a hack occurs the engineers not these managers get fired :(
> You are misunderstanding your job. You are a subject expert and paid accordingly. Answer with what the unknowns are, an optimistic and pessimistic take on them, what you could do and how long it would take to lift them.
My optimistic/pessimistic takes at my last job would look something like "roughly a week to roughly 2 years"
Depends on how deep the yak shaving goes, and you wouldn't know until you got halfway through fixing the problem.
Often I could take an educated guess as to which side of the scale it came down on, but I've definitely started pulling on a string inside of the codebase and it just kept going and going and going. All of those baked-in assumptions that other developers unconsciously made over the course of a decade can pile up into a mess that you just can't implement the feature/bugfix without doing a pile of potentially breaking changes that need to be shipped in major versions.
Generally if I worked the problem for a week then I could give you a much better estimate of when it would be done, and often that estimate was "it's done now" after a week. But if I hadn't looked at that particular subsystem for a few years, I couldn't trust my memory of it up front and had to do exploratory surgery first.
(Really exploratory surgery is a much better metaphor for what you're going to be doing, and surgeons can routinely find surprises -- "oh look that barely perceptible shadow on the x-ray turns out to be a stage 4 tumor...")
Yes, the codebase was horrible, yes it needed to be cleaned up, no we didn't have the manpower to do that and it would have taken me 10 years to rewrite it all.
> Generally if I worked the problem for a week then I could give you a much better estimate of when it would be done, and often that estimate was "it's done now" after a week.
You're lucky if your management chain allows you to work on problems for as long as a week without giving them an estimate first.
> You are misunderstanding your job. You are a subject expert and paid accordingly. Answer with what the unknowns are, an optimistic and pessimistic take on them, what you could do and how long it would take to lift them.
Add a small feature to an existing system. This estimate can be anywhere from several months to a day. Most of the error bars are org and system specific rather than anything to do with the feature. The exact same feature built by the same person in different orgs and different systems can be opposite ends of the range.
Seldom do people have the contextual understanding to provide an accurate narrow estimate, particularly since both org and system are in flux. They change constantly. Unseen landmines potentially await in both.
> If you don’t, your boss is going to ask you for exactly that in a roundabout way and micromanage you because you are forcing them to do your job.
And such orgs take forever to get anything done.
> It still baffle me than approximately half of developers don’t understand things which are obvious to any seasoned blue collar worker.
Software is probably the hardest kind of engineering you can do. All the system constraints are virtual. Just consider comparing a dirt ground for consistency and behavior vs libc... 304 stainless vs react.
You can go deep in both, but a shallow understanding of one takes you much much further than a shallow understanding of the other.
> Add a small feature to an existing system. This estimate can be anywhere from several months to a day.
That’s extremely untrue in my experience. Sure, hard to interface with systems are a thing and organisational roadblocks happen but an onboarded senior team member can tell you what’s going to take days and what’s going to take months or they have no business being senior in the first place.
I feel like my answers in this discussion have been particularly harsh but I would really like people calling themselves software engineers to start acting like some. I think my view on this subject has been strongly shaped by spending most of my career doing software development for industrial companies: people who claim software is particularly complicated just don’t understand and respect how complex the rest of an industrial project is.
You're making an unwarranted assumption here. There's many pieces of software that have no development team at all. There's no one to ask; the last person to touch this software left the company three years ago. All they can tell you about it is how they use it now.
It's likely there's at least someone that can explain their business process to you, but identifying them and receiving enough of their time to piece the picture together is another story. And maybe, if you're lucky, that person can tell you what the business process was at the time the software was written, and what has changed since then.
But this process can take weeks, and most of the time you're expected to provide an estimate before even talking to anyone from operations.
Sure, for internal software with an active maintenance/development team, informed estimates can be given with a relatively high level of confidence. But as an outside consultant working with non-software companies, all estimates are completely made-up based on your reading of the buyer's budget: there are no other data points to base your estimate on.
Working in the field, I’m fairly certain you already know that this kind of job always starts by a paid diagnostic and scoping phase which is there to give you the elements you need to do an actual estimation. It’s either that or you are lucky to still be solvent.
There are lots of places where you get 0 of that before providing an estimate.
And yes they are still solvent.
As others mention there are lots of unknowns in an organization.
I've been in places where the other team involved is across the globe in a different timezone and isn't even actively willing to respond and yet we continue...
There are also cases where lots of parts are outsourced and you're lucky to get a reasonable response i.e. changing part x requires vendor a, b, c & d, which have an SLA 1-8 weeks and so you're already starting 4-32 weeks without even working anything else out. If you're really lucky it could still all be done in 1 day.
Not sure if you're harsh as you say or just lucky. Not everyone and every team is well organized.
> Answer with what the unknowns are, an optimistic and pessimistic take on them, what you could do and how long it would take to lift them.
Yeah, most of the time my boss just needs to know if this job is a few hours, a few days or a few weeks. I never answer with exact numbers, but a range. If there are some major unknowns I let him know. If I made some important assumptions then I let him know.
Sometimes I'm wrong. Most of the time my estimates are sufficiently accurate.
Sometimes the customer really needs this done before some deadline. Then it often turns into "how can we make this happen before then". After a bit of back and forth with some added assumptions and cutting functionality, we usually end up with a solution that can be realized in time.
Thank God somebody said this. When I was working as an IC, I would be very communicative, highlight the unknowns, and get buy-in from my boss. If a surprise came up, I would make sure everyone knew the game had changed.
As a manager now, I get scoffed at if I even hint at a deadline. Excuse me what? We have a limited budget, a limited runway, and we need to prioritize which product bets we are going to make. This seems entirely logical, but a lot of devs I work with now find it impossible. They just enjoy tinkering away with whatever thing interests them, and delivering value only means what they personally find valuable.
Personally, I find it a bit pathetic and childish. I say this as a long time software engineer who knows perfectly well how this business works.
Will Wright had a quote on this … if you don’t have any constraints you are just playing.
That said, however, The problem I run into during practice is dependencies— people want estimates to integrate into systems you have never heard of or used before. Not only that, there is no in place architecture or design before estimation. Imagine a construction company estimating a house without knowing how big it is or without having blueprints.
Often times software development is designing and architecting the system — not just implementing it. And that is the expensive part.
Say that the estimation can depend on the complexity of the system. I would think worst case is when the system has these A, B problems, then it would take N time. Otherwise best case it could take M time. If we want more accurate idea, we can spend O time on it to figure out what it could be.
I would say this is being negligent if it is generalized. Often times architecting a system involves meetings with multiple parties — stake holders , legal counsels, and months or years of planning ahead to give a proper estimate for a large system. Time and materials is the right way to do this.
However I can see this working in a very small scenario.
> Personally, I find it a bit pathetic and childish. I say this as a long time software engineer who knows perfectly well how this business works.
How does this business work?
Why is software special? This happens the same in construction and other industries. The estimates usually make 0 sense and there are often huge delays. Same as store openings, film production, etc.
There's a problem with tinkering and devs going off in the wrong direction but on the flip side when people say that something is done in an hour or a day often never happens. Managers and agile practices that want tasks broken done to the smallest atom just messes things up. To make things work product often never has accurate requirements to that detail either so that complicates things even more. The requirements change on the fly.
I don’t see how it’s unreasonable that someone would want to make plans based on the development velocity of software. Just because it’s actually hard to estimate some things accurately doesn’t mean it’s not valuable from a business perspective. The original post was someone saying that they should never be asked for estimates because every single thing they make is a unique snowflake of unknowable complexity, which is absurd. There are definitely features which an engineer can easily estimate delivery timelines on, and even knowing ballparks like “this is hard (as in weeks)” or “this is easy (as in hours/days)” is super valuable to communicate to actual business stakeholders.
In a healthy culture it should be possible to have a back and forth about how accurate an estimate is likely to be, but when engineers insist that all estimation is useless you can imagine why other stakeholders may lost trust.
This is why it’s a good idea to have ranged estimates. If a simple known thing with zero unknowns has maybe a 200% margin (1-3 days) then a more unknown task can have 10x as wide margin like 1 to 20 days.
As a developer I never get “that’s a useless estimate we can’t prioritize based on that/tell the customer that”. It’s a perfectly good estimate. It tells everyone there is a lot of uncertainty. The low lower bound indicates there is a chance it might turn out to be cheap (so maybe worth taking a gamble and trying it and giving up). The big variance indicates it could/should be given some time to prep/investigate in order to narrow the estimate.
When there are unknowns, make a wide estimate. Convey the uncertainty. If the unknowns are ridiculous, make a ridiculous estimate. Even “5-200 days” is useful because something else with less uncertainty can be chosen instead.
The SO question addresses this, though: after giving an estimate with a range, the developer is pressured to narrow that range, mainly by pulling in the farther-out date in the range, and then the developer is blamed when the project inevitably comes in late because of that.
Maybe the solution here is "stick to your guns", but that can have negative career implications.
That’s perfectly expected. But the key is: narrowing the estimate isn’t free.
You can’t pull a different estimate out of thin air. And zero people will expect it.
The developer needs to say “I can say 10-100 days right now in this meeting, but given just 1 day of investigating, I’ll narrow the estimate range by an order of magnitude”.
I'm running through a similar situation now, where I'm somewhat adjacent to a client and a dev team. The client has a list of tasks he wants done, and he'd like banded estimates for the effort the dev team thinks it will take to complete each.
The developers are obliging, but one of them complained to me privately "why doesn't he just give us 2 months of budget and we'll get through as much as we can?" - which is an interesting statement. From the developer's perspective, this is ideal. They are very good and work hard and have integrity, so I'm sure something like that would work, from their perspective. They can just 'go' and get stuff done.
From the client's perspective though, it just raises a bunch of uncertainty. He wants to know what he could get out of his 2 months of development budget. If one task took all 2 months, and the others (15 in this case) didn't get done, is that OK? What if some important task wasn't that much effort? You'd potentially want to priortise that over a bigger item.
Just saying 'give us two months of budget' doesn't take the other party into account, and what they're trying to balance or plan for. I've sat on both sides of the table, and honestly just asking for a budget with no actual commitment to completion feels selfish in my opinion. Discounting their opinion because they're 'business' people and not developers, or treating them like an enemy to fight against won't win you any friends or long term relationships.
> Just saying 'give us two months of budget' doesn't take the other party into account, and what they're trying to balance or plan for. I've sat on both sides of the table, and honestly just asking for a budget with no actual commitment to completion feels selfish in my opinion. Discounting their opinion because they're 'business' people and not developers, or treating them like an enemy to fight against won't win you any friends or long term relationships.
The irony is that at least some of the time, a "safe" estimate will actually make the project take longer. Work naturally expands to fill a deadline, and not having a deadline but instead frequent progress checkins with an expectation of foward progress can result in more getting done, because it changes the conversation from "how at risk is the deadline" to "how much happened in the last week and what are the next steps", forcing progress to happen continuously rather than one big bang at the end.
I also often hesitate to give estimates / establish deadlines because it tends to create an unhealthy confrontational atmoshepere, where developers are incentivized to not incorporate feedback or make any changes even if they are the right thing to do, because it would put the deadline at risk (but if they incorprate allowances for these things during estimation it's considered sandbagging).
> a "safe" estimate will actually make the project take longer
That’s probably the most important thing I’ve taken away from software project management techniques - everything they do is hurting them, not me. If somebody insists on shooting themselves in the foot, and they’ll shoot you if you try to stop them, you just sit back and let them shoot themselves in the foot and shrug your shoulders when it happens.
It’s easy for this honest mentality to turn into malicious compliance. If you’re actively fulfilling that prophecy, this won’t just impact a manager or a PM, it will absolutely impact you as well. I’ve seen it happen many times, obviously this goes beyond just being wrong or missing deadlines but eventually it will come back around to bite you.
At the end of the day, these estimates are helpful but maybe not in ways that are intuitive from a developer’s perspective. They’re useful in weighing priorities, judging the uncertainty, and identifying where to fill the gaps in understanding, time-boxing research and scope (e.g. do we really need to spend 6 weeks reading release notes?), and help communicate to outside stakeholders. We see the successes of this all the time with big game releases, product launches, movie releases, etc. It’s a high level measurement of progression and devs need to be a part of that planning process. It gets easier for senior engineers to do this efficiently with experience. It’s not necessarily an exercise that a more junior engineer will a) get right consistently or b) think is valuable at all.
In my experience there's two main reasons that orgs outside engineering / product want estimates:
1. Coordination of external dependencies. Stuff like product marketing, sales training, documentation, etc. as well as sometimes things external to the company like customer commitments or sales commitments. These are for the most part legitimate, although in many cases the deadline doesn't need to be known at the outset of development, and too many of these is a very unhealthy sign.
2. Accountability / improved productivity - This usually manifests as an exec feeling that engineering is running too slowly / inefficiently, and by asking for deadlines (and often compressing them to "push" the team) they can improve productivity. These are almost always in my mind completely counterproductive, and there's much better ways to accomplish this.
> From the client's perspective though, it just raises a bunch of uncertainty.
It may feel good to have some of the times quantified. But... If your estimates are literally fiction. It won't stop being fiction in front of the client. You should really spend that energy that went into the estimate for something useful instead.
Not to say you can't derive some useful information from a short planning session. Just don't expect anything other than thumb to the wind guesses.
I'm fairly certain that you aren't accounting for uncertainty properly, especially with long-tail events. Let's say you have a ticket to enable a config flag. That's low risk and relatively small. There's still a risk that there's the flag hits a silicon bug. Does your estimate reflect that risk and is it useful if 95% of tasks don't hit those issues? Consider also that estimates delivered by contractors or even other teammates will not reflect these risks and thus you look correspondingly pessimistic or even lazy.
For context, my team is in exactly this situation with several silicon bugs we found last month.
Assuming you’re pretty close to the hardware? As a standard backend web service dev os/library/device bugs are incredibly unlikely and usually defective application software is to blame. Not saying it doesn’t happen, but I can’t recall in recent memory this sort of thing happening as high up in the stack I work.
That's just the sort of stuff I see, but long tail issues that take weeks or months to diagnose can happen anywhere in the stack. Sometimes it's a hardware issue, sometimes it's a compiler bug, sometimes a VM bug, a missing browser feature, etc. Any of that can crop up even with "trivial" changes. I'd argue that it's not useful to include that in your estimates, but they're nevertheless possible and need to be incorporated in scheduling.
I probably should have said hardware bugs, as only one is in an IC (specifically an IP block). It manifests as small (but out of spec) timing errors across a network. The rest are things like improper layout, faulty components, wrong value passives, etc.
> When there are unknowns, make a wide estimate. Convey the uncertainty. If the unknowns are ridiculous, make a ridiculous estimate. Even “5-200 days” is useful because something else with less uncertainty can be chosen instead.
As a PM, I couldn't possibly agree more with this. When you give me 5-200 days instead of "we can't estimate this", you've started the beginning of a productive conversation. I can ask you questions about why the range is so big, and you can explain to me what steps we can take (research, narrowing scope, etc.), to reduce the range.
I’ve tried this. Inevitably the response is: “We need to book UAT testers for a specific date.”
Ordinary LoB staff apparently are superstars that have their schedules locked in months into the future, but consultant SREs costing thousands a day can have their time wasted on a daily basis trying to determine how long it will take to complete a task that hasn’t even started yet.
This has literally been my last two months. The dev team lead is asking me — an outsider — how long it’ll take his team to finish something. Not because I know better, but because he doesn’t know either and prefers to make it my fault because I’m expendable.
Well, analysing the situation that seems like quite an easy one to negotiate through:
1. Talk to the person who'll get angry at you when the UAT testers are brought in at the wrong time.
2. Make it clear to them, with a paper trail, that this is an impossible task and you can either book in a best-estimate with the likely outcome that the UAT testers will arrive and there will be nothing to test OR that you can book conservatively and there is a risk the project will be delayed waiting for UAT. They can pick which risk they want to bear.
3. Put in formal feedback, as loudly as possible, that booking in UAT for a specific time introduces obvious risks and someone needs to optimise the process to be more flexible otherwise it will delay projects & waste the businesses' money.
Deadlines are a reality in all businesses (Real deadlines or made up ones). But if you have a thing to create that takes 5-200 days what will you do?
You can pull a different estimate of 20-40 out of your behind when pressured to narrow it, and make the deadline 40 days. That might be a 50/50 hit/miss the deadline, with a worse case of running over to the full 200 day worse case (or even worse - even the 200 was a guesstimate). In some cases that's fine too. The decisionmaker might just want a date with 50% chance of hitting because it's better to land a sale and apologize for the delay, than to not land the sale and go bankrupt.
The correct course of action though is to not set the deadline in that meeting. The 5-200 day estimate is a way of explaining "Either don't take this on at all, or at least don't try to schedule anything urgent based on it".
In that case as a developer I'd suggest "Let me look at this for one afternoon, and I can return with an estimate that's 1/10th as large. Then let's meet again ad prioritize some work, schedule UAT and so on". PM's often look really surprised when they realize that all it takes might be to check if there is a functionality available already or if it needs to be implemented - which represented the difference between 5days and 200. If I go off and find it's already there, the estimate is 5-10, if I go off and find there is nothing and it has to be built from scratch, it's 150-200. But without that knowledge: 5-200.
Or, if a 20-200 day estimate is actually due to real implementation uncertainty, I'd suggest "This is potentially large and uncertain. Let's try to break this apart if necessary into something that has a 10-20 day estimate and some parts that have a longer horizon. Schedule the delivery/UAT of the first functionality in 20 days, and we'll by then have a good estimate of when the remaining work will be done".
If someone in my organization would try to convince me to pull a stupid estimate out of thin air, then pressure me to haircut that estimate, then schedule something based on it (such as a delivery), then pressure me to work overtime to meet it, then one of us would be leaving the org very soon.
> what steps we can take (research, narrowing scope, etc.), to reduce the range
I quite literally quit my last job in February because of this shit[0]. Stop. No. These are not useful things for developers, much less senior developers, to spend their time on. 95% of my job ended up being coming up with estimates. My top deliverable to my management chain was estimates. They would only care about the actual development work insofar as it validated or invalidated the estimates deliverables.
Hard facts: If time is such an unimportant resource that you're willing to waste most of it doing "research" and worrying about scope, then you don't actually need an estimate. For reporting purposes, just BS one yourself, that's literally your job as a PM.
[0] It was well-paying, I was basically fucking around every week doing whatever I wanted, yet this issue alone outweighed all of that and was enough to make me quit.
Yes - it's better to be accurate than precise. If you said "1 to 20 days" and it took 15, you were accurate. Being too precise gives you no space for the unknowns to show themselves.
> Unfortunately this bit of illogic is so common that I know it’s a waste of time to argue, so I’ve learned how to figure out what they want the estimates to be, give them that, miss them (just like everybody else does), and then get to work on actually providing the feature.
I do that and I've become exceptionally good at blaming the business persons for the outcome, making them admit their guilt in public and all that in what seems a friendly, positive and constructive manner and using business motivation and explanation. They never understood what hit them. Every programmer can do that if he or she learns a bit of psychology.
I just confront business people with the results of their choices in a civil and polite manner.
If some manager made a promise to a client out of his/her ass without asking us, and if the PO pressures us towards meeting an unrealistic time-frame, when that explodes I just explain in a public meeting where upper management and all stake holders take part why it was wrong to promise that, using arguments and business considerents because bussineses people don't care about tech considerents. It usually results in that person admitting in public their mistake and even saying they are sorry.
I don't do that to humiliate people but to teach them to ask the team implementing the feature to estimate or to say if that feature is even reasonable or possible given current state and resources.
The estimates are useful even if they're wrong. Not all managers realize that, so they ask the question badly. And they never teach it to programmers.
The estimates aggregate to a number more accurate than each individual. The manager cares more about the sum total than each individual one. Many can even slip without affecting the end date.
This isn't so hard, if management and developers didn't treat each other as enemies. Each can get wrapped around the axle concerning each deadline, rather than the real goal of allocating resources across the entire scope.
It’s monumentally frustrating to be forced to give estimates too soon in the planning process. The compromise, I hope, is to end up in a place where most of the problem is known and some is unknown, and you get there by breaking down the original problem. Leave it up to the “buyer” to digest your best estimate into their planning process.
The response to "there are too many unknowns to provide any sort of estimate" shouldn't be exactly "well, we need estimates." But "okay, just do whatever" is also not a working outcome. Software is weird, yes. But other disciplines are able to provide useful planning information. Why are we unable to do that?
Most other disciplines have this exact same problem. Construction contractors for instance, are notorious for it.
The difference is, the law says if a contractor signs a contract to do something at x price, they HAVE to do so or suffer consequences. So contractors go broke/bankrupt, lose their licenses, or figure out how to bid correctly enough they can either weasel out of it if it goes sideways or they have enough wiggle room to deal with the inevitable problems.
Software Engineering Consultants have the same situation. But directly employed software engineers are in a different position of course.
One thing that typically makes this worse, is the implicit assumption that planning/quoting/estimating is ‘free’ work, and since it CAN make things easier to deliver, therefore we should do more and more of it as, since it’s free, it will solve the problem no?
Other disciplines have problems with slippage, but none of my friends who are in physical engineering fields insist that they shouldn't even have to provide estimates in the first place.
Unfortunately this bit of illogic is so common that I know it’s a waste of time to argue, so I’ve learned how to figure out what they want the estimates to be, give them that, miss them (just like everybody else does), and then get to work on actually providing the feature.
It gets even worse when you are in a line of work where the estimates are part of some kind of sales pipeline. Then you aren't just estimating based on effort required, but also against a likely number that the customer will ever agree to.
App idea for tracking projects: Each individual developer can anonymously indicate how far along they believe a project is. For example, Developer A thinks the project is 60% complete, but Developer B thinks the project is 30% complete. Use algorithms to predict the completion date.
However, I don't think this idea would be very popular because it removes the manager's ability to control the presentation and play politics.
See in this case the manager has done a bad job, but the developer has also done a bad job, both in the exact same way.
The correct solution here isn't just to insist estimation is impossible and be offended that someone would ask for it - that just misunderstands how businesses work.
The correct thing is for either of those parties to say, "Okay, I understand it's not estimable at the moment. What are the steps that we need to take in order to get a reasonable estimate?"
A good PM will do this, but where there is a bad PM, a developer can step in and say the same thing. In your scenario, "The developer says “there are too many unknowns to provide any sort of estimate”." What the developer could say is, "There are too many unknowns to provide any sort of estimate - let me give you a few examples. <examples>. What I can do is come back to you after <amount of time> with a list of the unknowns and an explanation of the work required to evaluate them, so that we have enough information to estimate. Once you have that list, then you can decide if we want to proceed with the work to understand those unknowns."
I totally understand why developers get frustrated when they're asked to give estimates for things that can't reasonably be estimated with the current information available. But the flip side is it's also very reasonable for PMs to get frustrated and ignore developers who insist that estimation is an impossibility, because that's virtually never true - maybe it's impossible to do now, but there is a path to make it possible. If you work proactively to expose that path to the PM, it'll generally result in everyone being happier.
> There are too many unknowns to provide any sort of estimate - let me give you a few examples. What I can do is come back to you after amount of time with a list of the unknowns and an explanation of the work required to evaluate them, so that we have enough information to estimate
Sometimes we don't know what we don't know. If we don't know what we don't know now and if we don't know what we'll find we don't know later, then we don't know how long we will need until we'll know.
Uff. Sorry. If I had more time, I would have written a better comment.
Let's put it this way, then - either you're totally incapable of building whatever we're talking about under any circumstances, or, even if it has unknowns, there are steps you can take in the right direction to get started and move things along. You will learn things that will help you understand the problem better along the way.
If it's the latter case, then a good developer can describe the immediate steps they'll take and what questions they expect to answer. Whatever the case, there is always some concrete next step you can take to which you can assign a reasonable estimate of time. That might just be that you'll spend one day perusing some documentation, and the only outcome you expect from that is that you'll develop a reasonable enough understanding of how large/complex the documentation is, such that you can come back with an estimate of how long it will take to find and carefully read all the relevant sections of the documentation.
That's totally fine - if it's the sort of wildly unknown problem that you're describing, then I recognize we'll have to take steps to understand the problem. There's still never a case where it's impossible to create an immediate plan of action and understand roughly how long that will take (unless you are simply unqualified to take on the problem).
I had a boss loved the phrase, "a date for a date." He just meant that if you can't give me a date for when the thing will be done, then you need to give me a date for when you can give me a date for when the thing will be done. Can't do that? Date for a date for a date. However many levels we have to go, you're not getting out of whatever meeting we're in until you can give me a date by which you'll be able to come back and give me another date.
The problem is that in programming the discovery part is often 90% of the task. There can be situations where there is no 10% discovery phase and the rest of the time we’re banging out predictable code.
Right now I’m replatforming a legacy app. It’s undocumented, developed by dozens of staff — all of whom are gone, and full of crazy bugs and gotchas.
Out of the box it wouldn’t compile, the source code was missing files, etc…
The task is to find every such hidden issue, fix it, which then allows us to find the next hidden issue, and so on.
When we’re done finding issues, we’re basically done.
Can you tell me how many issues we’ll encounter? I can’t. I’m being perpetually surprised by the code base, finding random binary blobs that actually are from some other project we didn’t even know exists until that moment.
Fundamentally, it’ll take as long as it takes. The application is mandated by law, and it must be made to work. It’ll cost what it costs, and it’ll be finished when it’s done.
Instead of begging for estimates, a good manager would be bending over backwards to accelerate the process. Get in the original devs for a week under contract. Procure tooling that helps find bugs faster. Instrument the source servers to get better comparative info. Etc…
I’ve never seen that occur. It’s always just the same question repeated over and over as if that can make the unknown known.
> The problem is that in programming the discovery part is often 90% of the task.
That's exactly it. Scoping out the solution is 90% as hard as just doing it. By the time you've figured out enough of the details that you're sure there are no more surprises waiting... you're just about done.
I suspect that a lot of counter-examples people have witnessed are examples of poorly factored codebases. I've seen teams/projects where they estimated projects accurately because the "new" thing being created was a slight variation on a kind of thing they'd created many times.
For example, I once worked for a company that generated e-commerce sites for clients. They'd make a new one basically by copy/pasting an existing site from another client and changing some text/images here and there. Super predictable after doing this a bunch of times... but also a maintenance nightmare having dozens of slightly different codebases. And how do you iterate? They couldn't evolve.
Shopify ate their lunch. I bet Shopfiy has a really hard time estimating the time to ship a new feature. But they add it once for everybody.
When the discovery process is iterative, there is never a point at which you have an estimate for the whole task. Each new discovery step may be the last one or it might lead to more steps, and you won't know until you complete some step, look for the next one, and realize the project is done.
A knowledgeable developer can predict how this cycle goes and explain that an estimate of the whole task is impossible, and they will be quite literally and completely correct in that case.
> The correct thing is for either of those parties to say, "Okay, I understand it's not estimable at the moment. What are the steps that we need to take in order to get a reasonable estimate?"
Ah, that would be nice. In my experience it's more like, "Okay, I understand it's not estimable at the moment. Just make your best guess and we can revise it later. (Psych! I will hold you to this estimate like you were under oath in court!)"
This leads to things like the Scotty rule. Since management will always misuse, misunderstand, or misrepresent your estimate anyway, might as well look out for number one and inflate the estimate, just in case.
Let me genuinely encourage you to try to find a job at a very early stage startup. I hear these horror stories about PMs/managers/etc., but I've yet to really experience them. I suspect that's because I usually join companies of <50 people. The engineers care about getting things right, the PMs care about getting things right, and management is well aligned with everybody, because there's little to no hierarchy.
The only time I've really had management "misuse, misunderstand, or misrepresent your estimate" is once they got past the point of being several hundred people with the bureaucracy that entails. Then I leave and go to a startup that's <50 people.
In my experience good communication solves this every time.
MGMT: We need an estimate
ENG: I have several unknowns
MGMT: We really need an estimate
ENG: This is my rough estimate, but be aware that the unknowns could change this drastically
MGMT: Ok, what do you need to get a firmer estimate.
And so on.
As a project progresses, the communication between ENG and MGMT is crucial to ensure that MGMT are aware how those unknowns are affecting the project, MGMT will decide at every juncture if the project is worthwhile to continue to invest resources into it (assuming they avoid the sunk cost fallacy)
The best thing to do is to define tradeoff ranges and set the deadline, so that developers could try an optimistic path, learn what’s wrong with it halfway, and then choose a tradeoff that both seems not that bad enough and, when it fails, leaves time to rethink and use another tradeoff which is even good now, in a sense.
Because that’s what we do either way. Round numbers from ass make no science to plan and prioritize upon.
A general framework is that on each decision, including architecture, cost, time one at least adds the following:
1. Assumptions (including soft such as who will actually do it, resources are not equally relevant or productive)
2. Constraints (as agreed by stakeholders)
3. Risks
4. Clear scope
3. Alternatives and why they were not preferred
Business should be aware and own the uncertainties, or else the system gets biased towards sales people offering vaporware.
The original question was "Why are developers expected to estimate tasks at all?" @benjaminwootton gave perfectly sane answer to that question.
HOW developers should make their estimation is a totally different question, but it is not some arcane knowledge, and it should be an everyday craft for every mid-level engineer. First of all, we are doing those tasks by hundreds through our careers, so an easy way is to remember how long similar tasks took us last 10 times an produce a bracket, like in "from 5 to 8 working days, let's say 10 to be sure". Every business person will be happy with that answer.
OP says that every engineering task is unique as a piece of art. This is BS, IMO, except for the very junior engineer. Most of the time business wants some well-known and hundred times done things they see their competitors do. All the uniqness stems from particular codebase. If your codebase is a mess - you are in trouble, but it is you problem, not the general problem of software engineering.
In case I have some really unique task, what I should do is to say: "I cannot estimate this right now. Give me two days and I will get you an estimation". In those days I decompose the task into smaller, and divide those smaller tasks into "well known" and "unknown". Then I imagine (or even draw) a trivial Gantt chart and will have my estimation.
It is also totally normal to change your estimation during the work. You just should do that the moment you see a problem, not at the day of the deadline.
It's not just about external factors, you can't let you dev go dev for an infinite amount of time. Task should be done at some point and you should roughly know how long it should take.
If the expected ROI on a particular project is so razor thin that accurate estimates are needed, don’t even start. Be sure to do your due diligence and aim for at least 10X (ideally > 100X). If this is provably true, then accurate estimates will not be needed. If this cannot be concluded, simply live with the limitations of off-the-shelf offerings.
I think this is a big part of it. For every manager with an unrealistic schedule demand, there's a manager above them with an unrealistic ROI demand. The projects with accurate schedules and accurate ROI's don't get approved.
And giving people a precise and inaccurate estimate makes this better? A capped time and budget won’t make it take that amount of time or money. It takes what it takes. The complexity and amount of randomness in these processes mean any expectation has wildly large variance, with a long tail in the wrong direction. If you want home builder like estimates, build a home, not software.
> I wonder if the person asking the question would be happy to let someone do a job in his home with an uncapped budget and timeline?
The author made the claim that software engineering is fundamentally different from carpentry ("Almost everything a developer writes is unique, they have never built that particular thing before. We are not cabinet makers...")
But does that hold up? Many software projects do in fact have a similar analogue, and each house is unique and carpentry projects will thus run into unexpected snags.
He/she hasn’t been around enough to see these patterns yet. The amount of times I’ve had to implement security, rbac, rest crud, modeling, service integrations, etc are too many to count. I rely on packages that help reduce the boilerplate and because I’ve done that so many times, I can give you an estimate on roughly how long it will take.
The fact that the OP doesn’t see this is fact that they are too junior to be asked for estimates at all.
> - The buyer has to make commitments externally, for instance to customers, partners, finance, marketing, his boss;
Usually a business person makes some promise to the buyer or his/her boss that a certain feature will be done in X amount of time without consulting the team who does the work. The result might be overworking, resentments, stuff not being done in time and technical debt.
You can't have it both ways. Either you ask the developers for an estimate and you get ridiculed because how can developers possibly have any clue how long it's going to take them something? A day, a month, a year, who knows how much time this CRUD service will take! Or you don't consult with your developers, but then you've made an unrealistic promise with a number pulled out of your pants and you're the cause of all the suffering in your team.
No. Realistic is talking to your team exposing all the knowns and unknowns, refine the feature as realistically possible, while keeping the client in the loop, and once the feature is decently refined you ask the team for an estimation, take the higher limit and add a buffer for things that are not development related.
In the best case the feature will be done earlier, in the worst case you won't look like an idiot which promised things and didn't deliver and you won't compromise your relationship with the team. Compromising relationships with either clients or tech teams aren't good for your career.
More simply put, try to be as honest as possible. Not over promise, not under deliver.
One of my former girlfriends went to a business school and worked in banking. She thought people in IT have better salaries and she thinked tech people are a bunch of idiots and it's easy to push them to do things and extract a fat paycheck. So she did some courses on IT management and applied to lots of product owner and product manager role. After tens of interviews she hadn't had one success. To add insult to the injury, I showed her I make more money as a team lead than the product owner. Of course, the reverse is true and I've seen product owners being genuinely useful and having lots of business knowledge, some even programing knowledge.
Some people are just after the money, no matter what role they occupy in an IT company. And they tend to be the least useful persons who are blocking others to reach full potential.
The problem is the writer of the question is conflating two things. Their quote: "A. Give a very wide estimate with a lot of padding. B. Get pressured to be more accurate and to bring the estimate down..." The real problem seems to be getting pressured to reduce their estimate, they only summarize things as the problem being that they're asked for an estimate at all.
I’ve known people whose version of A was “between a year and five years” for something where five years was clearly a ludicrous amount of time to spend and a year would be stunning. In those cases, the problem is that they aren’t really bothering to give an estimate at first, so the pressure to give one continues.
Every project I ever worked on was more like “okay we already building it” and estimate was the last step after spending hours of discussions. My manager was typically had enough clue to estimate themselves too.
> I wonder if the person asking the question would be happy to let someone do a job in his home with an uncapped budget and timeline?
Huh? Every major contract job on my house and my friends' houses that I have heard of has taken longer than originally "planned". More often than not unexpected expenses come up along the way. The same is true for commercial construction/remodeling. That's just how these things work; pretending otherwise is going to be a huge source of friction.
I do wonder if developers would accept the same from people they "employ" to do a task.
I imagine sometimes I renovate my kitchen, an expensive task, within my budget but with not a lot of leeway, I can't renovate the kitchen twice for example. If the workers say they don't know how long it will cost and they won't provide an estimate to when ... I would not be happy.
Even creatives, writers, musicians, graphic artists have deadlines and must estimate their work.
But we feel that our work is special and should not be asked to give an estimate.
I'm not sure why you feel the need to falsely frame the point as a request for a blank check.
I get all the reasons you provided, but at the end of the day an honest discussion needs to ask is estimating tasks effective.
Do developers get more done or less done when estimating tasks vs not estimating tasks? The only claim estimates would increase productivity is the extent they are deadlines. But I think you will see that when developers take meeting their estimates seriously their estimates tend to get more conservative. And then there can be a desire to not over deliver on the estimates to lend those estimates legitimacy. Without estimates, you still have a performance metric to use, individual velocity. If one developer isn't getting stuff done as fast as others, you don't need estimates to know they aren't performing as you would like.
Are estimates reliable? This of course varies greatly by the project and the team but I my experience estimates are typically no more precise than 50% of their length. So a 6 month project probably shouldnt take longer than 9 months. Your mileage may vary, but is there a way to crudely estimate a project without asking for estimates? I would say probably. Why not just measure the teams velocity and project it?
I would argue estimates were a first guess at a way to run a development team which has survived mostly because of a refusal to consider the first guess might not have been the best guess.
"Software development is not carpentry. Almost everything a developer writes is unique, they have never built that particular thing before. We are not cabinet makers repeating a variation of something we've built hundreds of times before."
Some software development fits this description. Quite a lot of it does not.
The difference between the buyer of a house and the buyer of software is that the house purchaser is clearly and succinctly able to communicate his / her desires and expectations. Where as the purchaser of software is making some vague request for functionality with poorly defined boundaries and requirements. There is little meaningful comparison of the two scenarios that can be made.
Hear, hear. Anyone who has worked in a quasi-management or leadership position before will understand the need for an answer to the question: "We don't have unlimited finances. Tell me how much money I need to put towards this."
Having said that, most developers refuse to respond intelligently to these questions. The answer to almost all of these question should be something like "I think delivering this goal might take X resources, but here's the smaller goals we're going to deliver along the way to the big goal so that we can reposition when we go off-track."
Instead, developers say "Well we have to make sure that EVERYTHING can fit into this timebox, and our manager is going to treat this estimate as a deadline, so estimates suck!"
This question is supposed to be a conversation starter. Don't shoot the manager for doing their job. And even when the manager does a poor job and treats your estimates as deadlines, call it out! It's up to the developers to put up a bit of a fight and make more intelligent decisions too.
I've only met a handful of developers who are able to have a conversation like this. It's what the 'Agile' movement was attempting to fix. But it turns out conversations skills are tricky things to impart.
My take is the comoditisation of developers, pushing as much responsibility to them in search of efficiency. Arguably they are best placed to know how long the implementation of X might take, but often the lines are blurred, i.e. developers also become business analysts to spec out the work to determine what the business really needs. Personally I find it exhausting and is one of the aspects that makes me dream of doing something else. Maybe GPT-4 will put me out of my misery soon.
“…completely unmoored from the purpose of business, which is to make money by providing a product or service within a given schedule, scope, and cost.”
Or perhaps to solve an important problem in the world, within a given scope and cost!
It is true that scope management through batch theory delivers consistent results, yet the quality is almost always below what would continue a products profitability, almost always initially falls below user expectations, and almost always includes a tradeoff for engineer sanity in the form of maintainability.
What the PM is not saying, or the quiet part, is that from their perspective "it is your job, do your job." Frankly, that's a piss poor take as well. People are griping about the relationship between engineers and businesses because it's taking a toll. Telling people to shut up and get back to work, worse, alluding their gripes are narcissistic at best is a woefully attrocious take.
Businesses can be better, but part of being better will be taking bets on how to make things better that the engineers (workers) don't solely shoulder or take blame for. Hell, it'd be nice to see the business aspire for internal change where the engineers aren't asked to change at all.
The one reason I have seen it useful for developers to estimate tasks is that it helps spot misunderstandings early on. As in: wait, your estimate is 3 days and mine is seven, are we talking about the same thing?
> Get pressured to be more accurate and to bring the estimate down
Of course people want it quicker, but this is the problem, stick to the estimate and the problem goes away for the most part. Negotiate on scope if it is too long.
Communicate as early as possible if there is a mistake in the estimate.
At that point the problem largely goes away.
Edit:
When communicating about the mistaken estimation, include which assumptions or surprises came up.
It is a rare combination to have the technical skill and experience along with soft skills like humility that make it somewhat rare. But useful estimates are absolutely possible.
The question is one of range, with humility you can hit estimation ranges defined by how familiar the project is and how well defined the project is.
There are many developed techniques that we just don’t implement. By in large, most estimates I have seen were some weird gut. Not defined no milestones. “Yeah, about two weeks boss”
Often estimates come before requirements. Those will be wrong more often than not.
Many times I will estimate the estimate. The “bigger” the project the longer the estimate takes.
People are afraid to admit any knowledge gap.
People switch jobs every 2 years, I have been at the same place 10, and before that 8. You learn the cadence, you learn the pain points. You learn the traps. You learn the domain.
These can be accurate.
But this is not the OP’s issue as stated, he is lowering estimates in the face of pressure. This will nearly always fail.
Yeah, all models are wrong, but some models are useful. As far as I understand, the goal of estimates is not to pressure you, it's to promise something to someone. As long as you are good enough, business decisions can be made. And that includes figuring out when something is off track, which is one of the primary goals of management.
I think a lot of people commenting are not giving a charitable interpretation to the question. Now why any estimate is needed, but why should the dev be expected to have the right prognostication? As a dev, I have to admit it's a fantastic question. The reason estimates go awry almost always come down to things outside my control. Why am I expected to make predictions based on that? The people getting paid salaries to have a bird's eye view and veto power over those kinds of changes that could screw with my deliverable should be the ones knowledgeable and accountable for that estimate. And if I agree to it and fail to meet it, the judgment should go above both of our heads: was I delinquent, or was I blocked because someone else didn't do due diligence?
I am responsible for telling the person promising how long the work would take as scoped out, given the competing or potential problems which that person is also responsible of informing me exist. It's not that I provide no info - it's that I can't be authoritative about delivery times in an ecosystem I don't entirely control.
I think this argument is not about engineer vs manager. It’s about junior vs senior engineer. As you gain seniority and tech-lead others you take on this responsibility. I agree that junior engineers shouldn’t do estimation, but rather senior engineers should set the expectations.
The people getting paid salaries to have a bird's eye view and veto power over those kinds of changes that could screw with my deliverable should be the ones knowledgeable and accountable for that estimate
But they are accountable to their stakeholders (clients, upper management, etc). They take your estimate and many others from other people in the team and work those into a delivery plan.
A good manager will know how to manage risks and remove blockers in a way that gives a developer the best chance to work within the estimate. A bad manager will usually have no plan and put all the blame on the developers if things go awry.
Agile frameworks often remove the developer as much as possible from the time estimation effort, instead relying on past performance, and attempting instead to have the developer focus on breaking work up evenly.
A key component of the due diligence you're expecting involves asking developers how long things will take. Savitha is building a new feature with about 2 more weeks to go, but it can't be deployed until a guy Jim on some other team deploys suchandsuch dependency upgrade. Should I commit us to a mid-April release date, or will Jim need more time? I don't know, I'd better go ask him.
Of course. What I'm saying is, I see failures IME as Jim putting something into the code base that makes Savitha's work consequently incompatible, which was not surfaced to her as even a possibility when she was asked to give her initial estimate.
I’m in precisely this situation right now. I’m an external consultant being asked by a dev team lead how long it’ll take for his team to deliver a project. He’s the one approving holidays, he’s the one allocating priorities. He’s the one that knows the skill levels of his staff.
I’ve been in the exact same situation a year ago, where another manager got mad at me for a delay. It was caused by him approving his entire team to simultaneously go on 3-month holidays when the COVID travel ban ended. Apparently that was my fault for not estimating accurately.
Perhaps by coincidence, there were a couple of front page threads last week about the cost and duration of major government infrastructure projects. But actually it's a recurring theme. People speculate about things like government bureaucracy, NIMBYism, and project complexity, but ultimately nobody knows the answer. And some of the easy answers break down if applied to the management of software projects.
Get your bathroom remodeled in your home and have the worker tell you it'll be done whenever and you'll understand why developers (or any hands on worker) is lacking in their skill and practice if they have no clue how long it takes them to perform their work.
"Management" or a PMs job will then be to aggregate those estimates so they can estimate at a higher level than you, similar to a project management in a construction site. But each person needs some semblance of an idea of when their part will be done.
The same way if you were remodeling your bathroom and your garden and want to tell your kids when the construction around the house will be done, you'll need the gardener's estimate and the plumber's estimate, plus maybe some buffer at the end, plus some time to go with the wife to get a new shower curtain, etc.
Thanks for taking this avenue, the comparison to carpentry really annoyed me because all bathrooms or cabinets, or e-commerce/CRUD apps are the same right? (just because we're all tired of working on them doesn't mean our work is all copy and paste)
The most difficult part of any "hands on" job is time estimation - two people could give estimates of 3 days or 3 weeks on the same project and both be right.
Plumbers are doing applied physics, not rendering a button on a webpage. Try telling a physicist to provide an "estimate" for the cost of proving the big bang theory. Why should you expect an estimate for fixing your leaky bathroom faucet?
Half this site is crapping its pants over being replaced by AI being able to do their jobs with a prompt yet it thinks their work is sooo novel and unknowable.
Look man, even plumbing estimates are usually way off - but they know how long it takes to hook up two pipes that are 30' long for your new bathroom. They don't need to have meetings about it during the process. They don't need to talk to Home Depot about integrating with their broken API.
I work in an industrial physics and mathematics domain, and in that domain specifically, I must say it is very charitable to describe a majority of software engineers as doing "applied mathematics", if we are to interpret that as meaning "software engineers are participating in the practice of doing research mathematics." Except perhaps in some extremely reductionist view of what it means to "write a program" (e.g., programs are proofs, Curry-Howard, or some other idea), I don't think "applied mathematics" truly characterizes the work of, say, someone writing out a GitHub Action or adding a new RESTful endpoint to list a company's product catalog.
There's a pattern amongst software engineers in these and other discussions. When it comes to discussing management and resource estimation, some people describe software as the most arcane, idiosyncratic craft to exist. In other situations (e.g., where actual research mathematics or computer science is proposed to be used to solve a problem), software engineering has to be "simple", "aligned with best practices", and almost anti-intellectual.
The pattern of discussion (here and elsewhere), to me, indicates a survival instinct of sorts, if that makes sense. To this end, there's a sort of hypocrisy if it means being a software engineer is as unrestrained as possible: software is purported to be like applied mathematics in situations where it's advantageous to describe it as such (e.g., "we can't estimate because proving conjectures is intrinsically unpredictable") , then when the proposition is brought forth to engage with software as applied math (e.g., "team, we can bound the 99%ile perf by proving this recurrence on resource usage, and that'll give us evidence to modify our search algorithm with an adaptive radix tree."), software engineers push it away for something satisfactory to a CS101 student (e.g., "we can just implement auto-completion of our product catalog by searching a sorted array of item names, why complicate it?").
I understand the troubles of estimating resources for tasks and projects, and indeed the requests can sometimes be unrealistic and overbearing—I don't mean to suggest otherwise—but even the most technically sophisticated pieces of software are flush with "ordinary" kinds of programming tasks.
In my experience just working by priority works better than endlessly estimating tiny chunks of a project, that doesn't actually give any real predictability. Workplaces that go heavy on estimation and "agile" methodology produce bad software and not very reliably.
We're performing applied mathematics, but we're performing that task as engineers. The key is in 'applied'. Going from theory to practice takes planning and is not as unbounded as performing science in my opinion.
I think it's about balance, and a lot of software engineering projects would do better by steering more toward the academic research group model than the current fad of two week "sprint", plenty of non-technical pseudo-managers and endless micro-estimations.
What ever model works for a group of people producing software I'm fine with.
But as someone else said in this thread: If the question is more about who provides the estimate and less about why is there an estimate (also plenty of opinions about that in this thread) then I would say that time estimation is a integral part of an engineers job.
What makes someone an engineer is being able to provide a weighted solution towards the needs of a client or consumer. Not the best solution in the world, but the best solution for that client. If the requirements are that it should be delivered fast, you understand that the quality goes down. If the client wants high quality, it takes more time.
You're not seriously comparing research mathematics with software development, are you? You think you're expanding the breadth of human knowledge with your CSS tweaks of the shopping cart implementation? Give me a break.
Applied mathematics is a branch of mathematics. It's research. Someone studying solutions of the Navier-Stokes equations is doing applied mathematics. An accountant uses mathematics, but their work isn't "applied mathematics" any more than a plumber is an "applied physicist", or a veterinarian is an "applied biologist".
They best contractors I've worked with do just that, that is they don't give time estimates. They will say when they will start and how they will proceed but they never give estimates. They don't for the same reasons, they don't know. In their case it mostly revolves around sub-contractors and supply issues but the outcome is basically the same. There is to much noise to make an remotely accurate estimate so it is better to not do it.
I have a hard time imagining how such contractors would ever get a contract in the real world. I certainly wouldn't think of a contractor as highly skilled if they can't give an estimate.
If you tell your bathroom remodelers that you want green tiles after they have already done half of it with red and by the way you want the bathtub in a different place and you also aren’t sure yet if if you need a toilet bowl then their estimates will get out of hand pretty quickly. And your bathroom should be able to handle a 200 person family just in case although right now there are only two people in the house. And to save money you hire people from the street who don’t speak your language and have no experience but we all know one worker is one worker. They are all the same so when you hire cheap workers your cost is less without loss of productivity.
Why would you change plans and expect the construction crew's prior estimate to remain unchanged? Estimates change if the project changes. A commercial developer's job includes recognizing that and communicating changes upstream. Usually that leads to a negotiation, collectively determining how much to (try to) spend and which boondoggles are trutly important.
If a hired software developer told me up front, "I can't estimate because you will change things and then the old estimate will be wrong," all I can conclude is that this relationship is already dysfunctional.
> I can conclude is that this relationship is already dysfunctional.
Here's the thing, though: you wouldn't be wrong.
It may be different for contract development. For in-house development though, everywhere I've worked we've all always known that the relationship is dysfunctional. We also know that the dysfunction is everyone else's fault, not our own. So we just put up with it, because those assholes will never consent to fixing it, anyway, and just try to shift the blame to us instead.
Yeah I'm used to in-house, no material contract experience.
> the dysfunction is everyone else's fault, not our own
I specifically said the relationship. It takes two parties for effective communication to occur. An engineer who have given up trying because they assume experienced management will not listen, is just as bad as a manager that steamrolls their experienced engineers.
In most companies it works this way "Give me an estimate based on what I am telling you now. I haven't really thought it through and there are a lot of unknowns I view this estimate as a commitment that should be kept although I will probably make a lot of changes during development"
I think this is close. IMO, the problem is that we're talking about a group of concepts using vague language. Is it an estimate, a target, or a deadline? "Estimate" gets used in place for targets and deadlines all of the time and it sets up situations where expectations aren't properly managed.
This is the trick, everyone in the room knows it makes no sense. You change the requirements and then hold someone accountable for an estimate that is based off of different, often simpler requirements.
What you do is extract an extra amount of economic output out of your worker, who wouldn't say no to that? Then when it hits the wall, it's the dev's fault anyway. If it works out, you get a pat on the back!
If the dev says NO, the dev will still have to do the work and may even be coerced into doing so. Many junior developers think it's their fault, but they're being manipulated. That's why nobody likes seniors, too.
Of course. If management expects precise estimates then they need to provide precise requirements and be okay with adjusted timelines if they change the requirements. But that's fine. They can do that.
While some, or a lot of that, is due to incompetence, it is also due to the nature of the beast: Software Development is an iterative process. You have to prototype and figure out what works and what not.
So regardless on how tight your processes are, whether you use agile or waterfall, it will never be possible to give perfect time estimates. You can only improve your risk-management.
Looking at the comments only makes more obvious the snowflakeness and detachment from reality of the "software artists" bunch.
If they hire a guy to (re)model their bathroom, they expect it for free done yesterday. What I mean is that there's usually a complete turnover effect on the demands they have on people working on THEIR money and the lax and forgiving way they expect to be treated on OTHERS people money.
When a contractor is remodeling your bathroom or your garden, the amount of time spent planning the design is insignificant compared to the cost of labor and supplies.
But in software, the planning is the entire project. The closer the plan is to covering every single possible case, the closer the software is to being done. In construction, the work doesn't even start until the plan is done.
Every one of these comparisons that involve a physical product whether it's contracting, bridge building, or manufacturing all completely miss the mark.
If you compare with other industries where the plan is the product, you'll start to see more similarities with software development. And the less the planning is cookie-cutter the more trouble those industries have with estimates as well.
The work developers do varies, so this question in general makes little sense.
Sometimes, developers literally do R&D without knowing, whether they will be able to solve the task at hand at all. Other times, they use proven solutions for typical problems.
I saw the most issues arises, when management starts to confuse between the two.
From the business perspective, you definitely have to balance the ratio of the two.
Producing estimates is a necessary evil for multiple business reasons. A fundamental skill of any experience Software Engineer is to be able to make decisions under lots of uncertainty, and that includes allocating resources to a task for which a lot of information is not yet available. Yes it sucks but it needs to be done.
If you want to see an engineer squirm, ask them to estimate how long something will take. If you want to see a manager / product person squirm, ask them what the net present value of the deliverable is.
We should just accept that significant uncertainty exists in both situations. If you can’t handle risk, liquidate company and invest in t-bills.
I agree that time estimates are a necessary evil, but I disagree that story points are useful for anything really.
This comment from the article was interesting and it matched my experience:
>
The law of large numbers only applies if a) the population is sufficiently large, otherwise outliers will dominate b) the estimates are at least mostly independent, otherwise systematic bias will skew the result and c) the estimate is in fact the expected value (mean average) of the probability distribution of time taken. If the estimate is a mode of the probability distribution, then you can not meaningfully add the estimate directly and be able to meaninfully say anything about the resulting distribution.
every business exist to make money or something of value to its shareholder.
You can hire contractor to modernize you kitchen or paint wall.. you need some understanding of exactly what is scope of work, what is $ cost, what is the quality, and any other constraint.
the contractor cannot just say i have no idea how long it will take. there has to be some start and end date.
why some programmer think they do not have to provide any estimate at all is perplexing. is your paycheck an infinite money stream? does your employer not have commitment to deliver something by specific date? or do you want someone else to go create estimate of how long YOU need to complete some work?
this is why tools like scrum come in. estimates not a perfect science..so break down scope of work into small and small chunks until we can say “ok…this piece take 3 days”.
As someone who has both built and sold software, it's simply to be able to sell software before it's complete. Without estimates, you can't sell something that doesn't exist yet.
This doesn't change the fact that requiring estimates is a bad idea if you want great software. The best software is built well, then sold.[0]
Great software later is more valuable long-term than bad software now.
--
[0] The best case is actually to sell software without a timeline. But most organizations are not able to operate this way.
Still, some level of estimates can always be given. I have never embarked upon a programming task, even a very novel one, where I couldn't say whether it will take two hours or two years. Now, whether it would take two weeks or two months has sometimes happened.
Most of us are carpenters since carpenters don't build literally the exact same thing over and over either. They build similar stuff like decks and houses over and over. But they're all different to some extent. Programming is the same. The exact functionality is the different but they're all accomplished using more or less the same techniques.
And remember - the people asking for estimates want the estimates right now. I could give you an amazingly accurate estimate of how long a software project will take - it just might take me longer to produce the estimate than it will to produce the software.
I expect a plumber to give me an estimate of time and cost for his "tasks" before starting work. Even if he doesn't tell me the exact cost upfront he'll still be able to tell me roughly if he'll be done in 2-3 hours or less/more.
If I have a big project and I'm dealing with a plumbing company I might not be talking directly to the plumbers but to their manager. The manager will usually ask for time/complexity estimates from his plumbers before sending me a quote. The manager will usually factor into the quote any risks plus the company's profit margin.
There's a lot of talk about agile, scrum, no-estimates and so on, but a big part of the software industry still works with time estimates and budgets communicated up front.
Perhaps a solution is to distinguish between an estimate and a detailed plan.
One is a quick guess with high risk, the other takes time and research to prepare and the risk of running into a delay is lower.
And often the best solution is to have none; perhaps part of the problem is that people are often too afraid or have not enough trust to work without plans or estimates when needed.
I have found engineers' estimates to be accurate and straightforward when the team owned all the resources needed to complete their tasks. Whenever it depends on some unreliable factor (often some unavailable expert) estimates are useless. Estimating tasks without estimating each team members capacity is basically lying to yourself.
387 comments
[ 2.8 ms ] story [ 312 ms ] threadthis is such hilariously distilled peak stackoverflow that it belongs in the hall of fame
"Most of your question is really a rant about how things work at your workplace. Discussions about toxic workplace practices per se are out of scope for PMSE."
Not a friendly, charitable answer, and it is right at the top.
As to the question itself, crowd-source the estimation, is the best approach, in my experience. Get the team together, each member write down an estimate, and reveal all the estimates at once. If there are outliers, have a discussion, otherwise round up and move on to the next estimation. If you're in a scrum shop, you can abstract the estimation by one level and call them "stories" and you're estimating "complication" not "time", which is even better.
A few obvious reasons:
- The buyer has to make commitments externally, for instance to customers, partners, finance, marketing, his boss;
- The buyer has dependencies on those external resources and needs to plan for them;
- The buyer has a limited pool of resources and needs to know when they are free for the next task or project;
- The buyer needs to get an idea of costs to complete the feature and secure the budget;
- The buyer needs to make priority calls. If feature X is significantly more effort than feature Y then we can prioritise accordingly;
- The buyer is paying and simply wants to know when he will get his shit.
I wonder if the person asking the question would be happy to let someone do a job in his home with an uncapped budget and timeline?
The quality of the estimate would be unaffected.
In some cases I’ve got agreements up front that if the dependencies aren’t done or being worked on by particular dates my team gets access to the codebase or systems and can take over. That type of consequence/potential embarrassment can set up potential adversarial positions but it has helped a few times.
Also, a job on his home is exactly what it should not be compared with, since that's the sort of repeatable job that can be estimated.
No, just all software that provides any sort of tangible benefit to anybody. I’m sure you could estimate fairly accurately how long it would take to implement quicksort - but nobody’s going to ask you that.
I know many developers enjoy the art of software development and want to continually improve it, but that isn’t always the task you are being paid to do.
There are many ways to remodel a bathroom. Some more optimal than others.
Now imagine your plumber tells you he discovered a better layout for the plumbing just after finishing his work, then he enthusiastically asks if he could tear everything out and start over (and pay him for it).
Vendors and even open-source libraries are often liars or just missing what seems like it ought to be obvious functionality. Sometimes tools are just broken. I once was assigned a Java AWS Lambda project, which, great, supported, right, says so right on the website? Except half the CLI tool functionality just didn't work, at the time, for Java projects, and I mean the official tools. It was alpha-tier supported at best. Usable, sorta, kinda, but you're gonna lose some time un-fucking the situation, and the whole thing would be slower to develop than it needed to be. You'd not know that unless you went and did a focused search on their issue tracker(s), which isn't really reasonable to do for all the stuff you might use in a typical project (some of which you might not even know you need until you've started work). "Great, so now you know that for estimating next time, if you get assigned a task exactly like that one" well, no, not really, because that was like four years ago and the situation's probably totally different now (maybe better, maybe worse, who knows?)
That's why I think giving 4-8 weeks of development time with some basic t-shirt sizing or pointing for tasks and just seeing how it's going is better, before estimating. I think if you can't afford that, before getting the estimate, you can't afford a half-decent estimate at all and just need to deal with the fact you're going to get a terrible estimate with an absolutely huge range ("six to eighteen months"). Which, fine to ask for that up front then refine it after some dev time! It just needs to be clear and expected that those early estimates are going to have some wild variance—but they can definitely tell you "is this more a 1-year project, or more a 5-year project?" which does have some real value.
It's not strictly correct that most software is open-ended research. Most economically valuable software is open-ended research, because otherwise a solution already exists and you don't need to pay a software developer to write it.
Software is a somewhat unique industry in having zero cost of replication. The only other industry like this is creative & artistic pursuits (writing, artwork, music, film, etc.). That industry is also notoriously difficult to estimate and tends to be an area where the the greatest works of art are done when they're done. (How long have people been waiting for George R. R. Martin to finish A Song of Ice and Fire?)
To some degree and depends on the job. Start opening up walls in an old house and tell me how cookie cutter it is. When I used to run shipyard jobs on offshore drilling rigs, I can tell you that every single job ran into all sorts of problems that were individually not predictable.
It's also just nonsense to think that most software is this uniquely open-ended snowflake. Most engineering jobs require problem-solving that can be affected by all sorts of things in the physical world--plus you sometimes just have to build and try out designs and maybe they work, maybe they need to be refined a bit, and sometimes you have to toss them out and start again from scratch.
I see people on here post construction projects as the poster child for things that can be estimated with traditional project management techniques. This is wrong because most construction projects come in over cost and over schedule.
This can be because materials costs change, labor costs change, the soil at your site may not be what you expected, weather impacts, and stakeholders may demand something different after the project is estimated.
Although these are unknown quantities, these factors existence are known. Whereas in software, you’re more likely to encounter unknown unknowns. These are factors that nobody thinks are even possibilities until you run into them because nobody has done anything similar before. An example from my experience is when I was working on implementing a marching cubes algorithm on a GPU, we did research to make sure that you could do it. But when it came time to implement, we found that the cube data was expressed as a set of faces, and not a set of vertexes and no efficient algorithm existed for that expression of the data that could be mapped to a single instruction multiple data paradigm. We could not preprocess the data to convert it to vertexes because it would not offer any speed up if we did. To implement the acceleration we wanted, you’d have to change how the data was stored in the application. Which is a very different project with very different timescales. We ended up abandoning the effort and moved to other work. This happened because we made a discovery that no one could have predicted, in the implementation, that had an unknown impact.
In physical projects, you might find there are a lot of unknown things that can happen, that you may not predict. You can certainly reduce this by bringing in experts to a project, people who have seen much more than you. Certainly there is still potential for black swan events. But these are rare. Where as in software, the experts you may bring on may not necessarily know more than you, you are frequently your own expert. Black swan events in software projects are common.
But they were surely predictable in aggregate, no? There's only so much volume in the house and a volume will take a bounded amount of time to deal with, even if the bound is very large. An oceanic vessel has a maximum amount of mass that can be added or removed, and every unit mass will take a bounded amount of time.
This isn't the case for development. There is a bound to how much code can possibly go in a project, but there is no bound for how much development work goes into any unit of code.
Only in the sense of jobs seem to take at least twice as long as they "ought" to. The volume of the vessel has very little to do with it. In a refit, you know some things going in--which may indeed be relatively straightforward (so long as the right parts arrive on time)--but for bigger jobs lots of things crop up including resource conflicts and parts that need to be ordered.
If software projects added more slack, they'd be more likely to be completed on time too. There's nothing magical about software. And, in fact, software doesn't need to deal with a lot of the quirks of physical systems.
It does seem like the responders have a little bit too much sunk cost in the whole famously fraught agile thing to understand critique of it unbiased. It's fine in theory but does have flaws.
That doesn't give you an accurate estimate, but it does usually give you a less inaccurate one.
And yet after three years at my first job, I could easily do a quick breakdown on a project, assign a three level complexity to each step (easy, medium, hard), put an amount of days corresponding to the selected complexity (to be reviewed after each project), add 25% for unforeseen events and contingency and my estimates were accurate enough most of the time.
It is not difficult to properly do rough estimates. Most developers just refuse to invest the time necessary to do it properly and sadly for some as the answers here already illustrate I do believe it’s because they think of themselves as special snowflake rather than team player.
Op is not saying that having estimates is useless, but rather it should be a management task rather than a developers burden.
To me, that sounds even worse, but I do think devs are expected to do wider and wider tasks across the business. Their post reeks of resentment against management and PMs and I don't blame them. Especially in my experience, I have worked with PMs that I have no idea what they do as devs need to step in to do all project management related tasks themselves
Asking the builders to estimate the time they need is a good sign of high trust in that team. It’s healthy. We should want more of it. The business’ side of the commitment is to understand that it is an estimate and should be treated accordingly.
I was tasked with creating an entire email campaign system (like mailchimp). But I had only 3 weeks to do it. I made the deadline no problem, but the system had numerous obvious issues.
My boss didn't care, because the task was not about delivering a perfect email system. The task was getting something useful in the hands of customers rapidly.
If a feature starts getting heavy use, then more investment is justified.
That's where it all breaks down. What does "treated accordingly" mean? In reality it means the estimates can't be used for anything because they are always unreliable. There is no such thing as a reliable software estimate, ever, unless what you're doing is so repetitive that you're about to be replaced by AI.
But that's not why people ask. Instead, half the time you give an estimate it gets immediately turned into a deadline, even if you were promised it wouldn't.
The top comment on this thread is naive. The reason devs hate giving estimates or point blank refuse is because it is meaningless, that's not how the software business works. Analogies to plumbers and builders just reinforce the naivety. Guess what, blue collar work is only predictable for as long as it's highly repetitive which being physical in nature, it often is. The moment what you're asking for is "build me an underground railway using the latest technology" it turns out construction estimates are worthless too (see: Crossrail).
One reason tech firms crush their competition so reliably is they don't have an estimates-deadline culture, because they're run from the top by programmers who understand intuitively why they're pointless. Instead developers are incentivized by equity, bonuses etc to do the job as fast as possible.
It’s less that I now have a specific endpoint, but rather I have a target (or multiple targets) at which I can decide if we’re on track and, if not, how I may want to change previously made prioritization choices.
If you tell me a day, then 5 days later if it’s still not done I may not rip you for being wrong but I may decide it’s time to pull the plug on that initiative since it no longer meets the criteria under which we agreed to do it.
I’m all ears for another proposal, but “just keep building stuff with no regard for how long any of it takes” is not much of a strategy and doesn’t reflect the way that decisions have to get made in the real world with limited resources.
Also and I cannot stress this enough, if you can make any estimates at all you're playing on easy mode. If the stuff you're programming isn't entirely rote boilerplate then you cannot estimate anything, ever, because your error rate will be measured in orders of magnitude not multiples.
Classic scenario: "how long will task X take?" ... "hmm well there's an open source library that claims to do X here, looks pretty easy to use, so maybe two days?". Some time later: "why is X taking 2 months and not 2 days?" ... "it turned out the library has some non obvious architectural limits we can't fix and which kill performance for our use case, so we have to write our own from scratch".
That sort of thing - you thought you could rely on another component and then you can't - can happen at any time, to anyone, and renders all estimates based on the assumed viability of that component meaningless. But you're always building on third party work. If you have absolute confidence in all your dependencies it, again, means you are doing repetitive work that isn't much different from your last task. Such works exists, I guess a lot of frontend dev falls into that category, but it's also highly susceptible to being automated away by better frameworks or AI.
>> it doesn’t reflect the way that decisions have to get made in the real world with limited resources.
My experience was the opposite actually, the big successful companies do exactly that. They may cancel projects if they seem to be no hopers, but they don't try and estimate up front how much a project will take because they don't think in terms of projects at all. If something is important, it gets staffed until it either stops being important or the team is beaten, at which point it gets unstaffed. If progress isn't fast enough they add more headcount. Everything is thought of in terms of relative resource flows, not timed Gantt charts.
This seems like throwing out the baby with the bathwater. An unknown thing can still have bounds on it: I don't know exactly how far it is to the nearest supermarket, but it's further than 50m away and closer than 1km away. That's useful information: I know I'll be able to walk that distance, I know I can probably carry a reasonably heavy load back, and I can roughly estimate how long I'll be gone.
Fwiw, I agree with you that estimates are too often used badly by management to set up deadlines rather than genuinely make planning decisions, but I also think developers often fail to communicate their confidence levels when giving estimates. There is a material difference between "I know exactly what the problem is and what needs to be done, this will take two days" and "this looks like a pretty minor issue that will probably take two days, but it's touching a system I'm less familiar with, so it could end up going on a lot longer if strange things are happening there".
The best combo project/product manager I had didn’t have either official title, but we were on an internal team and somehow she always managed to make sure people around the company knew what we were doing and delivering, pester people so that no tasks were dropped, etc.
Interestingly, she did most of this by putting tickets for projects into Google docs, and otherwise didn’t buy into any of the standard project management dogma. She had an ear/nose for who might want to use our stuff and was relentless at getting us on internal presentations of any teams/groups that would be interested.
Count how many people are also complaining about their managers giving timelines to clients. Damned if you do, damned if you don't.
In those cases estimates form part of a whole basket of mechanisms and rules designed to ‘shield’ developers from users and vice versa, which imo is among the biggest mistakes one can make in such software companies.
They key is negotiations, and both parties being able to balance a moving target of requirements and scope. Large software development is not a determinable thing as much as we would like. Perhaps ChatGPT will solve this?
>> I wonder if the person asking the question would be happy to let someone do a job in his home with an uncapped budget and timeline?
Even this is not always accurately determinable! If you've had work done in your home, or heard anecdotes from others, you would realize this.
Not surprised this discussion devolved into this bizarro world point so quickly. The developer says “there are too many unknowns to provide any sort of estimate”. The manager retorts “but we need estimates!” As if they expect the developer to say “oh, you need them? You didn’t say that. In that case, the unknowns are now known and it will take four days, three if you remove this inconsequential detail that nobody cares about”.
Unfortunately this bit of illogic is so common that I know it’s a waste of time to argue, so I’ve learned how to figure out what they want the estimates to be, give them that, miss them (just like everybody else does), and then get to work on actually providing the feature.
Seriously though. It's always a time tradeoff. How much time do you want an expensive developer investigating how long this problem will take. Will they learn something they can use during development.
I always follow up my above quote with the unknowns and how hard I think they really are; what are the possible risks; _and_ I include the experience of the developers working on the problem; our ability to bring in other devs who understand this better etc etc. I like to make sure whoever is quoting this problem out understand everything as well as we can in the given amount of time.
If that number is too high, then that’s a sign to management that the work needs to be changed. The only way to determine that is to estimate it then make that call.
And only works if management wants an estimate. More commonly, management wants a lie they can tell the client.
That’s absolutely preposterous. As a senior engineer, I have seen enough that I have at least a high-level idea of what most of the dev tasks and stories I’m estimating are. I don’t always have the specific details that may change the estimate, but I also have a whole team of other people who have seen even more things and can inform that opinion.
The point of refinement sessions is to share this information with everyone else so an estimate can be created and if needed, tasks can be better defined, split up or merged.
> And only works if management wants an estimate. More commonly, management wants a lie they can tell the client.
Wat? Estimates is used for plenty of reasons that have nothing to do with external clients.
How do you think your manager, their manager, etc knows what is going on and approximately how long it’s going to take? They’re going to be far more disconnected from the work than the people doing it day to day, so they would generally have no clue how long something is estimated to take without someone telling them.
You are misunderstanding your job. You are a subject expert and paid accordingly. Answer with what the unknowns are, an optimistic and pessimistic take on them, what you could do and how long it would take to lift them.
If you don’t, your boss is going to ask you for exactly that in a roundabout way and micromanage you because you are forcing them to do your job.
It still baffle me than approximately half of developers don’t understand things which are obvious to any seasoned blue collar worker.
In software, on the other hand, we often have most training being taught by professional academics who do not understand and are not qualified to teach the parts of the job that don't have a straightforward academic analogue. Combine that with a business environment where things like budgets and cost control are often only theoretical, and all but the most gross liability for defects due to poor workmanship has been waved away by EULAs, let it marinate for several decades, and... yeah. Not only do people not know these things, but I'm not entirely clear on how or why they should be expected to learn them, either.
If he really needs an estimate he'll tell me to start digging into it right away, and I'll let him know after a bit of time how it's going. If there are still large uncertainties and my boss still wants this completed by some specific date, we might try to explore alternative solutions.
In the above case, I didn't get that time I would need, they wanted the estimate to include that time I would need to dig into it. But that's above my estimation powers. I didn't even know if what they asked was possible at all.
They went and asked someone else who did guess some number of weeks, but the work never got priority so we still don't know.
Yeah that ain't gonna fly with me.
Did you say that
1) Without knowing further best case is it takes N time, worst case M time, and if there are certain obstacles it could be completely unfeasible. 2) To get more accurate time and to understand whether it's feasible I would need O time to figure that out first, and I can give you the more accurate estimate then.
The problem is when we assume everything is a known. I try not to do that and I try to set that expectation with my manager. If everything had an easy answer we wouldn't need highly paid problem solvers.
Well, that's the problem right there. An estimate is not a promise. It should not be treated as such.
It's not an apples to oranges comparison. You give me an architect's blueprint and guys who build to the legal building code, and I could probably get a house built. Expensive and painfully, but it can be done.
I don't even know what the coding analog to an architect's blueprint is. They are fundamentally different jobs.
Yes, I work as a delivery lead nowadays. Producing and respecting estimates for software projects is literally my job.
I routinely fire developers who pretend they can’t do estimate and strangely the ones I keep are actually not only able to do it but good at it.
I've met your sort before. You're a net waste of resources in an org: instead of developers getting things done, they're dealing with you asking them to do your job for you, i.e. produce estimates for the things they'll never actually get around to doing (because of producing estimates).
Your username would be accurate if s/My/Everyone's/
If it helps you sleep at night, sure, assume I’m a net waste of ressource. Good luck fighting with the other seven teams for some time with the shared senior UX expert.
If you rarely eat your time buffers, great. If you do that often, maybe you just coinflip-fix deadlines this way and maneuver between them under the guise of successful estimation.
You always get what you ask for. Like sure I can estimate. If you ask for a web server you'll just have that. It won't be secure. It won't be fast. It may work. It will certainly pass the tests you think of.
Is that actually good? No. The real estimates get rejected. Everyone cuts corners. When something does happen e.g. a hack occurs the engineers not these managers get fired :(
My optimistic/pessimistic takes at my last job would look something like "roughly a week to roughly 2 years"
Depends on how deep the yak shaving goes, and you wouldn't know until you got halfway through fixing the problem.
Often I could take an educated guess as to which side of the scale it came down on, but I've definitely started pulling on a string inside of the codebase and it just kept going and going and going. All of those baked-in assumptions that other developers unconsciously made over the course of a decade can pile up into a mess that you just can't implement the feature/bugfix without doing a pile of potentially breaking changes that need to be shipped in major versions.
Generally if I worked the problem for a week then I could give you a much better estimate of when it would be done, and often that estimate was "it's done now" after a week. But if I hadn't looked at that particular subsystem for a few years, I couldn't trust my memory of it up front and had to do exploratory surgery first.
(Really exploratory surgery is a much better metaphor for what you're going to be doing, and surgeons can routinely find surprises -- "oh look that barely perceptible shadow on the x-ray turns out to be a stage 4 tumor...")
Yes, the codebase was horrible, yes it needed to be cleaned up, no we didn't have the manpower to do that and it would have taken me 10 years to rewrite it all.
Quitting of course was an excellent solution.
You're lucky if your management chain allows you to work on problems for as long as a week without giving them an estimate first.
Add a small feature to an existing system. This estimate can be anywhere from several months to a day. Most of the error bars are org and system specific rather than anything to do with the feature. The exact same feature built by the same person in different orgs and different systems can be opposite ends of the range.
Seldom do people have the contextual understanding to provide an accurate narrow estimate, particularly since both org and system are in flux. They change constantly. Unseen landmines potentially await in both.
> If you don’t, your boss is going to ask you for exactly that in a roundabout way and micromanage you because you are forcing them to do your job.
And such orgs take forever to get anything done.
> It still baffle me than approximately half of developers don’t understand things which are obvious to any seasoned blue collar worker.
Software is probably the hardest kind of engineering you can do. All the system constraints are virtual. Just consider comparing a dirt ground for consistency and behavior vs libc... 304 stainless vs react.
You can go deep in both, but a shallow understanding of one takes you much much further than a shallow understanding of the other.
That’s extremely untrue in my experience. Sure, hard to interface with systems are a thing and organisational roadblocks happen but an onboarded senior team member can tell you what’s going to take days and what’s going to take months or they have no business being senior in the first place.
I feel like my answers in this discussion have been particularly harsh but I would really like people calling themselves software engineers to start acting like some. I think my view on this subject has been strongly shaped by spending most of my career doing software development for industrial companies: people who claim software is particularly complicated just don’t understand and respect how complex the rest of an industrial project is.
You're making an unwarranted assumption here. There's many pieces of software that have no development team at all. There's no one to ask; the last person to touch this software left the company three years ago. All they can tell you about it is how they use it now.
It's likely there's at least someone that can explain their business process to you, but identifying them and receiving enough of their time to piece the picture together is another story. And maybe, if you're lucky, that person can tell you what the business process was at the time the software was written, and what has changed since then.
But this process can take weeks, and most of the time you're expected to provide an estimate before even talking to anyone from operations.
Sure, for internal software with an active maintenance/development team, informed estimates can be given with a relatively high level of confidence. But as an outside consultant working with non-software companies, all estimates are completely made-up based on your reading of the buyer's budget: there are no other data points to base your estimate on.
And yes they are still solvent.
As others mention there are lots of unknowns in an organization.
I've been in places where the other team involved is across the globe in a different timezone and isn't even actively willing to respond and yet we continue...
There are also cases where lots of parts are outsourced and you're lucky to get a reasonable response i.e. changing part x requires vendor a, b, c & d, which have an SLA 1-8 weeks and so you're already starting 4-32 weeks without even working anything else out. If you're really lucky it could still all be done in 1 day.
Not sure if you're harsh as you say or just lucky. Not everyone and every team is well organized.
Yeah, most of the time my boss just needs to know if this job is a few hours, a few days or a few weeks. I never answer with exact numbers, but a range. If there are some major unknowns I let him know. If I made some important assumptions then I let him know.
Sometimes I'm wrong. Most of the time my estimates are sufficiently accurate.
Sometimes the customer really needs this done before some deadline. Then it often turns into "how can we make this happen before then". After a bit of back and forth with some added assumptions and cutting functionality, we usually end up with a solution that can be realized in time.
As a manager now, I get scoffed at if I even hint at a deadline. Excuse me what? We have a limited budget, a limited runway, and we need to prioritize which product bets we are going to make. This seems entirely logical, but a lot of devs I work with now find it impossible. They just enjoy tinkering away with whatever thing interests them, and delivering value only means what they personally find valuable.
Personally, I find it a bit pathetic and childish. I say this as a long time software engineer who knows perfectly well how this business works.
That said, however, The problem I run into during practice is dependencies— people want estimates to integrate into systems you have never heard of or used before. Not only that, there is no in place architecture or design before estimation. Imagine a construction company estimating a house without knowing how big it is or without having blueprints.
Often times software development is designing and architecting the system — not just implementing it. And that is the expensive part.
Say that the estimation can depend on the complexity of the system. I would think worst case is when the system has these A, B problems, then it would take N time. Otherwise best case it could take M time. If we want more accurate idea, we can spend O time on it to figure out what it could be.
How does this business work?
Why is software special? This happens the same in construction and other industries. The estimates usually make 0 sense and there are often huge delays. Same as store openings, film production, etc.
There's a problem with tinkering and devs going off in the wrong direction but on the flip side when people say that something is done in an hour or a day often never happens. Managers and agile practices that want tasks broken done to the smallest atom just messes things up. To make things work product often never has accurate requirements to that detail either so that complicates things even more. The requirements change on the fly.
In a healthy culture it should be possible to have a back and forth about how accurate an estimate is likely to be, but when engineers insist that all estimation is useless you can imagine why other stakeholders may lost trust.
As a developer I never get “that’s a useless estimate we can’t prioritize based on that/tell the customer that”. It’s a perfectly good estimate. It tells everyone there is a lot of uncertainty. The low lower bound indicates there is a chance it might turn out to be cheap (so maybe worth taking a gamble and trying it and giving up). The big variance indicates it could/should be given some time to prep/investigate in order to narrow the estimate.
When there are unknowns, make a wide estimate. Convey the uncertainty. If the unknowns are ridiculous, make a ridiculous estimate. Even “5-200 days” is useful because something else with less uncertainty can be chosen instead.
Maybe the solution here is "stick to your guns", but that can have negative career implications.
You can’t pull a different estimate out of thin air. And zero people will expect it.
The developer needs to say “I can say 10-100 days right now in this meeting, but given just 1 day of investigating, I’ll narrow the estimate range by an order of magnitude”.
I'm running through a similar situation now, where I'm somewhat adjacent to a client and a dev team. The client has a list of tasks he wants done, and he'd like banded estimates for the effort the dev team thinks it will take to complete each.
The developers are obliging, but one of them complained to me privately "why doesn't he just give us 2 months of budget and we'll get through as much as we can?" - which is an interesting statement. From the developer's perspective, this is ideal. They are very good and work hard and have integrity, so I'm sure something like that would work, from their perspective. They can just 'go' and get stuff done.
From the client's perspective though, it just raises a bunch of uncertainty. He wants to know what he could get out of his 2 months of development budget. If one task took all 2 months, and the others (15 in this case) didn't get done, is that OK? What if some important task wasn't that much effort? You'd potentially want to priortise that over a bigger item.
Just saying 'give us two months of budget' doesn't take the other party into account, and what they're trying to balance or plan for. I've sat on both sides of the table, and honestly just asking for a budget with no actual commitment to completion feels selfish in my opinion. Discounting their opinion because they're 'business' people and not developers, or treating them like an enemy to fight against won't win you any friends or long term relationships.
The irony is that at least some of the time, a "safe" estimate will actually make the project take longer. Work naturally expands to fill a deadline, and not having a deadline but instead frequent progress checkins with an expectation of foward progress can result in more getting done, because it changes the conversation from "how at risk is the deadline" to "how much happened in the last week and what are the next steps", forcing progress to happen continuously rather than one big bang at the end.
I also often hesitate to give estimates / establish deadlines because it tends to create an unhealthy confrontational atmoshepere, where developers are incentivized to not incorporate feedback or make any changes even if they are the right thing to do, because it would put the deadline at risk (but if they incorprate allowances for these things during estimation it's considered sandbagging).
That’s probably the most important thing I’ve taken away from software project management techniques - everything they do is hurting them, not me. If somebody insists on shooting themselves in the foot, and they’ll shoot you if you try to stop them, you just sit back and let them shoot themselves in the foot and shrug your shoulders when it happens.
At the end of the day, these estimates are helpful but maybe not in ways that are intuitive from a developer’s perspective. They’re useful in weighing priorities, judging the uncertainty, and identifying where to fill the gaps in understanding, time-boxing research and scope (e.g. do we really need to spend 6 weeks reading release notes?), and help communicate to outside stakeholders. We see the successes of this all the time with big game releases, product launches, movie releases, etc. It’s a high level measurement of progression and devs need to be a part of that planning process. It gets easier for senior engineers to do this efficiently with experience. It’s not necessarily an exercise that a more junior engineer will a) get right consistently or b) think is valuable at all.
1. Coordination of external dependencies. Stuff like product marketing, sales training, documentation, etc. as well as sometimes things external to the company like customer commitments or sales commitments. These are for the most part legitimate, although in many cases the deadline doesn't need to be known at the outset of development, and too many of these is a very unhealthy sign.
2. Accountability / improved productivity - This usually manifests as an exec feeling that engineering is running too slowly / inefficiently, and by asking for deadlines (and often compressing them to "push" the team) they can improve productivity. These are almost always in my mind completely counterproductive, and there's much better ways to accomplish this.
It may feel good to have some of the times quantified. But... If your estimates are literally fiction. It won't stop being fiction in front of the client. You should really spend that energy that went into the estimate for something useful instead.
Not to say you can't derive some useful information from a short planning session. Just don't expect anything other than thumb to the wind guesses.
For context, my team is in exactly this situation with several silicon bugs we found last month.
As a PM, I couldn't possibly agree more with this. When you give me 5-200 days instead of "we can't estimate this", you've started the beginning of a productive conversation. I can ask you questions about why the range is so big, and you can explain to me what steps we can take (research, narrowing scope, etc.), to reduce the range.
Ordinary LoB staff apparently are superstars that have their schedules locked in months into the future, but consultant SREs costing thousands a day can have their time wasted on a daily basis trying to determine how long it will take to complete a task that hasn’t even started yet.
This has literally been my last two months. The dev team lead is asking me — an outsider — how long it’ll take his team to finish something. Not because I know better, but because he doesn’t know either and prefers to make it my fault because I’m expendable.
1. Talk to the person who'll get angry at you when the UAT testers are brought in at the wrong time.
2. Make it clear to them, with a paper trail, that this is an impossible task and you can either book in a best-estimate with the likely outcome that the UAT testers will arrive and there will be nothing to test OR that you can book conservatively and there is a risk the project will be delayed waiting for UAT. They can pick which risk they want to bear.
3. Put in formal feedback, as loudly as possible, that booking in UAT for a specific time introduces obvious risks and someone needs to optimise the process to be more flexible otherwise it will delay projects & waste the businesses' money.
If you're the user or representing the user, shouldn't you be doing the UAT?
You can pull a different estimate of 20-40 out of your behind when pressured to narrow it, and make the deadline 40 days. That might be a 50/50 hit/miss the deadline, with a worse case of running over to the full 200 day worse case (or even worse - even the 200 was a guesstimate). In some cases that's fine too. The decisionmaker might just want a date with 50% chance of hitting because it's better to land a sale and apologize for the delay, than to not land the sale and go bankrupt.
The correct course of action though is to not set the deadline in that meeting. The 5-200 day estimate is a way of explaining "Either don't take this on at all, or at least don't try to schedule anything urgent based on it".
In that case as a developer I'd suggest "Let me look at this for one afternoon, and I can return with an estimate that's 1/10th as large. Then let's meet again ad prioritize some work, schedule UAT and so on". PM's often look really surprised when they realize that all it takes might be to check if there is a functionality available already or if it needs to be implemented - which represented the difference between 5days and 200. If I go off and find it's already there, the estimate is 5-10, if I go off and find there is nothing and it has to be built from scratch, it's 150-200. But without that knowledge: 5-200.
Or, if a 20-200 day estimate is actually due to real implementation uncertainty, I'd suggest "This is potentially large and uncertain. Let's try to break this apart if necessary into something that has a 10-20 day estimate and some parts that have a longer horizon. Schedule the delivery/UAT of the first functionality in 20 days, and we'll by then have a good estimate of when the remaining work will be done".
If someone in my organization would try to convince me to pull a stupid estimate out of thin air, then pressure me to haircut that estimate, then schedule something based on it (such as a delivery), then pressure me to work overtime to meet it, then one of us would be leaving the org very soon.
I quite literally quit my last job in February because of this shit[0]. Stop. No. These are not useful things for developers, much less senior developers, to spend their time on. 95% of my job ended up being coming up with estimates. My top deliverable to my management chain was estimates. They would only care about the actual development work insofar as it validated or invalidated the estimates deliverables.
Hard facts: If time is such an unimportant resource that you're willing to waste most of it doing "research" and worrying about scope, then you don't actually need an estimate. For reporting purposes, just BS one yourself, that's literally your job as a PM.
[0] It was well-paying, I was basically fucking around every week doing whatever I wanted, yet this issue alone outweighed all of that and was enough to make me quit.
I do that and I've become exceptionally good at blaming the business persons for the outcome, making them admit their guilt in public and all that in what seems a friendly, positive and constructive manner and using business motivation and explanation. They never understood what hit them. Every programmer can do that if he or she learns a bit of psychology.
If some manager made a promise to a client out of his/her ass without asking us, and if the PO pressures us towards meeting an unrealistic time-frame, when that explodes I just explain in a public meeting where upper management and all stake holders take part why it was wrong to promise that, using arguments and business considerents because bussineses people don't care about tech considerents. It usually results in that person admitting in public their mistake and even saying they are sorry.
I don't do that to humiliate people but to teach them to ask the team implementing the feature to estimate or to say if that feature is even reasonable or possible given current state and resources.
The estimates aggregate to a number more accurate than each individual. The manager cares more about the sum total than each individual one. Many can even slip without affecting the end date.
This isn't so hard, if management and developers didn't treat each other as enemies. Each can get wrapped around the axle concerning each deadline, rather than the real goal of allocating resources across the entire scope.
The difference is, the law says if a contractor signs a contract to do something at x price, they HAVE to do so or suffer consequences. So contractors go broke/bankrupt, lose their licenses, or figure out how to bid correctly enough they can either weasel out of it if it goes sideways or they have enough wiggle room to deal with the inevitable problems.
Software Engineering Consultants have the same situation. But directly employed software engineers are in a different position of course.
One thing that typically makes this worse, is the implicit assumption that planning/quoting/estimating is ‘free’ work, and since it CAN make things easier to deliver, therefore we should do more and more of it as, since it’s free, it will solve the problem no?
And their disciplines involve massive capital outlays which can’t be refactored in a cost effective way, or at all really.
Not defending the ‘no estimate ever’ approach, it doesn’t work. They are different though.
Tell me, how accurate were the estimates for the launch date of JWST?
I’ll wait here while you look up how many times and how many years that “slipped”.
I bet some dumbass NASA manager asked the team to schedule the launch date back in the 1990s.
It gets even worse when you are in a line of work where the estimates are part of some kind of sales pipeline. Then you aren't just estimating based on effort required, but also against a likely number that the customer will ever agree to.
However, I don't think this idea would be very popular because it removes the manager's ability to control the presentation and play politics.
The correct solution here isn't just to insist estimation is impossible and be offended that someone would ask for it - that just misunderstands how businesses work.
The correct thing is for either of those parties to say, "Okay, I understand it's not estimable at the moment. What are the steps that we need to take in order to get a reasonable estimate?"
A good PM will do this, but where there is a bad PM, a developer can step in and say the same thing. In your scenario, "The developer says “there are too many unknowns to provide any sort of estimate”." What the developer could say is, "There are too many unknowns to provide any sort of estimate - let me give you a few examples. <examples>. What I can do is come back to you after <amount of time> with a list of the unknowns and an explanation of the work required to evaluate them, so that we have enough information to estimate. Once you have that list, then you can decide if we want to proceed with the work to understand those unknowns."
I totally understand why developers get frustrated when they're asked to give estimates for things that can't reasonably be estimated with the current information available. But the flip side is it's also very reasonable for PMs to get frustrated and ignore developers who insist that estimation is an impossibility, because that's virtually never true - maybe it's impossible to do now, but there is a path to make it possible. If you work proactively to expose that path to the PM, it'll generally result in everyone being happier.
Sometimes we don't know what we don't know. If we don't know what we don't know now and if we don't know what we'll find we don't know later, then we don't know how long we will need until we'll know.
Uff. Sorry. If I had more time, I would have written a better comment.
If it's the latter case, then a good developer can describe the immediate steps they'll take and what questions they expect to answer. Whatever the case, there is always some concrete next step you can take to which you can assign a reasonable estimate of time. That might just be that you'll spend one day perusing some documentation, and the only outcome you expect from that is that you'll develop a reasonable enough understanding of how large/complex the documentation is, such that you can come back with an estimate of how long it will take to find and carefully read all the relevant sections of the documentation.
That's totally fine - if it's the sort of wildly unknown problem that you're describing, then I recognize we'll have to take steps to understand the problem. There's still never a case where it's impossible to create an immediate plan of action and understand roughly how long that will take (unless you are simply unqualified to take on the problem).
I had a boss loved the phrase, "a date for a date." He just meant that if you can't give me a date for when the thing will be done, then you need to give me a date for when you can give me a date for when the thing will be done. Can't do that? Date for a date for a date. However many levels we have to go, you're not getting out of whatever meeting we're in until you can give me a date by which you'll be able to come back and give me another date.
I love deadlines. I love the whooshing noise they make as they go by.
- Douglas Adams
Right now I’m replatforming a legacy app. It’s undocumented, developed by dozens of staff — all of whom are gone, and full of crazy bugs and gotchas.
Out of the box it wouldn’t compile, the source code was missing files, etc…
The task is to find every such hidden issue, fix it, which then allows us to find the next hidden issue, and so on.
When we’re done finding issues, we’re basically done.
Can you tell me how many issues we’ll encounter? I can’t. I’m being perpetually surprised by the code base, finding random binary blobs that actually are from some other project we didn’t even know exists until that moment.
Fundamentally, it’ll take as long as it takes. The application is mandated by law, and it must be made to work. It’ll cost what it costs, and it’ll be finished when it’s done.
Instead of begging for estimates, a good manager would be bending over backwards to accelerate the process. Get in the original devs for a week under contract. Procure tooling that helps find bugs faster. Instrument the source servers to get better comparative info. Etc…
I’ve never seen that occur. It’s always just the same question repeated over and over as if that can make the unknown known.
That's exactly it. Scoping out the solution is 90% as hard as just doing it. By the time you've figured out enough of the details that you're sure there are no more surprises waiting... you're just about done.
I suspect that a lot of counter-examples people have witnessed are examples of poorly factored codebases. I've seen teams/projects where they estimated projects accurately because the "new" thing being created was a slight variation on a kind of thing they'd created many times.
For example, I once worked for a company that generated e-commerce sites for clients. They'd make a new one basically by copy/pasting an existing site from another client and changing some text/images here and there. Super predictable after doing this a bunch of times... but also a maintenance nightmare having dozens of slightly different codebases. And how do you iterate? They couldn't evolve.
Shopify ate their lunch. I bet Shopfiy has a really hard time estimating the time to ship a new feature. But they add it once for everybody.
A knowledgeable developer can predict how this cycle goes and explain that an estimate of the whole task is impossible, and they will be quite literally and completely correct in that case.
Ah, that would be nice. In my experience it's more like, "Okay, I understand it's not estimable at the moment. Just make your best guess and we can revise it later. (Psych! I will hold you to this estimate like you were under oath in court!)"
This leads to things like the Scotty rule. Since management will always misuse, misunderstand, or misrepresent your estimate anyway, might as well look out for number one and inflate the estimate, just in case.
The only time I've really had management "misuse, misunderstand, or misrepresent your estimate" is once they got past the point of being several hundred people with the bureaucracy that entails. Then I leave and go to a startup that's <50 people.
In my experience good communication solves this every time.
MGMT: We need an estimate ENG: I have several unknowns MGMT: We really need an estimate ENG: This is my rough estimate, but be aware that the unknowns could change this drastically MGMT: Ok, what do you need to get a firmer estimate.
And so on.
As a project progresses, the communication between ENG and MGMT is crucial to ensure that MGMT are aware how those unknowns are affecting the project, MGMT will decide at every juncture if the project is worthwhile to continue to invest resources into it (assuming they avoid the sunk cost fallacy)
As an industry we just seem incapable of coming up with any other way of negotiating and running projects.
This would allow for planning, prioritising and is the most accurate communication we can achieve.
Because that’s what we do either way. Round numbers from ass make no science to plan and prioritize upon.
1. Assumptions (including soft such as who will actually do it, resources are not equally relevant or productive) 2. Constraints (as agreed by stakeholders) 3. Risks 4. Clear scope 3. Alternatives and why they were not preferred
Business should be aware and own the uncertainties, or else the system gets biased towards sales people offering vaporware.
HOW developers should make their estimation is a totally different question, but it is not some arcane knowledge, and it should be an everyday craft for every mid-level engineer. First of all, we are doing those tasks by hundreds through our careers, so an easy way is to remember how long similar tasks took us last 10 times an produce a bracket, like in "from 5 to 8 working days, let's say 10 to be sure". Every business person will be happy with that answer.
OP says that every engineering task is unique as a piece of art. This is BS, IMO, except for the very junior engineer. Most of the time business wants some well-known and hundred times done things they see their competitors do. All the uniqness stems from particular codebase. If your codebase is a mess - you are in trouble, but it is you problem, not the general problem of software engineering.
In case I have some really unique task, what I should do is to say: "I cannot estimate this right now. Give me two days and I will get you an estimation". In those days I decompose the task into smaller, and divide those smaller tasks into "well known" and "unknown". Then I imagine (or even draw) a trivial Gantt chart and will have my estimation.
It is also totally normal to change your estimation during the work. You just should do that the moment you see a problem, not at the day of the deadline.
> https://www.computer.org/csdl/magazine/so/2011/06/mso2011060...
The author made the claim that software engineering is fundamentally different from carpentry ("Almost everything a developer writes is unique, they have never built that particular thing before. We are not cabinet makers...")
But does that hold up? Many software projects do in fact have a similar analogue, and each house is unique and carpentry projects will thus run into unexpected snags.
The fact that the OP doesn’t see this is fact that they are too junior to be asked for estimates at all.
It really depends on what domain you are programming in.
I found that assumption to be ludicrous too.
Usually a business person makes some promise to the buyer or his/her boss that a certain feature will be done in X amount of time without consulting the team who does the work. The result might be overworking, resentments, stuff not being done in time and technical debt.
In the best case the feature will be done earlier, in the worst case you won't look like an idiot which promised things and didn't deliver and you won't compromise your relationship with the team. Compromising relationships with either clients or tech teams aren't good for your career.
More simply put, try to be as honest as possible. Not over promise, not under deliver.
One of my former girlfriends went to a business school and worked in banking. She thought people in IT have better salaries and she thinked tech people are a bunch of idiots and it's easy to push them to do things and extract a fat paycheck. So she did some courses on IT management and applied to lots of product owner and product manager role. After tens of interviews she hadn't had one success. To add insult to the injury, I showed her I make more money as a team lead than the product owner. Of course, the reverse is true and I've seen product owners being genuinely useful and having lots of business knowledge, some even programing knowledge.
Some people are just after the money, no matter what role they occupy in an IT company. And they tend to be the least useful persons who are blocking others to reach full potential.
Huh? Every major contract job on my house and my friends' houses that I have heard of has taken longer than originally "planned". More often than not unexpected expenses come up along the way. The same is true for commercial construction/remodeling. That's just how these things work; pretending otherwise is going to be a huge source of friction.
The real reason is to provide a scapegoat for the 50% of times it fails (ratio from Mythical Man Month)
"But but but but OUR organization isn't like that!"
Sure.
Relative estimates like planning poker can reveal hidden assumptions.
Commercial estimates can reveal risk appetite, desperation and more.
All estimates create insights but imho. it is best to burn the paper with the number after writing.
I imagine sometimes I renovate my kitchen, an expensive task, within my budget but with not a lot of leeway, I can't renovate the kitchen twice for example. If the workers say they don't know how long it will cost and they won't provide an estimate to when ... I would not be happy.
Even creatives, writers, musicians, graphic artists have deadlines and must estimate their work.
But we feel that our work is special and should not be asked to give an estimate.
I get all the reasons you provided, but at the end of the day an honest discussion needs to ask is estimating tasks effective.
Do developers get more done or less done when estimating tasks vs not estimating tasks? The only claim estimates would increase productivity is the extent they are deadlines. But I think you will see that when developers take meeting their estimates seriously their estimates tend to get more conservative. And then there can be a desire to not over deliver on the estimates to lend those estimates legitimacy. Without estimates, you still have a performance metric to use, individual velocity. If one developer isn't getting stuff done as fast as others, you don't need estimates to know they aren't performing as you would like.
Are estimates reliable? This of course varies greatly by the project and the team but I my experience estimates are typically no more precise than 50% of their length. So a 6 month project probably shouldnt take longer than 9 months. Your mileage may vary, but is there a way to crudely estimate a project without asking for estimates? I would say probably. Why not just measure the teams velocity and project it?
I would argue estimates were a first guess at a way to run a development team which has survived mostly because of a refusal to consider the first guess might not have been the best guess.
"Software development is not carpentry. Almost everything a developer writes is unique, they have never built that particular thing before. We are not cabinet makers repeating a variation of something we've built hundreds of times before."
Some software development fits this description. Quite a lot of it does not.
Having said that, most developers refuse to respond intelligently to these questions. The answer to almost all of these question should be something like "I think delivering this goal might take X resources, but here's the smaller goals we're going to deliver along the way to the big goal so that we can reposition when we go off-track."
Instead, developers say "Well we have to make sure that EVERYTHING can fit into this timebox, and our manager is going to treat this estimate as a deadline, so estimates suck!"
This question is supposed to be a conversation starter. Don't shoot the manager for doing their job. And even when the manager does a poor job and treats your estimates as deadlines, call it out! It's up to the developers to put up a bit of a fight and make more intelligent decisions too.
I've only met a handful of developers who are able to have a conversation like this. It's what the 'Agile' movement was attempting to fix. But it turns out conversations skills are tricky things to impart.
Or perhaps to solve an important problem in the world, within a given scope and cost!
What the PM is not saying, or the quiet part, is that from their perspective "it is your job, do your job." Frankly, that's a piss poor take as well. People are griping about the relationship between engineers and businesses because it's taking a toll. Telling people to shut up and get back to work, worse, alluding their gripes are narcissistic at best is a woefully attrocious take.
Businesses can be better, but part of being better will be taking bets on how to make things better that the engineers (workers) don't solely shoulder or take blame for. Hell, it'd be nice to see the business aspire for internal change where the engineers aren't asked to change at all.
Of course people want it quicker, but this is the problem, stick to the estimate and the problem goes away for the most part. Negotiate on scope if it is too long.
Communicate as early as possible if there is a mistake in the estimate.
At that point the problem largely goes away.
Edit: When communicating about the mistaken estimation, include which assumptions or surprises came up.
If you were right. I’ve never met anybody who could accurately estimate a software project, and I’ve been doing this for 30 years.
The question is one of range, with humility you can hit estimation ranges defined by how familiar the project is and how well defined the project is.
There are many developed techniques that we just don’t implement. By in large, most estimates I have seen were some weird gut. Not defined no milestones. “Yeah, about two weeks boss”
Often estimates come before requirements. Those will be wrong more often than not.
Many times I will estimate the estimate. The “bigger” the project the longer the estimate takes.
People are afraid to admit any knowledge gap.
People switch jobs every 2 years, I have been at the same place 10, and before that 8. You learn the cadence, you learn the pain points. You learn the traps. You learn the domain.
These can be accurate.
But this is not the OP’s issue as stated, he is lowering estimates in the face of pressure. This will nearly always fail.
But they are accountable to their stakeholders (clients, upper management, etc). They take your estimate and many others from other people in the team and work those into a delivery plan.
A good manager will know how to manage risks and remove blockers in a way that gives a developer the best chance to work within the estimate. A bad manager will usually have no plan and put all the blame on the developers if things go awry.
Eg: I've seen many scrum teams transforming story ponts into days or vice versa missing the actual purpose.
I’ve been in the exact same situation a year ago, where another manager got mad at me for a delay. It was caused by him approving his entire team to simultaneously go on 3-month holidays when the COVID travel ban ended. Apparently that was my fault for not estimating accurately.
"Management" or a PMs job will then be to aggregate those estimates so they can estimate at a higher level than you, similar to a project management in a construction site. But each person needs some semblance of an idea of when their part will be done.
The same way if you were remodeling your bathroom and your garden and want to tell your kids when the construction around the house will be done, you'll need the gardener's estimate and the plumber's estimate, plus maybe some buffer at the end, plus some time to go with the wife to get a new shower curtain, etc.
The most difficult part of any "hands on" job is time estimation - two people could give estimates of 3 days or 3 weeks on the same project and both be right.
Half this site is crapping its pants over being replaced by AI being able to do their jobs with a prompt yet it thinks their work is sooo novel and unknowable.
There's a pattern amongst software engineers in these and other discussions. When it comes to discussing management and resource estimation, some people describe software as the most arcane, idiosyncratic craft to exist. In other situations (e.g., where actual research mathematics or computer science is proposed to be used to solve a problem), software engineering has to be "simple", "aligned with best practices", and almost anti-intellectual.
The pattern of discussion (here and elsewhere), to me, indicates a survival instinct of sorts, if that makes sense. To this end, there's a sort of hypocrisy if it means being a software engineer is as unrestrained as possible: software is purported to be like applied mathematics in situations where it's advantageous to describe it as such (e.g., "we can't estimate because proving conjectures is intrinsically unpredictable") , then when the proposition is brought forth to engage with software as applied math (e.g., "team, we can bound the 99%ile perf by proving this recurrence on resource usage, and that'll give us evidence to modify our search algorithm with an adaptive radix tree."), software engineers push it away for something satisfactory to a CS101 student (e.g., "we can just implement auto-completion of our product catalog by searching a sorted array of item names, why complicate it?").
I understand the troubles of estimating resources for tasks and projects, and indeed the requests can sometimes be unrealistic and overbearing—I don't mean to suggest otherwise—but even the most technically sophisticated pieces of software are flush with "ordinary" kinds of programming tasks.
But as someone else said in this thread: If the question is more about who provides the estimate and less about why is there an estimate (also plenty of opinions about that in this thread) then I would say that time estimation is a integral part of an engineers job.
What makes someone an engineer is being able to provide a weighted solution towards the needs of a client or consumer. Not the best solution in the world, but the best solution for that client. If the requirements are that it should be delivered fast, you understand that the quality goes down. If the client wants high quality, it takes more time.
If a hired software developer told me up front, "I can't estimate because you will change things and then the old estimate will be wrong," all I can conclude is that this relationship is already dysfunctional.
Here's the thing, though: you wouldn't be wrong.
It may be different for contract development. For in-house development though, everywhere I've worked we've all always known that the relationship is dysfunctional. We also know that the dysfunction is everyone else's fault, not our own. So we just put up with it, because those assholes will never consent to fixing it, anyway, and just try to shift the blame to us instead.
> the dysfunction is everyone else's fault, not our own
I specifically said the relationship. It takes two parties for effective communication to occur. An engineer who have given up trying because they assume experienced management will not listen, is just as bad as a manager that steamrolls their experienced engineers.
What you do is extract an extra amount of economic output out of your worker, who wouldn't say no to that? Then when it hits the wall, it's the dev's fault anyway. If it works out, you get a pat on the back!
If the dev says NO, the dev will still have to do the work and may even be coerced into doing so. Many junior developers think it's their fault, but they're being manipulated. That's why nobody likes seniors, too.
> They can do that.
Yeah, no, they can't. Not at all.
While some, or a lot of that, is due to incompetence, it is also due to the nature of the beast: Software Development is an iterative process. You have to prototype and figure out what works and what not.
So regardless on how tight your processes are, whether you use agile or waterfall, it will never be possible to give perfect time estimates. You can only improve your risk-management.
Looking at the comments only makes more obvious the snowflakeness and detachment from reality of the "software artists" bunch.
If they hire a guy to (re)model their bathroom, they expect it for free done yesterday. What I mean is that there's usually a complete turnover effect on the demands they have on people working on THEIR money and the lax and forgiving way they expect to be treated on OTHERS people money.
If a client asks me for "an endpoint that accepts a text file and gives me a PDF" I can let them know how long it will take.
Most software projects can't be described like that!
But in software, the planning is the entire project. The closer the plan is to covering every single possible case, the closer the software is to being done. In construction, the work doesn't even start until the plan is done.
Every one of these comparisons that involve a physical product whether it's contracting, bridge building, or manufacturing all completely miss the mark.
If you compare with other industries where the plan is the product, you'll start to see more similarities with software development. And the less the planning is cookie-cutter the more trouble those industries have with estimates as well.
Sometimes, developers literally do R&D without knowing, whether they will be able to solve the task at hand at all. Other times, they use proven solutions for typical problems.
I saw the most issues arises, when management starts to confuse between the two.
From the business perspective, you definitely have to balance the ratio of the two.
We should just accept that significant uncertainty exists in both situations. If you can’t handle risk, liquidate company and invest in t-bills.
This comment from the article was interesting and it matched my experience:
> The law of large numbers only applies if a) the population is sufficiently large, otherwise outliers will dominate b) the estimates are at least mostly independent, otherwise systematic bias will skew the result and c) the estimate is in fact the expected value (mean average) of the probability distribution of time taken. If the estimate is a mode of the probability distribution, then you can not meaningfully add the estimate directly and be able to meaninfully say anything about the resulting distribution.
You can hire contractor to modernize you kitchen or paint wall.. you need some understanding of exactly what is scope of work, what is $ cost, what is the quality, and any other constraint.
the contractor cannot just say i have no idea how long it will take. there has to be some start and end date.
why some programmer think they do not have to provide any estimate at all is perplexing. is your paycheck an infinite money stream? does your employer not have commitment to deliver something by specific date? or do you want someone else to go create estimate of how long YOU need to complete some work?
this is why tools like scrum come in. estimates not a perfect science..so break down scope of work into small and small chunks until we can say “ok…this piece take 3 days”.
As someone who has both built and sold software, it's simply to be able to sell software before it's complete. Without estimates, you can't sell something that doesn't exist yet.
This doesn't change the fact that requiring estimates is a bad idea if you want great software. The best software is built well, then sold.[0]
Great software later is more valuable long-term than bad software now.
--
[0] The best case is actually to sell software without a timeline. But most organizations are not able to operate this way.
If I have a big project and I'm dealing with a plumbing company I might not be talking directly to the plumbers but to their manager. The manager will usually ask for time/complexity estimates from his plumbers before sending me a quote. The manager will usually factor into the quote any risks plus the company's profit margin.
There's a lot of talk about agile, scrum, no-estimates and so on, but a big part of the software industry still works with time estimates and budgets communicated up front.
One is a quick guess with high risk, the other takes time and research to prepare and the risk of running into a delay is lower.
And often the best solution is to have none; perhaps part of the problem is that people are often too afraid or have not enough trust to work without plans or estimates when needed.