> Say that you are an experienced programmer, and you only run into unexpectedly complexity about 5% of the time. If you are starting a new project and split it up into 10 tasks [...] there’s a 40% chance that you will run into a complex task, and blow your estimations for this project.
Except that if I'm hiring someone to build my house, if they tell me "well, I can guarantee you that we'll prioritize iteratively and get it done as fast as possible" I'd tell them to piss off, I can't throw good money at that. You at least need a baseline estimate, and some data as to how often things go over or under.
Apparently software engineers as a group seem to think "fuck you, it's done when we feel like it" is a valid answer?
I disagree, and have always offered estimates, probabilities and contingencies when talking to product owners when I was a lead. I tracked data, and adjusted accordingly, and ended up giving estimates that were statistically sound.
I don't necessarily expect individual contributors to do the same, but anyone that's a lead should be an adult and account for their team's actual productive capacity. This is absolutely expected of a building company, even if you don't expect a single dude with a hammer to be able to estimate how long the project will take.
Given the number of software estimates that fail and the number of software projects that collapse, the vast majority of developers are incapable of doing so.
If you are giving regularly accurate long-term estimates, I'd suggest that you create a consultancy that performs this service or teaches how to do it. Most companies and teams just aren't capable of doing this.
OP means this in the sense of "good estimates for software are impossible", but I submit that the alternate reading also obtains: that estimates for GOOD software are impossible. This is because, pace Brooks, the process of creation is also the process of eliciting the requirements that demand that creation in the first place. "Good" software, software that solves a real problem in a feasible way, is rarely so well-defined at the outset that your only risk factors are difficulties in finding the right library or algorithm. Frequently it is only possible for requirements to be understood once an early version of a solution actually exists.
Perfectly well-defined software is completed software. The act of defining the requirements completely and absolutely is writing the code.
This is what distinguishes software development from other disciplines. If you want to build a bridge, you create the plans and derive your estimate from the plan. But software development is the creation of a plan. Do engineers know exactly how long that planning process takes? Probably not. It's insignificant compared to actual construction. But for software development, that is everything.
Exactly. I think we really should be comparing software development creating processes instead of creating objects.
Look at something like a consultancy that develops safety procedures for organizations or a company that handles building logistics operations.
Software doesn't have the same physical constraints that bridge building does. Anytime you have fewer constraints, you have more possible solutions which take more time to sort through.
As a team lead who lives in this world.=, good software estimates are not impossible. If you are doing the kind of work that you have done before, then it's very possible. If you are doing R&D, then sure it's damn near impossible. But if the work and requirements are clear, then there are plenty of ways to make "Good software estimates".
I fight estimates when I don't have a good business requirement, but tell me what your problem is and the solution you desire and I will give you a good enough estimate. Estimate is never just a date, it also includes the unit of work to be done, and the resources.
The only one of those analogies that even remotely holds is the civil engineering one, and even then, it depends what the engineer (and the developer) are doing. If you're grunting out bridge foundations or pages of A CRUD app then sure, your job will be repetitive. If you're working on AGI or designing the Burj Khalifa then you'll be facing some unique challenges.
So to revisit an earlier question: do you think they should have not estimated? That all other professions should refuse to render estimates because -- gasp -- they might be wrong?
I'm not saying you should refuse to estimate, I'm saying giving an accurate estimate is verging on the impossible. You can attempt to do it, but people shouldn't expect it to be accurate.
I'm just saying they are probably way out, and probably arn't that useful compared to other methods of building software.
A lot businesses people who get this, then ask why bother if they are going to be really inaccurate?
Instead focus on small intervals, work out what you should try to build in those intervals. Implement. Gather feedback. Repeat.
Or even better do the implement and feedback cycle continuously.
If you are not happy on the progress of a feature, cancel it.
Because you doing things in small intervals, and then adjusting. It won't go wildly of the tracks, before realising you have to adjust or cancel it.
> Instead focus on small intervals, work out what you should try to build in those intervals. Implement. Gather feedback. Repeat.
I absolutely agree with you. I think we're closing on the key difference, which is the scope of the estimation.
I am accustomed to estimating the smallest self-contained unit of user-facing value. I'm used to doing it weekly. I'm used to being in a room with my fellow engineers to do a simple points-based estimate of complexity, with projected dates derived from the velocity of the past 3 weeks. I am used to PMs who understand that sometimes we hit iceberg stories, but that most of the time, the projections are good into the next few months.
What I don't have to do is build a magical estimate of the next three years. If I was asked, I would try to based on historical data and research, but the bands would be extremely wide.
Thinking about the analogy that we often make back to construction, the problem is that classical project management is essentially an estimate of an integral. There is a discrete endpoint, and we can estimate (and re-estimate) leading up to that end point. Funds are committed in large chunks and the cheque-signers need high confidence to proceed.
But software is never finished, only abandoned. Estimation has to be differential and constantly updated. If the software is developed to be releasable at all times, commitment can be incremental and early abandonment due to negative feedback is advantageous.
I'm not saying anything new here. But I understand better where you were coming from.
Other professions base their estimates on the blueprints or the plan. You plan out a bridge and then you base on the estimates on that.
Software development is construction of the plan. And all industries also estimate how long planning will take but it's a vastly more significant for software as that's the entire thing.
My point is that estimating the planning (which for software dev is the software dev) isn't necessarily a solved task in any industry. So making such comparisons is meaningless.
Civil engineering and software engineering are fundamentally different because software is digital.
You can't download the building package. You actually have to build the building again. If it's the same building I would expect them reuse the same blueprints. So that holds true.
Where as in software it's quite easy and desirable to reuse software. Which is why you should not be rewriting the same thing over and over.
As a result, the problems you face are unique. Your not just stamping out the same building again and again.
You also have to factor engineering time into the plan. If you read Industrial Megaprojects, the author shows that projects which don't do a bit of upfront estimating and research tend to go off the rails on a scale that software struggles to match.
Some upfront planning is required but, there is an overall problem with comparing software to building construction.
If you make a blueprint and give it to 2 construction companies to build the building, you'll end up with 2 very similar buildings.
If you give a software spec to 2 separate software companies you'll get 2 completely different programs.
You can't create a blueprint/spec that is sufficiently detailed so that 2 different companies/teams will build essential the same software.
A blueprint/spec that detailed would be the code itself.
The closest analog to software would be designing processes, or maybe designing a factory or assembly line. And the analog to the code in that situation would be the factory or assembly line itself.
Software shouldn't be compared to building construction, more appropriately compared to building design. Building construction is analogous to software deployment: a largely solved problem that is predictable.
Err, not really. Completely different workflows. Solving different problems. For different types of people.
They have probably spent a lot of time and money paying UX guys trying to optimise for their unique situation.
I would assume they reusing either in house or 3rd parry libraries in their code, but that stuff isn't where estimates go wrong.
It's usually experimentation of trying to get UI right for the user.
I mean could tell you how long it would take to make a html page, if you tell me what tags and elements you want in it.
It's a whole different ball game, if you ask for a estimate of creating a complete feature because it will probably involve a little trial and error and feeback from multiple people. The first few stabs at it won't be quite right etc
Whatever you're doing, if you're honest with yourself, are you really doing something new and interesting every day/week/month, or are you just digging deeper into what you already know?
I've found that without straight up hopping jobs, it's almost impossible to do truly new things rather than settle into a role where you're too valuable to not exploit for the same old crap you've done before.
I'd still call that doing a good job - businesses need people that can pump out good but similar work. People that always need something new once they master the old thing can be liabilities, depending on the context.
the reason good software estimates are impossible is because the career/political cost of your project being late is less than the career/political cost of making a longer (and more correct) estimate and sticking to it.
I feel like this has been posted before, not too long ago.
And even though it's "impossible", I and hundreds of my peers at work do it every week with a workable degree of accuracy, relative to company-wide work in progress.
In any case, the usual problems are:
1. We software developers think we're the first to face uncertainty, including massive uncertainty. We are not. We are in fact lightweights in this department. Ask a civil, structural or plant engineer for war stories. Or read Industrial Megaprojects to see how skipping a single water-temperature test cost billions of dollars.
2. We tend to embrace the nirvana fallacy. "Perfect or very accurate estimation is worthless; therefore, all estimation is worthless".
3. "Some projects are pure research". Yes. But very few. And even they contain bog-ordinary implementation details. Writing a fantastical new language? Well guess what: parsers have been written before, and someone with data or relevant experience could break down the task into subtasks and estimate them.
I think the real reason that we so dread estimates is:
1. They aren't treated as estimates. People turn them into deadlines. Or try to negotiate with the estimate. Which means they aren't estimates any more.
2. We get them wildly wrong the first few times and throw our hands in the air. Improving estimates is hard. If it were easy, we wouldn't need to consciously do it.
The hard part in estimating a language wouldn't be writing the parser.
It would be experimenting with different ideas for the language, which would require trial and error and experimentation. That would be hard to estimate.
As with most things that programmers do over and over again, creating parsers has been automated. That's the issue with estimation in programming, if you do the same things over and over they get automated. What's left is the unique issues which need to be estimated.
I would not class it as research, but unique problems.
Say you are integrating some software into PayPal. Lots of people have integrated PayPal.
However no one has integrated your software into PayPal before otherwise you wouldn't be doing it.
Your software may have a weird workflow that makes PayPal difficult to integrate. So your going to have figure that out, and that's unique to your software
Your software may not have an easy way integrate 3rd party payment providers. So your going to have figure out the best place for those extension points, in reusable way for multiple payment providers. Figuring out those extension points is unique to your software.
There will be a lot of unique situations.
If however I was using common e-commerce package as opposed to some unique software, I would just install the PayPal add on. no estimation required.
You're right, but it's a categorical argument that doesn't allow for the fact that the world is made of blended probabilities.
Strictly speaking, every keystroke is unique. And strictly speaking, I cannot probably show that I exist independently of someone else's perception, or that I'm not in a computer program and so on and so forth.
Meanwhile, back in people-need-to-make-decisions land, as an engineer I don't get to just shrug when someone asks me for my professional opinion.
I may ask for time to investigate. I may give a wide range due to uncertainty. I may recommend a different approach. I may suggest we bring risky unknowns forward.
But ultimately I give my best estimate. If I get it wrong, that's not a sin. Because it's an estimate.
I don't mind giving estimates, if they used it as estimates, and people internalise that I may have gotten it badly wrong because something wildly unexpected has happened.
In reality though, business try to use estimates to preasure software teams, and punish people who got the estimate wrong.
You also work in a place where it is not acceptable to be performing a task for the first time, as they are of importance enough to require prior experience, and hence usually do not evolve quickly. Software is constantly changing and software developers are much more likely to completely overhaul a product than a physical design. They will also deal with changing dependencies due to updates and, in things like Javascript land, what way the wind happened to blow that day. Software developers are constantly on adapting and uneven ground for most products which strive to keep very up to date.
At places where true software engineering work is required, like mission critical systems, the type of planning that you may be used to will be more common. However even for large companies like Oracle, things like the healthcare site can go wrong [1]. That is just the nature of having to constantly adapt. If things just stayed stagnant, we could get the same level of safety and generations of consistent improvement that things like cars have received, which does happen with maybe Unix or C perhaps. But that does not change the fact that most day-to-day software development is all about hiring someone who knows slightly better than you, but, more importantly, knows how to learn how to get many reasonably new things done which involve parts of what he knows. We don't have the luxury of using safety or danger as a warning, usually.
Given that Pivotal wrote and then donated ownership of an entire containerised cloud platform, built from scratch, before any of Mesos, Kubernetes or Docker existed, I feel like we've done some fairly heavy original engineering.
We estimate every story. Sometimes hilariously wrong (Diego was a classic example of a research-heavy project blasting past the initial gut-check estimates and improving in accuracy as it matured).
But a lot of the time well enough to allow for useful business decisions to be made by ourselves and other members of the Cloud Foundry Foundation.
The reason that I keep saying that useful estimation is possible is because I do it every single week. Hundreds of my peers in Pivotal and partner companies do it every single week. Hundreds of client developers do it every single week.
But I hear that it's impossible. So I guess I'll change my résumé to read "Wizard".
Oracle set that up to fail when they wrote the contract with payment not tied to milestones and parachuted in liberal arts majors with no technical or healthcare background as project managers.
It's because most software is under the umbrella of office people playing grown-up and not something where reality matters or thinking happens. Engineering disciplines, when faced with uncertainty do some math and decide that if the deviation of possible quantities is too close to the tolerable limits of those quantities, you have to adjust one or the other.
High school level math and problem solving plus having ever worked with software clearly state that if you have a software team, you could at the very least collect a single statistic of estimation error (actual effort over expected effort) and get a long-tail (it's going to be more more than it'll be less) distribution. Now you can spec out new projects, get values for expected effort and factor in the error distribution to get a distribution of possible actual effort for the project. If some high percentile of that distribution is more time/money than you can tolerate you have to reduce risks by cutting features or increase the budget (or have a significant fraction of projects fail).
Of course, there's lots more sophistication that could (should) go into this but even this is beyond any part of software industry I've been allowed to participate in, because it gives answers that people with authority find inconvenient.
It occurs to me that software has never had to adopt these practices.
It's an industry with a high gross profit margin. If you have a hit, you can get with just about any process or non-process. So you never need to invest heavily in better estimation.
If you don't have a hit, you will probably go out of business. So you never get the chance to invest heavily in better estimation.
You're talking as if better estimation were an end in itself. It's not. The goal is a successful business. There's no point in spending any more effort on estimation than is needed to support the planning required to make the business a success; any more effort than that is simply wasted.
If the expected ROI on a project is small, then having really good estimates matters a lot, because it can make the difference between profit and loss. But if the expected ROI is large, precise estimates matter less. Tom DeMarco has famously made this point [0].
> There's no point in spending any more effort on estimation than is needed to support the planning required to make the business a success; any more effort than that is simply wasted.
This is basically what I argued. Software businesses often have such fantastic margins that you can do any old thing and still rake it in.
> Or try to negotiate with the estimate. Which means they aren't estimates any more.
Oh yes this annoys me. I've been asked to break it into sub tasks then justify those. So best case estimates based on some predicted sub tasks ... before any design has been done! Then the surprise when it takes longer
As a note, breaking into subtasks does improve estimate accuracy on just about anything, due to the "unpacking effect".
Humans routinely underestimate the time and difficulty involved in just about everything. Breaking a task down into subtasks leads to higher estimates which will typically be more accurate.
You are of course being asked to do something very difficult. In 3-point estimation schemes, you'd give back very wide bands for a few subtasks that you identified. Maybe point at the cone of uncertainty a few times.
I know it often doesn't work. To me, most of the problems with estimating aren't with the estimating.
That's true the problem here is the negotiation process. If you have a technical manager they can use the technical hat to argue the estimate down. And the manager hat is the motivation to do so. I kinda prefer the planning poker where all members are encouraged to have equal say and there is no power imbalance
> They aren't treated as estimates. People turn them into deadlines. Or try to negotiate with the estimate.
I wish I had more than a single upvote to give, because that is absolute gold.
In my previous life I had the dubious pleasure of occasionally doing project estimates. I followed a fairly simple sequence which worked for that particular company and their projects. Please - do not try to blindly use this method, it will give you misleading estimates.
First I checked how much proper knowledge of the problem domain we had in house. Then I estimated (in man-weeks) how long it would take for a single senior engineer to cover a given subtopic, if everything worked the first time. Then, I multiplied that estimate by 2.5 - this covered pretty well the time it _actually_ took them. Then I added a 15% "fuzz factor" to cover for still unexpected surprises.
Multiply that by the number of engineers really needed for the project, and you get a reasonable ballpark estimate of how long in walltime the project would take, as well as how much effort it would require.
And because every single client was always haggling with the estimates, trying to turn them into hard deadlines... I added another +30% on top. We all knew that was the haggling headroom, and expected the salespople and clients to eventually carve that one out.
Our projects were mostly on time and within budget. Yup - mostly.
TL;DR: please do not try the above at home or at work. Those figures worked for a very specific company, for their highly specific projects.
"SIMONYI: We have great difficulty in predicting the time it will take to write a program. There are valid reasons why this is so. It doesn’t mean we shouldn’t try our best to predict, because there are reasons why a prediction might be useful, just like when a weather prediction has economic as well as other benefits.
Really good programs will live forever and take forever to write, at least as long as the hardware exists, and maybe even longer."
1. If you are doing something that people have done before, then it is possible to come up with a reasonable estimate. How? Just talk with someone who has done something similar before and they will help you understand the complexity involved. Example : Rubik's cube.
2. If you are doing something that no one has done before, then it is more difficult to come up with an estimate. Example: Unified theory of forces. In the 1970, people assumed that we are very close. It's been 45 years since then and we aren't any closer.
Of course, I have often made the mistake of underestimating tasks of type 1 because I didn't talk to anyone with enough experience. Maybe this has to do with self confidence. I am very confident of my abilities so I usually tend to underestimate things rather than overestimate them.
Even 1.) is very hard to estimate, and info from people who did something similar is of quite limited value.
Skills, technical environment (system, programming language, libraries, available tooling) and team efficiency vary A LOT.
--
Regarding your comment of overestimation: I think this is a problem for almost everyone. When you get a task described, you often think "hey, this will be easy", without understanding all the requirements in detail, the complexity it will entail or the broader effects in a system. That's where only experience can help (your own or someone elses input)
On a different note, I have heard it being said that it is better to overestimate than underestimate. This is apparently the best way to manage other people's expectations and maintain credibility.
Over is probably the wrong word. I call it realistic estimation. Every stakeholder/business person wants it done yesterday. Their job is to push and prod and squeeze out as much efficiency as possible. It's the estimators job to give a realistic estimate, which often means pushing back to a lot of pressure, and that can be very hard. It's important to realize the most important thing for the business is an estimate as close to reality as possible though. This point is often missed when there is pressure to estimate it shorter, when in fact it is not the estimate the business person wants shorter, but the complete task.
And yes, it does take time to build that trust. Often times when I'm at a new company, the business people are used to short estimates and overruns. I have to train them to understand that while my estimates are longer, they are typically closer to correct. It only takes once or twice of having something done when expected to win them over.
> Over is probably the wrong word. I call it realistic estimation.
I generate realistic estimations by attempting to overestimate and failing to do so sufficiently.
Watching coworkers try to estimate how long it'll take to deliver software really makes you wonder if pessimists actually exist, or if they're rather some kind of myth or mirage.
With regards to 1, the beauty of software is you're almost never doing exactly something that has been done before else you would not have to do it. It may be similar or close in a lot of ways, but those differences are where schedules and budgets go to die.
The best we can do is break tasks that we know about into tiny chunks and estimate those. Then, as we are working and discover the unknown unknowns, break those down and estimate those.
Bad estimation techniques - eg. guesses without a basis in historical data (most developers don't collect historical data) - are usually wrong. No one would argue this point.
If you do iterative development with intelligent planning and collect some historical data to use in estimates the game changes. First, most of the examples of "unexpected complexity" you gave can now be identified early and factored into your estimates. That leaves the 1% of truly unexpected stuff that you can then deal with as it comes up and factor into your future thinking in a post-mortem. Second, since you now identify risks earlier and can factor them into estimates, most of your work becomes estimable, and with historical data your estimates will improve.
There's even multiple process products around this idea already, but they're mostly used in the "safety critical" industry, see [1] [2].
The SEI work has been around a long time and it has had very limited success outside of academic or very controlled settings. It had its heyday in the late 90s but never really caught on broadly because it is impractical to collect the level of detail necessary, with consistent teams, languages, environments, and practices in a fast moving computing environment.
This is an interesting read, mostly because I remember working in the world it was murky and depressing. The unexpected happens, this is an undeniable truth but blowing estimates because of the unexpected should be an edge case and not the norm. There are 3 key thing the team I work with now does to manage the unexpected.
1. Top level commitments should be vague
-the statement "we will build you a website" says nothing about what will be on it
2. Scope the top level commitment, don't over scope, commit only to achievable scope
- If you don't know if something can be done don't commit to it, commit to identifying if it is achievable. Green engineers I have worked with tended to miss deadlines and when we dug into it the problem was often that they took on tasks we didn't even want completed. Commit to delivering something you are certain you can do.
3. Write the scope down, and stick to it.
- A good scoping document is like a good legal contract, it keeps everyone honest while limiting liability. It gives someone a commitment they can hold you to, and it gives you an exit clause if they try to rewrite the rules. If someone tries to change your agreed scope after the fact, adding work means timelines must be adjusted and reducing timelines means work must be removed.
4. If you work somewhere where you can't seem to make 1-3 happen find a new job its just not worth it.
It's even worse than that. Knowledge decay let you run into complex tasks earlier. Your acquired experience gets less useful over time or as stated in this article: Half of what a programmer knows will be useless in 10 years.
The way to access a Rubik's cube stays the same whereas accomplishing a coding task is enviromental dependend. Software changes over time making your experience how to solve a task redundant at some point.
Montgomery Scott: Almost done, sir! You'll be fully automated by the time we dock.
James T. Kirk: Your timing is excellent, Mr. Scott. You've fixed the barn door after the horse has come home. How much refit time before we can take her out again?
Montgomery Scott: Eight weeks, sir -- [Kirk opens his mouth] -- but ye don't have eight weeks, so I'll do it for ye in two.
James T. Kirk: Mr. Scott. Have you always multiplied your repair estimates by a factor of four?
Montgomery Scott: Certainly, sir. How else can I keep my reputation as a miracle worker?
James T. Kirk: [over the intercom] Your reputation is secure, Scotty.
This is good advice and I've actually followed it for estimating software projects. If you say 4x of what you think, and your boss/client doesn't balk, you probably have it right.
I'm roughly between 3x and 10x off, depending on the type of work. For greenfield work on small well-understood tasks and with application and test infrastructure setup, I'm ~3x off. Constantly changing complex applications without tests, I'm ~10x off. However even these multipliers have a 1-20x margin of error.
"invest in automated, repeatable practices (i.e. true Engineering)."
We do. We call those "libraries". We have more of them than most disciplines could dream of, and they work better than most disciplines could dream of.
So where most disciplines are taking the closest things to libraries they have and still applying a lot of drudge work to put them into the current context, with "unexpected work" forming small single digit percentages of the work, we get to yank all the easy drudge work off the shelf and the resulting "unexpected work" makes up the majority of what we're doing.
I'm not going to pretend everything is perfect in the land of software engineering, but the answer to "why are we so stupid that we don't do the obvious thing that people have been complaining about for 50 years now?" is that we aren't that stupid. We are very good at shrinking the obvious, easy parts of the problem down, so all that's left is the hard part. Which, since everything isn't hunky dory in the world of software engineering, we also often add to with self-inflicted injuries. Still, the amount of off-the-shelf crap that I can just push a button and have staggers the mind.
If "real engineering" worked like computer science, we'd be down to a guy pointing and five minutes of fiddling with his phone to lay a bridge down across the average river, four minutes of which would be rebuilding the bridge in various different aesthetic styles to see which suited the location best. "Do I like the fake aged wood look, or do I go with modernistic concrete? Let's see the wooden one again...."
Your homework is to write "hello world" in your favorite language and step through the resulting program in gdb or equivalent, starting from the moment the OS turns control over to the program, one assembler instruction at a time.
I apologize if this comes off a touch crabby, though upon re-reading I'm not sure exactly what to tone down about it. I'm getting really tired of the inferiority complex programmers have about "real engineering". Yes, we do real engineering. If other engineers had the advantages we had, they'd be much more "agile", and they'd make the same messes we do. They work the way they do because of the constraints they have on them, for exactly the same reason the space programmers work the way they do and for exactly the same reasons it would be wildly inappropriate for all of us to work the same way NASA does all the time.
Great points! One other remark I'll share about software engineering is that the amount of engineering rigor required or warranted depends a lot upon the task and problem.
If I'm building a back-office website for a few colleagues, I may not invest the same rigor as if I'm building a website that will be accessed by hundreds of millions of people. Many software tasks do not require great rigor: you add rigor until the incremental benefit is less than the incremental cost.
As a result, I would guess that the average software engineering task is less rigorous than the average real-world engineering task, but that's only because real-world engineering frequently risks or protect lives. Even the tire rope swing, the hammock, and the treehouse put lives at risk if built poorly. The back-office website on average does not. Business might be disrupted or individuals might be inconvenienced, but typically no one can die.
Yet, many types of software involve just as much rigor if not more as other engineering disciplines. For an example, read "They Write the Right Stuff", a story of NASA avionics software engineers [0]. I consider software engineering to be an engineering discipline because of the fact that rigor is required, and because a good engineer must know how much of it is appropriate, and be able to apply it effectively. A software engineer produces software to specified reliability and tolerances, according to requirements.
"Real engineering" does not mean building things. Real engineering means knowing how to model processes - so that when you actually do build something, your model gives you a strong signal ahead of time that it's going to do what you want.
Engineering is not libraries or cook books: engineering is math.
Civil engineers model buildings and other structures. Electronic engineers model circuit behaviours. Aerospace engineers model aerodynamics and materials. Traffic engineers model traffic flows.
Stuff only gets built after a model appears to work. This is a good thing, because otherwise we'd be build bridges with rule of thumb and trial and error, and they'd fall down a lot, and kill people.
Knowing the math makes it as easier to model a business case, complete with deadlines, as a process.
Why isn't software engineering like this? It's not impossible to model software. It's just conceptually demanding, and very expensive.
Professional software engineering doesn't look much like vanilla coding. The percentage of developers who can hack it - or could ever hack it - is not large.
So we muddle along as a trade, not an engineering profession, hammering and gluing libraries together.
The pay is still good, hardly anyone gets asked if they know what the ACM is in job interviews, and a lot of customers don't mind that much if a site only works in Safari and not in Chrome. (NB: actual experience I had with my online banking account last week.)
It's true that being asked to estimate how long something will take may be met with a sharp intake of breath. But the reality is that very similar projects have usually been completed many times in other companies already, and the amount of truly unique and original R&D needed for most work need is miniscule.
You're offering a lot of derision without any useful information. If you've discovered the secret to repeatable, predictable software development, by all means please share it.
What I've discovered is that software is craft. Everyone progresses through phases of apprenticeship, journeyman, and master; with the art of "estimation" progressively improving.
Yet when we call it Engineering, we hold ourselves accountable to a different set of expectations.
Is that supposed to be insightful? The first part is obvious, the second part is nonsense. Neither offers a secret to repeatable, predictable software development. If you're going to make a fantastic claim that you can make software engineering time estimations reliable you should back that up with how you do that. Saying "I've been doing it for a long time and I call it engineering." Is not enough.
Software people have dealt with nothing like the number of bodies buried under the Great Wall its engineers had. "Engineering" came from sappers, originally.
Engineers work for insurers. For the moment, software engineers work for people other than insurers. I'm not quite ready to turn my culture over to the actuaries just yet.
You're not grokking the scope of "software engineering". Software itself is pure, arbitrary logic. Advances in microprocessors, networks, and storage has blown up the scope of what software can accomplishment to unfathomable scales. It's as if the humble hobbyist carpenter building a doghouse on the weekend had to know the intimate details of quarks, atoms, molecules, all the way up to the interaction of planets, star systems and galaxies, except without the immutable laws of physics, just layer upon layer of human-invented abstraction with no guarantee of universality.
The blessing and the curse that makes software unique is that you get to make the rules, all of them. Let that sink in for a minute because if you look at all other engineering practices they are all based on the rigorous and immutable constraints they have to deal with.
Any set of automated practices for producing software will only make sense for a tiny slice of software for which a set of constraints can be reasonably agreed upon.
This is wrong. If the variables that affect estimates can be predicted, then they can be accounted for.
Source: i used to work at an organization you've heard of that could reliably predict, down to the day, how long it would take ~4000 engineers to build and release the next version of its product. Three years in advance.
I almost think it's far easier to estimate the output of 4000 engineers than the output of 10. It's also far easier to cut things to fit a deadline 3 years out.
One thing i'm a big fan of is the 'eta till the eta'. So many developers/managers often shoot from the hip with estimates - usually with the rationale that if it's all made up anyway might as well seem smart.
Yet if you spend just like 20 minutes to try and break down the tasks and do some research you can usually reduce the margin of error significantly. Our brains are good at ignoring complexity and our egos are good at telling ourselves we understand things we don't. Break it down and do you research. Stuff like poking through github repos of libraries you might be adopting. Lots of forks? lacking documentation/tests? What kinds of issues? Who is using it and why? You still can certainly still run into unexpected complexity but you get good at recognizing where it's likely to pop up. It's hard, and you won't get it right always, but it doesn't mean that shit is 'impossible'.
I've always thought, if you can, with reasonable accuracy, predict how long it will take to build a piece of software, then it's probably already done, and you should just apt-get it or whatever.
However, if it's something truly novel, then kind of by definition, it's impossible to predict how long it will take, because it's never been done before.
Of course, there's a million shades of gray in between, but it's not like bridge building. Bridges are pretty unchanged in 1000's of years. Most software you write, should be brand new, otherwise, why do it?
Heh. How long does apt-cache search take. Kidding, and I do recall that being a real problem for most things at one point. You'd go to FreshMeat and just hope that the highest rated one was good. But, re: apt, if it gets in the main repo, then I'm hard-pressed to think I'm smarter than all those people.
That said, for some classes of software, Drupal plugins and Minecraft mods come to mind, are all varying degrees of abysmal. Those take a lot longer to sort through.
Things like "evidence based scheduling" and your permutations around "velocity" and burndown charts seem to help, particularly over time, but I've found no magic bullet.
If you have done it before it's easy to estimate how much it's going to take. Some things are actually really hard to estimate, like estimating how much root causing and fixing a bug will take.
Now, if you want to estimate properly... ask for a min time estimate, max time estimate, and confidence level.
To say "I am not sure", is actually fine. That can be the start of a conversation where you can have the opportunity to improve that estimate. Whereas a closed answer like "4 hours" won't.
Good software estimates are impossible because business stake holders don't like the true amount of time to get software done and always push to shorten the estimates.
103 comments
[ 3.0 ms ] story [ 208 ms ] threadSolution: divide the project into fewer tasks :)
Prioritized (short) iterative development is the only path to success.
Apparently software engineers as a group seem to think "fuck you, it's done when we feel like it" is a valid answer?
I disagree, and have always offered estimates, probabilities and contingencies when talking to product owners when I was a lead. I tracked data, and adjusted accordingly, and ended up giving estimates that were statistically sound.
I don't necessarily expect individual contributors to do the same, but anyone that's a lead should be an adult and account for their team's actual productive capacity. This is absolutely expected of a building company, even if you don't expect a single dude with a hammer to be able to estimate how long the project will take.
If you are giving regularly accurate long-term estimates, I'd suggest that you create a consultancy that performs this service or teaches how to do it. Most companies and teams just aren't capable of doing this.
Please explain exactly what you mean by this.
This is what distinguishes software development from other disciplines. If you want to build a bridge, you create the plans and derive your estimate from the plan. But software development is the creation of a plan. Do engineers know exactly how long that planning process takes? Probably not. It's insignificant compared to actual construction. But for software development, that is everything.
Look at something like a consultancy that develops safety procedures for organizations or a company that handles building logistics operations.
Software doesn't have the same physical constraints that bridge building does. Anytime you have fewer constraints, you have more possible solutions which take more time to sort through.
As a team lead who lives in this world.=, good software estimates are not impossible. If you are doing the kind of work that you have done before, then it's very possible. If you are doing R&D, then sure it's damn near impossible. But if the work and requirements are clear, then there are plenty of ways to make "Good software estimates".
I fight estimates when I don't have a good business requirement, but tell me what your problem is and the solution you desire and I will give you a good enough estimate. Estimate is never just a date, it also includes the unit of work to be done, and the resources.
> If you're doing similar work over and over again as a civil engineer, you're not doing a good job.
Hmm. Not sure I feel comfortable with that.
> If you're doing similar work over and over again as a small business accountant, you're not doing a good job.
Actually, I'd rather prefer you to stick to the basics, please, no need for discussions of complex derivatives of treasury stock.
> If you're doing similar work over and over again as an orthopaedic surgeon, you're not doing a good job.
Thanks, but I think I'll get a second opinion.
I expect those engineers still gave estimates and were chosen because they had the most relevant experience and expertise.
I'm just saying they are probably way out, and probably arn't that useful compared to other methods of building software.
A lot businesses people who get this, then ask why bother if they are going to be really inaccurate?
Instead focus on small intervals, work out what you should try to build in those intervals. Implement. Gather feedback. Repeat.
Or even better do the implement and feedback cycle continuously.
If you are not happy on the progress of a feature, cancel it.
Because you doing things in small intervals, and then adjusting. It won't go wildly of the tracks, before realising you have to adjust or cancel it.
I absolutely agree with you. I think we're closing on the key difference, which is the scope of the estimation.
I am accustomed to estimating the smallest self-contained unit of user-facing value. I'm used to doing it weekly. I'm used to being in a room with my fellow engineers to do a simple points-based estimate of complexity, with projected dates derived from the velocity of the past 3 weeks. I am used to PMs who understand that sometimes we hit iceberg stories, but that most of the time, the projections are good into the next few months.
What I don't have to do is build a magical estimate of the next three years. If I was asked, I would try to based on historical data and research, but the bands would be extremely wide.
Thinking about the analogy that we often make back to construction, the problem is that classical project management is essentially an estimate of an integral. There is a discrete endpoint, and we can estimate (and re-estimate) leading up to that end point. Funds are committed in large chunks and the cheque-signers need high confidence to proceed.
But software is never finished, only abandoned. Estimation has to be differential and constantly updated. If the software is developed to be releasable at all times, commitment can be incremental and early abandonment due to negative feedback is advantageous.
I'm not saying anything new here. But I understand better where you were coming from.
Software development is construction of the plan. And all industries also estimate how long planning will take but it's a vastly more significant for software as that's the entire thing.
Which means that we're doing it more often, so we have more data and experience to call on when estimating.
And it means it's more important to estimate the design task, since it's vastly more significant.
They're estimates. They're not called "certains", "definites" or "numerical solutions" because they are none of these things.
You can't download the building package. You actually have to build the building again. If it's the same building I would expect them reuse the same blueprints. So that holds true.
Where as in software it's quite easy and desirable to reuse software. Which is why you should not be rewriting the same thing over and over.
As a result, the problems you face are unique. Your not just stamping out the same building again and again.
You also have to factor engineering time into the plan. If you read Industrial Megaprojects, the author shows that projects which don't do a bit of upfront estimating and research tend to go off the rails on a scale that software struggles to match.
If you make a blueprint and give it to 2 construction companies to build the building, you'll end up with 2 very similar buildings.
If you give a software spec to 2 separate software companies you'll get 2 completely different programs.
You can't create a blueprint/spec that is sufficiently detailed so that 2 different companies/teams will build essential the same software.
A blueprint/spec that detailed would be the code itself.
The closest analog to software would be designing processes, or maybe designing a factory or assembly line. And the analog to the code in that situation would be the factory or assembly line itself.
It's that professionals perform similar tasks very often. Including us.
Massive amount of very similar work being done. How much shared code?
They have probably spent a lot of time and money paying UX guys trying to optimise for their unique situation.
I would assume they reusing either in house or 3rd parry libraries in their code, but that stuff isn't where estimates go wrong.
It's usually experimentation of trying to get UI right for the user.
I mean could tell you how long it would take to make a html page, if you tell me what tags and elements you want in it.
It's a whole different ball game, if you ask for a estimate of creating a complete feature because it will probably involve a little trial and error and feeback from multiple people. The first few stabs at it won't be quite right etc
I've found that without straight up hopping jobs, it's almost impossible to do truly new things rather than settle into a role where you're too valuable to not exploit for the same old crap you've done before.
I'd still call that doing a good job - businesses need people that can pump out good but similar work. People that always need something new once they master the old thing can be liabilities, depending on the context.
They have issues with, let's integrate x product with our product.
And even though it's "impossible", I and hundreds of my peers at work do it every week with a workable degree of accuracy, relative to company-wide work in progress.
In any case, the usual problems are:
1. We software developers think we're the first to face uncertainty, including massive uncertainty. We are not. We are in fact lightweights in this department. Ask a civil, structural or plant engineer for war stories. Or read Industrial Megaprojects to see how skipping a single water-temperature test cost billions of dollars.
2. We tend to embrace the nirvana fallacy. "Perfect or very accurate estimation is worthless; therefore, all estimation is worthless".
3. "Some projects are pure research". Yes. But very few. And even they contain bog-ordinary implementation details. Writing a fantastical new language? Well guess what: parsers have been written before, and someone with data or relevant experience could break down the task into subtasks and estimate them.
I think the real reason that we so dread estimates is:
1. They aren't treated as estimates. People turn them into deadlines. Or try to negotiate with the estimate. Which means they aren't estimates any more.
2. We get them wildly wrong the first few times and throw our hands in the air. Improving estimates is hard. If it were easy, we wouldn't need to consciously do it.
It would be experimenting with different ideas for the language, which would require trial and error and experimentation. That would be hard to estimate.
As with most things that programmers do over and over again, creating parsers has been automated. That's the issue with estimation in programming, if you do the same things over and over they get automated. What's left is the unique issues which need to be estimated.
Is it your sincere, genuine contention that 100% of software development is pure research?
Say you are integrating some software into PayPal. Lots of people have integrated PayPal.
However no one has integrated your software into PayPal before otherwise you wouldn't be doing it.
Your software may have a weird workflow that makes PayPal difficult to integrate. So your going to have figure that out, and that's unique to your software
Your software may not have an easy way integrate 3rd party payment providers. So your going to have figure out the best place for those extension points, in reusable way for multiple payment providers. Figuring out those extension points is unique to your software.
There will be a lot of unique situations.
If however I was using common e-commerce package as opposed to some unique software, I would just install the PayPal add on. no estimation required.
Strictly speaking, every keystroke is unique. And strictly speaking, I cannot probably show that I exist independently of someone else's perception, or that I'm not in a computer program and so on and so forth.
Meanwhile, back in people-need-to-make-decisions land, as an engineer I don't get to just shrug when someone asks me for my professional opinion.
I may ask for time to investigate. I may give a wide range due to uncertainty. I may recommend a different approach. I may suggest we bring risky unknowns forward.
But ultimately I give my best estimate. If I get it wrong, that's not a sin. Because it's an estimate.
In reality though, business try to use estimates to preasure software teams, and punish people who got the estimate wrong.
I'm only such a stick in the mud about the topic because I'm lucky to work in a place where estimates are respected as estimates.
At places where true software engineering work is required, like mission critical systems, the type of planning that you may be used to will be more common. However even for large companies like Oracle, things like the healthcare site can go wrong [1]. That is just the nature of having to constantly adapt. If things just stayed stagnant, we could get the same level of safety and generations of consistent improvement that things like cars have received, which does happen with maybe Unix or C perhaps. But that does not change the fact that most day-to-day software development is all about hiring someone who knows slightly better than you, but, more importantly, knows how to learn how to get many reasonably new things done which involve parts of what he knows. We don't have the luxury of using safety or danger as a warning, usually.
[1]: http://www.bizjournals.com/sanjose/news/2013/12/03/cover-ore...
We estimate every story. Sometimes hilariously wrong (Diego was a classic example of a research-heavy project blasting past the initial gut-check estimates and improving in accuracy as it matured).
But a lot of the time well enough to allow for useful business decisions to be made by ourselves and other members of the Cloud Foundry Foundation.
The reason that I keep saying that useful estimation is possible is because I do it every single week. Hundreds of my peers in Pivotal and partner companies do it every single week. Hundreds of client developers do it every single week.
But I hear that it's impossible. So I guess I'll change my résumé to read "Wizard".
High school level math and problem solving plus having ever worked with software clearly state that if you have a software team, you could at the very least collect a single statistic of estimation error (actual effort over expected effort) and get a long-tail (it's going to be more more than it'll be less) distribution. Now you can spec out new projects, get values for expected effort and factor in the error distribution to get a distribution of possible actual effort for the project. If some high percentile of that distribution is more time/money than you can tolerate you have to reduce risks by cutting features or increase the budget (or have a significant fraction of projects fail).
Of course, there's lots more sophistication that could (should) go into this but even this is beyond any part of software industry I've been allowed to participate in, because it gives answers that people with authority find inconvenient.
They are looking for estimates to hold you to account, because that's how you run a good business right?
If you gave your answer to someone in management. It would just look like your trying obfuscate the answer.
There's a difference between estimates and their misuse.
Or huge range of possible release dates(because there is huge variation between the estimate and actuals), as with other estimation methods.
It's an industry with a high gross profit margin. If you have a hit, you can get with just about any process or non-process. So you never need to invest heavily in better estimation.
If you don't have a hit, you will probably go out of business. So you never get the chance to invest heavily in better estimation.
If the expected ROI on a project is small, then having really good estimates matters a lot, because it can make the difference between profit and loss. But if the expected ROI is large, precise estimates matter less. Tom DeMarco has famously made this point [0].
[0] http://www.systemsguild.com/pdfs/DeMarcoNov2011.pdf
This is basically what I argued. Software businesses often have such fantastic margins that you can do any old thing and still rake it in.
Oh yes this annoys me. I've been asked to break it into sub tasks then justify those. So best case estimates based on some predicted sub tasks ... before any design has been done! Then the surprise when it takes longer
Humans routinely underestimate the time and difficulty involved in just about everything. Breaking a task down into subtasks leads to higher estimates which will typically be more accurate.
You are of course being asked to do something very difficult. In 3-point estimation schemes, you'd give back very wide bands for a few subtasks that you identified. Maybe point at the cone of uncertainty a few times.
I know it often doesn't work. To me, most of the problems with estimating aren't with the estimating.
I wish I had more than a single upvote to give, because that is absolute gold.
In my previous life I had the dubious pleasure of occasionally doing project estimates. I followed a fairly simple sequence which worked for that particular company and their projects. Please - do not try to blindly use this method, it will give you misleading estimates.
First I checked how much proper knowledge of the problem domain we had in house. Then I estimated (in man-weeks) how long it would take for a single senior engineer to cover a given subtopic, if everything worked the first time. Then, I multiplied that estimate by 2.5 - this covered pretty well the time it _actually_ took them. Then I added a 15% "fuzz factor" to cover for still unexpected surprises.
Multiply that by the number of engineers really needed for the project, and you get a reasonable ballpark estimate of how long in walltime the project would take, as well as how much effort it would require.
And because every single client was always haggling with the estimates, trying to turn them into hard deadlines... I added another +30% on top. We all knew that was the haggling headroom, and expected the salespople and clients to eventually carve that one out.
Our projects were mostly on time and within budget. Yup - mostly.
TL;DR: please do not try the above at home or at work. Those figures worked for a very specific company, for their highly specific projects.
"SIMONYI: We have great difficulty in predicting the time it will take to write a program. There are valid reasons why this is so. It doesn’t mean we shouldn’t try our best to predict, because there are reasons why a prediction might be useful, just like when a weather prediction has economic as well as other benefits.
Really good programs will live forever and take forever to write, at least as long as the hardware exists, and maybe even longer."
Source: https://programmersatwork.wordpress.com/programmers-at-work-... (great book, btw!).
1. If you are doing something that people have done before, then it is possible to come up with a reasonable estimate. How? Just talk with someone who has done something similar before and they will help you understand the complexity involved. Example : Rubik's cube.
2. If you are doing something that no one has done before, then it is more difficult to come up with an estimate. Example: Unified theory of forces. In the 1970, people assumed that we are very close. It's been 45 years since then and we aren't any closer.
Of course, I have often made the mistake of underestimating tasks of type 1 because I didn't talk to anyone with enough experience. Maybe this has to do with self confidence. I am very confident of my abilities so I usually tend to underestimate things rather than overestimate them.
Skills, technical environment (system, programming language, libraries, available tooling) and team efficiency vary A LOT.
-- Regarding your comment of overestimation: I think this is a problem for almost everyone. When you get a task described, you often think "hey, this will be easy", without understanding all the requirements in detail, the complexity it will entail or the broader effects in a system. That's where only experience can help (your own or someone elses input)
And yes, it does take time to build that trust. Often times when I'm at a new company, the business people are used to short estimates and overruns. I have to train them to understand that while my estimates are longer, they are typically closer to correct. It only takes once or twice of having something done when expected to win them over.
I generate realistic estimations by attempting to overestimate and failing to do so sufficiently.
Watching coworkers try to estimate how long it'll take to deliver software really makes you wonder if pessimists actually exist, or if they're rather some kind of myth or mirage.
The best we can do is break tasks that we know about into tiny chunks and estimate those. Then, as we are working and discover the unknown unknowns, break those down and estimate those.
Bad estimation techniques - eg. guesses without a basis in historical data (most developers don't collect historical data) - are usually wrong. No one would argue this point.
If you do iterative development with intelligent planning and collect some historical data to use in estimates the game changes. First, most of the examples of "unexpected complexity" you gave can now be identified early and factored into your estimates. That leaves the 1% of truly unexpected stuff that you can then deal with as it comes up and factor into your future thinking in a post-mortem. Second, since you now identify risks earlier and can factor them into estimates, most of your work becomes estimable, and with historical data your estimates will improve.
There's even multiple process products around this idea already, but they're mostly used in the "safety critical" industry, see [1] [2].
[1]: http://resources.sei.cmu.edu/library/asset-view.cfm?assetid=... [2]: http://www.sei.cmu.edu/tsp/
For people having trouble with this, Jim Shore does a good job giving tips: http://www.jamesshore.com/Agile-Book/estimating.html
"Estimates or No Estimates?"
https://vimeo.com/145049619
1. Top level commitments should be vague
-the statement "we will build you a website" says nothing about what will be on it
2. Scope the top level commitment, don't over scope, commit only to achievable scope
- If you don't know if something can be done don't commit to it, commit to identifying if it is achievable. Green engineers I have worked with tended to miss deadlines and when we dug into it the problem was often that they took on tasks we didn't even want completed. Commit to delivering something you are certain you can do.
3. Write the scope down, and stick to it.
- A good scoping document is like a good legal contract, it keeps everyone honest while limiting liability. It gives someone a commitment they can hold you to, and it gives you an exit clause if they try to rewrite the rules. If someone tries to change your agreed scope after the fact, adding work means timelines must be adjusted and reducing timelines means work must be removed.
4. If you work somewhere where you can't seem to make 1-3 happen find a new job its just not worth it.
Developing software also involves discovering the real user issues by experimentation.
If you scope it, you may deliver the project, but you probsbly haven't delivered what they actually needed.
The best software is developed with feedback loops.
The way to access a Rubik's cube stays the same whereas accomplishing a coding task is enviromental dependend. Software changes over time making your experience how to solve a task redundant at some point.
Montgomery Scott: Almost done, sir! You'll be fully automated by the time we dock.
James T. Kirk: Your timing is excellent, Mr. Scott. You've fixed the barn door after the horse has come home. How much refit time before we can take her out again?
Montgomery Scott: Eight weeks, sir -- [Kirk opens his mouth] -- but ye don't have eight weeks, so I'll do it for ye in two.
James T. Kirk: Mr. Scott. Have you always multiplied your repair estimates by a factor of four?
Montgomery Scott: Certainly, sir. How else can I keep my reputation as a miracle worker?
James T. Kirk: [over the intercom] Your reputation is secure, Scotty.
https://m.youtube.com/watch?v=t9SVhg6ZENw
By all means, take risks and explore Software. But do it under the premise of Software Development if Engineering practices will not be applied.
In agile environments, push for more "spikes" if estimates cannot reliably be provided for a story.
We do. We call those "libraries". We have more of them than most disciplines could dream of, and they work better than most disciplines could dream of.
So where most disciplines are taking the closest things to libraries they have and still applying a lot of drudge work to put them into the current context, with "unexpected work" forming small single digit percentages of the work, we get to yank all the easy drudge work off the shelf and the resulting "unexpected work" makes up the majority of what we're doing.
I'm not going to pretend everything is perfect in the land of software engineering, but the answer to "why are we so stupid that we don't do the obvious thing that people have been complaining about for 50 years now?" is that we aren't that stupid. We are very good at shrinking the obvious, easy parts of the problem down, so all that's left is the hard part. Which, since everything isn't hunky dory in the world of software engineering, we also often add to with self-inflicted injuries. Still, the amount of off-the-shelf crap that I can just push a button and have staggers the mind.
If "real engineering" worked like computer science, we'd be down to a guy pointing and five minutes of fiddling with his phone to lay a bridge down across the average river, four minutes of which would be rebuilding the bridge in various different aesthetic styles to see which suited the location best. "Do I like the fake aged wood look, or do I go with modernistic concrete? Let's see the wooden one again...."
Your homework is to write "hello world" in your favorite language and step through the resulting program in gdb or equivalent, starting from the moment the OS turns control over to the program, one assembler instruction at a time.
I apologize if this comes off a touch crabby, though upon re-reading I'm not sure exactly what to tone down about it. I'm getting really tired of the inferiority complex programmers have about "real engineering". Yes, we do real engineering. If other engineers had the advantages we had, they'd be much more "agile", and they'd make the same messes we do. They work the way they do because of the constraints they have on them, for exactly the same reason the space programmers work the way they do and for exactly the same reasons it would be wildly inappropriate for all of us to work the same way NASA does all the time.
If I'm building a back-office website for a few colleagues, I may not invest the same rigor as if I'm building a website that will be accessed by hundreds of millions of people. Many software tasks do not require great rigor: you add rigor until the incremental benefit is less than the incremental cost.
As a result, I would guess that the average software engineering task is less rigorous than the average real-world engineering task, but that's only because real-world engineering frequently risks or protect lives. Even the tire rope swing, the hammock, and the treehouse put lives at risk if built poorly. The back-office website on average does not. Business might be disrupted or individuals might be inconvenienced, but typically no one can die.
Yet, many types of software involve just as much rigor if not more as other engineering disciplines. For an example, read "They Write the Right Stuff", a story of NASA avionics software engineers [0]. I consider software engineering to be an engineering discipline because of the fact that rigor is required, and because a good engineer must know how much of it is appropriate, and be able to apply it effectively. A software engineer produces software to specified reliability and tolerances, according to requirements.
[0] https://www.fastcompany.com/28121/they-write-right-stuff
Engineering is not libraries or cook books: engineering is math.
Civil engineers model buildings and other structures. Electronic engineers model circuit behaviours. Aerospace engineers model aerodynamics and materials. Traffic engineers model traffic flows.
Stuff only gets built after a model appears to work. This is a good thing, because otherwise we'd be build bridges with rule of thumb and trial and error, and they'd fall down a lot, and kill people.
Knowing the math makes it as easier to model a business case, complete with deadlines, as a process.
Why isn't software engineering like this? It's not impossible to model software. It's just conceptually demanding, and very expensive.
Professional software engineering doesn't look much like vanilla coding. The percentage of developers who can hack it - or could ever hack it - is not large.
So we muddle along as a trade, not an engineering profession, hammering and gluing libraries together.
The pay is still good, hardly anyone gets asked if they know what the ACM is in job interviews, and a lot of customers don't mind that much if a site only works in Safari and not in Chrome. (NB: actual experience I had with my online banking account last week.)
It's true that being asked to estimate how long something will take may be met with a sharp intake of breath. But the reality is that very similar projects have usually been completed many times in other companies already, and the amount of truly unique and original R&D needed for most work need is miniscule.
Yet when we call it Engineering, we hold ourselves accountable to a different set of expectations.
Engineers work for insurers. For the moment, software engineers work for people other than insurers. I'm not quite ready to turn my culture over to the actuaries just yet.
The blessing and the curse that makes software unique is that you get to make the rules, all of them. Let that sink in for a minute because if you look at all other engineering practices they are all based on the rigorous and immutable constraints they have to deal with.
Any set of automated practices for producing software will only make sense for a tiny slice of software for which a set of constraints can be reasonably agreed upon.
Source: i used to work at an organization you've heard of that could reliably predict, down to the day, how long it would take ~4000 engineers to build and release the next version of its product. Three years in advance.
Yet if you spend just like 20 minutes to try and break down the tasks and do some research you can usually reduce the margin of error significantly. Our brains are good at ignoring complexity and our egos are good at telling ourselves we understand things we don't. Break it down and do you research. Stuff like poking through github repos of libraries you might be adopting. Lots of forks? lacking documentation/tests? What kinds of issues? Who is using it and why? You still can certainly still run into unexpected complexity but you get good at recognizing where it's likely to pop up. It's hard, and you won't get it right always, but it doesn't mean that shit is 'impossible'.
Mathematical Limits to Software Estimation: Supplementary Material: http://scribblethink.org/Work/Softestim/softestim.html
However, if it's something truly novel, then kind of by definition, it's impossible to predict how long it will take, because it's never been done before.
Of course, there's a million shades of gray in between, but it's not like bridge building. Bridges are pretty unchanged in 1000's of years. Most software you write, should be brand new, otherwise, why do it?
This leads to the next quagmire:
How do you reliably estimate how long it takes to find the already written software that does X?
That said, for some classes of software, Drupal plugins and Minecraft mods come to mind, are all varying degrees of abysmal. Those take a lot longer to sort through.
Things like "evidence based scheduling" and your permutations around "velocity" and burndown charts seem to help, particularly over time, but I've found no magic bullet.
Now, if you want to estimate properly... ask for a min time estimate, max time estimate, and confidence level.
To say "I am not sure", is actually fine. That can be the start of a conversation where you can have the opportunity to improve that estimate. Whereas a closed answer like "4 hours" won't.