It is common programmer folklore to apply the 80/20 rule to estimate the time required to implement a feature - and I don't mean the Pareto principle, but "Do a worst case estimate, add 80 hours and multiply this sum by 20". :-)
Guess what number they'll accept and double it, put up a fight when they halve it, blame someone else when you miss it, and take the credit when you hit it. At most companies estimates are about control, expectation management, and blame distribution and not about information discovery.
Most people really suck at expectation management, much more than they suck at estimating work.
I make my best guess and quadruple it. That tends to be the most accurate I get.
I underestimate everything and also forget interruptions, new "on fire" issues, life events (turkey day, etc) and everything else.
So instead of worrying about all the stuff I'm missing, I just give my best guess and say it's that. A guess. If you want to hold me to an estimate, I need weeks of planning to get back to you. In that time I can do the missing BA work, mock up what needs to be done and get a better estimate, which I'll still quadruple.
An engineering manager I used to work with would drive me crazy. He had this "methodology" he referred to as a 90% schedule but it was a 90% confidence schedule only in the sense of no one ever getting sick for a couple of days, tests not failing, no hardware being defective, etc. etc. It won't come as a surprise that this schedule was basically never met.
Sounds a lot like the scene in Star Trek: The Next Generation, where Geordi La Forge meets Scotty.
> Lt. Commander Geordi La Forge: Look, Mr. Scott, I'd love to explain everything to you, but the Captain wants this spectrographic analysis done by 1300 hours.
> [La Forge goes back to work; Scotty follows slowly]
> Scotty: Do you mind a little advice? Starfleet captains are like children. They want everything right now and they want it their way. But the secret is to give them only what they need, not what they want.
> Lt. Commander Geordi La Forge: Yeah, well, I told the Captain I'd have this analysis done in an hour.
> Scotty: How long will it really take?
> Lt. Commander Geordi La Forge: An hour!
> Scotty: Oh, you didn't tell him how long it would really take, did ya?
> Lt. Commander Geordi La Forge: Well, of course I did.
> Scotty: Oh, laddie. You've got a lot to learn if you want people to think of you as a miracle worker.
Sometimes there are legitimate reasons to come up with schedules that are as accurate as you can make them, understanding that stuff almost always happens. Sometimes a project or some dependencies don't make sense if they can't be done by a particular date.
But, yeah, it's good advice in general. For most of the work I do (not programming), I have a pretty good sense of the minimum time I need and I almost always significantly pad that to accommodate priority interrupts or just so I can take a bit of extra time if I feel the task warrants it.
I always hated that scene, because it fundamentally changed the character of Mr. Scott from engineer to engineering manager. Which, he was, but that wasn't what made his character so compelling to begin with.
LaForge was a competent engineering manager. The writers tended to focus more on the character's people management skills (and, sometimes, terrible personal relationship stuff). He kept things running smoothly. He was smart. You could imagine a detailed maintenance work log for every system on the ship, and there'd be no gaps in it ever since he took the position of chief engineer.
But while Scotty cared about his staff, it was the machinery that he knew inside and out. He never needed to conjure up a hologram of the designer of the warp engines, because he knew them as well as anyone else. His maintenance logs would have gaps because he'd know which things actually needed regular attention, and which ones were just bureaucratic nonsense. He didn't just understand all of it on the technician level, but on the theoretical level too, enabling him to pull off some unlikely saves.
He actually was a miracle worker and maybe one of the best fictional representations of an engineer, and that one scene was written to make him look a little more like a bureaucrat.
I felt that it was more for comedy as well as helping to develop the difference in character between the two.
LaForge is competent, he's also newer in his overall career. Scotty has been around the block so many times he knows where every crack and seam is.
That was a moment where Scotty's wisdom was being handed down; it's OK to know what it will take, but stuff happens, and there's also the potential to hand tasks off to less senior staff that might not do it at perfect speed. Giving a padded estimate gives you options and room for corrections if there are issues.
Also I don't think it makes him seem like a bureaucrat. I'm no manager. I'm an engineer at the bottom of the org chart, but I still need to deliver estimates to my supervisor. And even if I'm an exceptional engineer, I can choose to estimate the time required for a task as though I were merely average.
PM always wants ROMs (Rough Order of Magnitude estimates), then they bitch when actual project cost is under 5%, or head will roll if it's 1 penny over. Not understanding an order of magnitude, much less a rough one, is why they are not in engineering. They don't like it when I say a ROM is roughly accurate to 0.1X to 10X cost.
I once read someone give similar advice about estimation. What stuck with me most is that if someone pressures you to change an estimate, it's no longer an estimate. Most managers don't actually want estimates.
Many moons ago when I was doing a combination of engineering and project management, I had to come up with a schedule for a shipyard project. Which I dutifully did. It was probably two months or something like that.
Ran it up the flagpole. It got cut in half. Fortunately I didn't need to change the schedule I had drawn up. I just said each box is now half a day. (This was before we used computers for this sort of thing.) As I recall, the job came in very close to my initial estimate.
It's only about control in the sense that it matters what a business invests in. If I believe that feature X will be worth $Y, whether I invest in that, or some other feature, depends a whole lot on how much I expect to pay to get X.
Opportunity cost is the important metric here, not just absolute RoI.
Instead, try actually estimating the work. List components touched. List technologies involved. List screens needed. All of these are measurable. A point estimate is too easy to get wrong and impossible to question.
Not sure what you mean by point estimates, because point estimates are the only way to accurately gauge the work in a project.
Elaborate work estimations are a fools errand. No one can argue that you can make far more accurate estimates by spending 50% of your time doing research for estimates instead of 2% of your time. But spending 2% of your time on estimates also means you'll finish every project twice as fast.
Point estimate, in this case, is reducing down to a single number. If I'm using the wrong term, please correct me.
Now many people doing this use a "point system" where they escalate the allowed votes as you go up. This is a neat heuristic that accounts for some uncertainty. But it really only boils down the estimates to the easy to estimate tasks. Which are often not worth estimating.
I'm not claiming to be elaborate. But do realize if someone estimates a house project is large, I will have less faith in their estimate compared to someone estimating it will take about 300 square feet of tile, plus for gallons of paint and probably to gallons of mortar.
One shows they have thought and didn't gut shot it. Even better, we have something to actually burn down in the supplies. We can also gauge it with previous jobs to know how long it took to use that much supplies.
Software is tougher, because we don't have the same supplies. But this is why you don't list lines of code, but required screens, technical integrations, and general features. Agreed that you don't want to get too elaborate. But also don't give me some stupid t-shirt sizing or other nonsense.
Not the least of the reasons why, is negotiating a size is dumb. There are literally no reasons not to convince someone they estimated high. However, cutting an integration is a choice that has obvious downsides to account for any speed up in delivery. Even better, it is empowering to choose what you won't deliver, instead of having it chosen for you.
Ok, then we agree. If you are forced to give an estimate, give a range, and make it a wide range.
When I was referring to "points", I was talking about Agile development. It's planning process is far more accurate than time estimates because it uses points as a measure of the relative work in tasks. It's far easier to accurately say this task is a 3, and this other task is a 5, than to estimate their hours. In Agile planners don't need time estimates because they prioritize tasks based on relative work required (and customer benefit), and the team just works down the priorities until end of it's sprint, and then ships whatever got done.
We marginally agree. I would argue that breaking up between 3, 5, 8, etc. point tasks is a bit of a charade. It isn't that you can compare many tasks of slightly different sizes that is important. Instead, it is knowing all of the tasks necessary.
In particular, those point estimates are not negotiable. Which is why they are less meaningful. Talking someone down from a 5 to a 3 just convinced them to agree they can do a task faster. Presumably at higher risk. Convincing them they don't have to do two of the tasks? That is a clear win, because it is work they don't have to do. Not work they have to try and do faster.
There is also the generative problem of software. As time on a project goes on, changes become slower. So, some task may be a 5 if done now, but a 12 if done later. Similarly, some tasks can piggy back of effort on others, such that they get cheaper in terms of work needed. Though, with time, they probably keep the high time cost.
Planning poker is not done by negotiating. It's done by doing point estimates simultaneously by the entire engineering team. Providing simultaneous estimates reduces cognitive bias caused by having the lead engineer's estimates influence junior engineers, etc. If they all agree on the point value, no problem. If there is a big difference, then it should open a fruitful discussion so the team can reach consensus. The key is that points don't mean hours, or days or any time measure, it's all a relative measure of the work involved. As long as it is, consensus is usually easy.
All tasks should be pointed in Agile. If anything was left out it's on the team to make sure a story is written for it. For example lets say my PM wants us to estimate 10 stories for the next sprint. If I know I have to do some low level work to update the database to make those 10 stories feasible I'll say we need a story for that, and it needs to be our highest priority since all the other stories are dependent upon it.
But good Agile is done with a Kanban approach, we have a ranked board of tasks, and work on the highest priorities first. If we forgot a task, we add it to the board and point it, and our PMM/PM prioritize it. This can be done at any time in the schedule. When we reach the end of the sprint, we make sure all the completed tasks are done, bug free and we ship.
Then we start a new sprint, working on the remaining tasks, pointing new ones, and working from the top of the Kanban board again.
No one ever need to make a single time estimate, or waste more than a tiny fraction of development time doing any type of estimation.
My main concern here is precisely that it is not negotiated. I like the idea of making sure folks are on the same page. However, the idea of another meeting to discuss what we should instead be building is nauseating.
Instead, meetings that decide what we are not going to build are much more productive feeling. Which is why they should be negotiations. If there is just a list of tasks that has to be done, just keep the list up to date and let the team work on them. If there is concern on priority, make a choice and let the team weigh in. Don't add to their work by asking them to prioritize on behalf of stakeholders.
And kanban is good, but really needs handoff spots between teams. Otherwise, you are just in a constant swarm. Often rewarding the fastest workers. Which is fine, but risks alienating the slower ones that could contribute in other circumstances.
We seem to argue a lot for people who agree so much.
My viewpoint is let marketing estimate the value of features, my job is to give them a rough estimate of how hard each one is to build, and they can decide priorities based on those two things, and tell me what to work on first. If I go too fast and lesser developers feel left out, I'm happy to pair program and do code reviews with them to help them improve.
Yea, all my projects have to have a defined release process (code freeze, final bug fixing/deferall, final acceptance tests), you can't kanban your way through that, but you can kanban your way to it.
I wish more arguments were like this. :) And I have found violent agreement to be much more of a discussion than some disagreements. At least online. (Which yeah, sorta sucks.)
And I should be very clear, if what you are doing is working for you, that is by far the most important thing. I am decidedly not trying to convince you to stop and change.
Further, If it can be formulated and shared with others, I'd be interested in the results. However, I have come to find that I do not expect things to generalize between people nearly as often as my instincts would want them to.
I do take this as a challenge to how I've viewed stuff. Currently, I'm on leave for about a month, but when I get back I plan on paying more attention to the process. I'm hoping I don't miss any retrospectives on projects that were finishing up as I took leave. I'm not convinced people typically zero in on the important points. I am still fully convinced you should always try.
Execs view it as being able to make informed decisions about where to spend the company's time and money.
Let's say you have projects A and B you want to do, and you've estimated revenues from each at $11 MM and $4 MM. You really want to do A, since it would establish a business relationship with a new strategic partner, but estimates come back that A will take 16 weeks and B will take 4 weeks. You only have time to focus on one project at a time, so you choose B to try to capture a quick turnaround before moving onto A. Plus it would make an existing partner happy.
Execs know from experience that the engineering estimates usually suck. Sometimes they're really high and sometimes they're really low. Execs decide as long as B doesn't exceed 5 weeks, we're OK. They don't communicate this down the chain because if everyone knows 5 weeks is the "real" number they'll (grudgingly) accept, they are pretty sure someone somewhere in the chain will spend a week optimizing the test harness or refactoring the build process or something.
At week 3, engineering says they're gonna need 2 more weeks. Execs: Sure.
At week 4, engineering says they're gonna need 2 more weeks again. Execs are annoyed but now the work is mostly done, and everyone promises on their children's lives it will actually be done in 6 weeks, so they let it finish.
At week 5, we're still on track for week 6. Execs: Sure.
At week 6, we're gonna need another week. Execs: Whatever. At this point they've abandoned all faith in this engineering team. They re-task some other critical part of the business with beginning work on project A. Some part of the business suffers.
B actually takes 7 weeks when all is said and done. Execs wouldn't have bothered with it if they knew that up front. Everyone involved in B takes a reputation hit: the product manager, the project manager, the engineering manager, and the engineers themselves. The people who blame others get an even bigger reputation hit.
Now there's pressure from some sides of the exec table to make up time on project A and get the estimate below 16 weeks from a team that had nothing to do with B.
Happily for everyone but the first team, it turns out the other team is able to get project A out the door in just 12 weeks. Everyone on the first team takes another reputation hit.
You might say that the root problem was the estimate for B was too low. That is a problem, but the fact that the estimate for A was 25% is equally problematic. Had either estimate been more accurate, the company would have focuses on A first.
All of these execs are accountable to the CEO, who's accountable to the board. The CPO or CTO or whoever are not escaping unscathed from this, because at the next board meeting the CEO has to answer for why project A was delayed into Q2.
What is the core argument you're making here? I mean sure, I too can argue any point by making up examples about abstract projects A and B, but I'm not seeing what fundamental dynamic you're trying to illustrate here.
There's no argument; as I said, I'm sharing a perspective. Most engineers, and you can see it in the comments here, approach estimation as an us vs. them battle of wills. The perception battle that they deal with on a regular basis isn't going to improve that way.
We are just finishing a long refactor that strongly improved the product. But the planning process was poorly done. They took a couple of lead engineers, made them draw a bunch of diagrams we didn't need, and made them estimate all the work time, instead of involving the entire team. And all of the commitments in the schedule were not commmunicated to the entire team till the end.
From the exec's standpoint, it looks like we can't be trusted. But the truth is the process can't be trusted. Never give out time estimates. Use Agile and point planning, and relative cost estimates.
In your example, the team should have said that we estimate A is 4x as much work as B. Pick one, we'll get it done as fast as possible.
You can have a fixed time schedule, or a fixed project feature set. You can't have both, without the project being late, and features being compromised.
You identified the first one correctly: they had the lead engineers estimate in isolation and not involve the team in any way until the end. Beyond being a bad way to estimate, it's a bad way to manage a team.
But then you said you should never provide a real estimate. That conclusion doesn't really follow. The leadership team has to come up with real estimates. "A is 4x as much work as B" is good information, but it's pretty useless when it comes to actual calendar planning.
The second issue, and the larger one, is that long, stop-the-world refactors (or rewrites) are the fastest way for the engineering team to lose all credibility in my experience. The business keeps moving even if new feature development doesn't. I've found incremental refactors to be the best way to go. And those are much easier to estimate.
As far as schedule vs. feature set, if you estimate properly and prioritize all of the least-needed features at the end, if you find yourself falling behind you just pop those tasks off. But you build that contingency in and communicate it up front.
You contradict yourself st the end. If the organization can accept lower value features being deferred at the end, you never need a time estimate. Just a finish date.
In our case bogging down the entire team in time estimates would have wasted even more time and we would have accomplished even less. Had we kept to an agile process with sprints we could have met the fixed schedule expected, and delivered more benefits given that losing our leads in planning for weeks on end meant they contributed relatively little to the refactor.
Wow! If your planning is taking weeks, something has gone seriously wrong.
What I said was no contradiction. How does one arrive at the finish date? Finger in the air? There are essential features and nice-to-haves, and to get a delivery date you need to estimate them. But it's an estimate and so you plan for contingencies.
Yes, something went seriously wrong. When it became clear we had no idea how much longer it would take (because the app wasn't in a fully testable state so our bug counts were meaningless), we got asked to spend half a day re-estimating task and bug times. Which was time not spent fixing bugs or making the product testable, or even making the schedule predictable, because fixing blocker bugs for testing was the only way to do it.
At the end they required twice a day reporting meetings, the standup to say what you were doing, and an end of day status meeting to say what you did so it could be communicated to the exacs. Two more meetings where zero development gets done.
I am not really disagreeing with you. It's reasonable to do basic estimation in most projects. Usually marketing/management need to know when something can ship in order to plan the launch plan. So you figure out how long you need to be really confident the essentials will be finished, and use the nice-to-haves as padding that can be deferred if necessary to make the date.
But if you don't have to hit a specific date, say the product is already shipping and just needs regular constant improvement, just do it in fixed schedule sprints. Focus on the highest priorities every sprint, get as many done as possible during it, ship the new release, rinse and repeat. A well run sprint can be great because eschewing time estimates means more time spent actually building the product.
But if you are forced to give them, never give a single number. Use ranges. Your range for a confident estimate should be roughly 4x between the fastest and longest components. For example, 1 to 4 hours for a task you are very confident about. For things you have less confidence on, your range should be closer to 8X.
Why do this? Because wide ranges are the ONLY accurate software engineering time estimates. You need to communicate the inherent lack of precision in any estimate. For every time estimate you don't know
1) How many hours a day you'll be allowed to actually code, outside of meetings, standup, planning sessions, emergency bug fixes, etc, etc.
2) Which member of your team will actually end up doing it, the fastest one, the slowest one, or someone in between.
3) How many days will be lost to illness or personal issues.
4) How much the actual feature or story will change as it's developed and they realize the design is actual shit.
5) How much other code will need to be changed when you actually rip the lid off some of the older code it will interact with.
A good way to enforce this is to always estimate things in steps of 2^x. Eg you are not allowed to estimate something as taking 20 days, it's either 16 days or 32 days. That way it's obvious that the larger the estimate becomes the more inaccurate it is and you are forced to automatically double to 32 if your "inner" estimate is 17 days.
Scrum has a similar technique only allowing Fibonacci numbers, I think the reasoning behind it is similar.
Obviously the receiver of the estimate should know that you are using this method, otherwise they might start questioning you thinking this estimate should only be 17 for example.
The best approach to this I have found is to ask why.
What decision are you trying to make using this estimate? Understanding context helps you to understand what sort of estimate is needed - or whether an estimate is needed at all.
I think what the GP is getting at is that an estimate in absence of a context is meaningless. The organization may just be trying to get a handle on the rough scope of a job to make strategic decisions. They may also be in the middle of a discussion with a customer about delivering a feature.
The best thing for both the company and yourself is to try to be as high bandwidth as possible and discuss all the potential risks and tradeoffs. Find out what the real goals are so you can optimize a solution to provide for them. Many managers and customers get stuck early on with a particular approach, but if you apply some creative engineering you can come up with a solution which is easier, or folds in with some other goal you were already working on, leverages existing functionality etc.
its on you to express it in terms they can understand from their own position: risk, cost, headcount, schedule, dropping existing features.
if your organization doesn't want to engage in that kind of semantically meaningful discussion, or comes back and says 'I know you said 3 months, but get in don't in 2 weeks' then you need to start managing upward and looking for a new job. or you can just overestimate, and enjoy your free 2 hour lunches for as long as the money lasts.
I don't do estimates any more but I use forecasting. Like the weather, it's an idea of what might happen given what we know now. I once laughed in a job interview when the interviewer said that they expect 'accurate estimates'.
There is a difference between establishing a deadline and estimating relative effort and complexity. The former is problematic. The latter is important and should be done and then revised continuously.
The process of building software includes discussing what has the best impact to effort ratio. Both are estimates and will not come out as expected.
The logical response to managers that expect you to always hit deadlines is to inflate estimates enough that you will.
1. Why do you need the estimate? Is it to get budget approval, or because you're trying to meet an approved budget?
If they don't have budget approval then I'm going to charge them something to gather requirements and deliver an estimate.
No budget approval = low on priority list.
2. We charge a blended rate of $xxx per hour. If you have a detailed requirements list I can tell you if we can do this within your budget.
If they don't have a detailed requirements list then I'm going to charge them to provide one.
After I have the requirements list completed I'm asking for their budget.
3. Don't want to pay for a requirements list? I understand. Its okay. Something like this typically costs (low bid), but that will likely change if the requirements are different then my assumptions.
Could you elaborate on #3? Do you walk away at that point? A last attempt to show the value of good requirements and get them back to #2, else they are premtively "fired" as a client?
In enterprise software, at least the companies I know, they will underbid and return to the well many times for increases. They will blame this on poorly scoped requirements, for which they were not responsible. They will be right, and they count on this for much profit. If you quote virtuously, you will be a victim of your client and competitors who scope/quote less virtiously.
At our company we didn’t estimate entire projects, but separated tasks until it was 3-5 hours piece of work that everyone agreed “it’s realistic, I can do it”. The big time then simply summed up with 30% buffer for delays. Since our customers mostly paid for hours, it was also a good base price builder.
Big projects can suffer from this because little architecture is involved (and we worked on already existing systems). Otoh, I’ve seen no big local project that wasn’t an architectural mess once it hit a deadline hard. It is better to not have one than to have a set of rules that you must obey, while other parts of system don’t even care of or contradict these.
Personally I don’t believe this “I’ll get back to you with an estimate” thing. It is just first iteration where you go relax and try to formulate at least 5-10 questions that you think you should ask foremost to start building an estimate. If you come back with an answer instead, then it is still wrong, maybe even worse than random, since you estimated a different thing. For us, estimates and requirements always moved, as they were functions of delays and business priorities that change unpredictably. I didn’t like to deliver the entirety on a deadline because my job was to support that for years, not throw a bill at them and run, as all integrators did.
Maybe, maybe these SE-users are speaking about coding phase, when all “business analysis” was done perfectly. But ime it is you coder who collects all the knowledge, because no one can. Or it is you who is non-coding analyst who must estimate, because non-analytic coders don’t have a clue what a big picture is.
engineers hate giving estimates for a variety of reasons that i'm not going to get into here. most modern agile organizations don't value spending time on getting accurate estimates, but some kind of best-faith estimate is still valuable for a lot of reasons.
for medium to large size projects i like to give a confidence score to my estimate to give a sense of how much unknown there currently is.
for example: we think we can get this ticket done in 5 days, but there's a 50% chance it will take two weeks (because of complications in X, Y, and Z).
or the other way, my estimate is 3 days, but if this one thing works out that we're looking into, there's a chance we can finish it in 1 day.
what people are generally asking for with estimates is sizing and complexity, so providing some additional information helps.
I like to do something similar - I break down what I see are the risks, add a risk factor accordingly, then provide the estimate as a range. For example, the risk factor my be 50%, so I'll say the cost will be between £100k and £150k (for example). I find this really helps to focus the customer stakeholders who can control some of those risks.
Some devs seem to have actual phobias of estimates. My team was asked if we can deliver features x or x+ some time next year. (IMO as the dev who knows both the code and x best, we can deliver x in 2018 easily and x+ maybe). In our discussions about it, one guy would only say you can't predict the future, and another guy argued we should say either of them would take two years so that we could have time to rebuild the app from scratch while we delivered them. It's a web app.
tl;dr - we engineers are bad at estimation because we're too optimistic, and then we don't want to be accountable to the estimate given based on limited information.
i think we have phobias for estimates because of management repercussions that only happen when you miss an estimated delivery date, or for organizations that put external dates based on a best-case aggressive engineering estimate. then when the engineering team misses the date, there's a retro (when retros should happen for both good and bad sprints.)
i've never been clear why more organizations don't soft launch features or products, and then pick a marketing/external announcement date.
Engineers hate giving estimates because they can never be as accurate as planners want.
The only accurate time estimate is a range with at minimum a 4x difference between best and worst case, and preferably more the less certain the estimate is.
One thing I've been unable to ever see adopted is a true bottom up planning strategy, where the time to delivery is estimated from historical data. Say last X project took Y time, similar Z project will thus take Y time. And from there, the business plans timeline and delivery dates.
The reason for this is that in practice I've found estimates are an indirect way to define scope.
You need to ballpark the minimal viable product in a range from under a month, a month to 3 months, 3 to 6, 6 to 1 year, over a year.
After that, its just about downscoping all the "good to haves" based on if there's time left from the desired deadline or not. As long as you delivered the "must have" in your timeframe it should be good.
Pro tip: Don't laugh. The boss once asked me for an estimate and I laughed. He said that wasn't funny and I said that he started it!
On the other hand: Do it. Laugh. Laugh in his face when you are called to help on a project that isn't ready after 1 year and started without you. When they show some wire-frames and ask you how long it takes without telling you about the actual technology used for the project.
Summer 2015 I was uninvited from a project meeting. My coworker was planning the project and discussed it a bit with me. He wasn't the one who decided that I shouldn't take part.
Summer/Fall 2016 and the project isn't going anywhere. I'm invited to help. I do my best, besides the other projects.
January 2017 and my coworker is gone. He gave his 3 months (+ x days) notice and told me December 2016. Shortly after it I gave my 4 1/2 months notice.
The new deadline for the project was May 2017.
They launched November 2017.
"Cool story, bro. What was it? Some revolutionary VR or AR application? AI supported super gadget? Brainwave reader and interpreter?"
Any project like this one can often go off the rails if the right decision makers are not involved. This is not necessarily the fault of the project leader as it can be unintuitive in bureaucracies.
May of 2015: hired at startup that wants to use Natural Language Processing so salespeople can send text messages directly to their CRM (Salesforce, PipeDrive, etc). Very excited about the project. The app's initial UI design is a traditional one, with NLP helping to smooth the basic task. The full app would take about a year to build but we all agreed we could have an MVP by August.
August of 2015: without consulting anyone on the tech team, the Board Of Directors decides the app should get rid of all standard UI elements: no forms, no buttons, no links, no drop down menus. Instead, the interface should be a pure chat app. This makes the project much more ambitious, which I was very excited about, but which I felt would delay the project 2 or 3 months. Nobody was happy with my estimate.
September of 2015: the CEO was able to show a demo to the Board Of Directors. The demo was an illusion, as we had no error handling, and it only worked for some carefully planned examples, but I thought it was a good sign that we could show the Board that we were making progress. Unknown to me, the Board then asked the CEO for real feedback from real customers at the next meeting, a month later, in October. There was no possible way for us to finish the product, and find customers, and get feedback, all in a month, but the CEO was a bit of a coward, so he promised the Board we would do this.
October of 2015: obviously we did not have customer feedback by the next meeting of the Board. At this point the Board decides we have missed our schedule and they begin to panic. We are slowed by the fact that our "NLP expert" is inexperienced. Myself and our iPhone programmer tell management that we need to fire the current NLP guy and hire someone with real experience. Management initially agrees but then later changes its mind, for reasons unstated.
November of 2015: we get a basic demo working, and it has enough error checking that we can show it to potential customers, and not be entirely embarrassed. But the stress of the previous month has wrecked relationships inside the company, people are shouting at each other constantly. At this point I step away from the project, but I remain on friendly terms with the guy who is doing the iPhone programming.
January of 2016: company gets two trial customers, but these customers won't pay for the product. The NLP engine is improving, but at glacial speed, as the "NLP expert" is at the beginning of his career. He was a nice person, and I can believe he will eventually become a good programmer, but it didn't make sense that a company that wanted to move fast also remained so loyal to a guy who could not do what was needed. An established company could/should afford to have an apprenticeship program, but a fast moving startup can not offer an apprenticeship to someone working on a core technology.
April of 2016: the iPhone guy again asks management to find a new NLP expert, someone who can move the company forward. Management reacts by listing every bug ever discovered in the iPhone app, as a way of telling the iPhone guy to shut up.
May of 2016: the iPhone programmer quits and gets a job elsewhere.
June of 2016: the company is almost out of money, so it cuts back on spending, reduces the team.
summer and autumn of 2016: without much staff, the company crawls along at a glacial pace
Early 2017: the company gets more funding, begins moving forward again.
Summer of 2017: the company now has a few trial customers, but they are paying trial rates, which is to say, almost nothing. The company is not anywhere close to the breakeven point.
The company continues to burn money without making much progress. There are clearly some fundamental leadership issues that should be addressed, and one of those are the ways that estimates and budgets are created. Also, the company would be in much better shape if the leadership listened to feedback from the tech team.
My PM has a great way for dealing with engineer estimates: increment both the number and the unit.
1 hour becomes 2 days. 4 days becomes 5 weeks. And so forth.
I'm actually somewhat decent at estimating well defined work, so it actually annoys him when one day is one day; but I appreciate his work when some odd stumbling block (usually political; my estimating skills crash and burn when it involves working with humans) comes up and the time doubles or triples the estimate.
A software project that takes a year will costs a fortune (salaries are expensive). It will probably have to be maintained and running for the next decades to give enough returns to be worthwhile.
So, 1 year is the time the first interns team will work on it, 10 year is the duration before the project is retired. Looks like both estimates were right.
The joke is for hours/days/weeks estimates, not months, years or decades. Hopefully nobody is asking an individual contributor for estimates they expect to be in years.
My engineering manager at a games company I worked for told me that "three weeks" was the estimate that really meant they had no idea how long it would take, because the task was too hard to estimate. Anything longer than three weeks could, and did in fact, sometimes take six months.
That sounds right because it's clear that the 1-year estimate lacks understanding and forethought. If you start a project with that large of a swagged estimate, it mostly won't ever be completed. The only way I'd even consider an estimate that large to have any basis in reality is if it was 1 year, 2 months, 6 days and 4 hours. That would mean the estimate is composed of many smaller estimates for tasks that are much closer to being understood.
Large projects can always be broken down into smaller, more well-understood tasks. You can't build something large all at once. Large estimates are just an indication of a lack of discipline in planning because they mean that the project hasn't been decomposed into its constituent tasks. Large estimates are lazy and can only be right out of sheer luck. In the team's I manage, I don't let anyone start a story with an estimate of more than two days.
Are you being serious? That only works if the people consuming these estimates don't have a clue and therefore can't call BS. This would get me aughed out of my job.
This also astounds me. I can't graps the art of these estimates. I suck at estimating deliviery times, and I'm a pretty good engineer. But I either over-estimate or under-estimate, I find it hard to be on point, there are always unexpected side effects. And I'm known as a sort of miracle worker, when they have something they can't solve, they come to me. But I HATE giving estimates.
If I were to estimate things as suggested in this thread, it seems to me things would never get done. Ever. It simply boggles my mind people want and will fudge estimates on this scale.
Pretend, for a moment, that you're working on a one week project. You're estimating that week based on the ability to use an open source project which promises to take care of a large portion of the complexity for you. Also pretend you're new enough to not realize the hollowness of that promise.
It's only three days in to the project that you realize there's a mismatch between what you need and what the project provides. You start a conversation with the project's lead developer; but they're only able to have half of any given conversation in a day. How hard would it be to spend five days researching the cause and severity of the mismatch while discussing solutions with the remote developer?
There's 8 days.
You decide to write a patch for the open source project shepherd it through the internal and external approval process, burning up another two weeks of time.
18 days.
You finally get back and finish the original project, redoing a fair bit of work to account for the changes your patch introduced.
23 days.
You're interrupted throughout this process several times by fires in production that need to be fixed RIGHT NOW.
28 days.
Let's make the assumption your PM did their little trick, and slated 2 months for this project. Despite your own estimate being off (through no ill-will by anybody) by about 6x, your work is still completed before it's expected, and the teams depending upon your work are happy as clams. Even if it had finished in the original 5 days, everyone would be thrilled, and you'd get more work to do.
From the PM's point of view, it's a win-win situation. From your point of view, the worst you have to endure is a twinge of "don't you trust me?", offset by the relief of not having to answer "why isn't it done yet" every day from day 6 on.
Ideally, from my point of view, estimations should be not over nor under-estimated. They should be on point. Sure, on first glance, under-estimation leads to worse outcomes than oversetimation. But it will still lead to negative outcomes. Rational actors on the wrole chain will benefit and extract the most utility when they are as correct as possible.
While your example showed how project implementation can creep up, I already knew that and my problem is more my inability to give accurate estimates and people's willingness to stretch their estimates by a lot. I KNOW some of you will not agree with me, but I think that giving 4x as long estimates will lead to developers spending more time on the actual implementation without noticeable improvements in the delivery. I know my natural tendency is to do so.
A PM is responsible for coordinating disparate groups of people. So if one person says “A will take 3 days” and someone else says “I can do B in 3 days, which I can’t start until A is done, and C in 2 days” what’s the correct time estimate for all the work? The first person will probably have issues they didn’t anticipate, so now their work is really 1 week long. The second will start on C, hopefully, then either spend a bit of time following up to see if they can start on B, or miss the fact that A is finished by a few days. So while people may be only a few days off per individual task, for a PM this has a multiplicative effect. So the PM can either act like an engineer and say “well there were X Y and Z unexpected things which is why this is going to take 3x as long” or just assume 4 “days of work” across 4-5 people really means 5 weeks before all the work is done (because not everyone can devote 100% to that work, some dependencies are difficult to hand off and can’t be done in parallel, etc).
Bill your clients for estimates and still give ranges. The more they want to dial in those ranges, the more billable time you spend estimating. "How long will it take to estimate" is a question you should be able to answer because you will control the deliverable in this case. Also, if you spend time with them asking questions and helping them discover the complexity of their request, they will soon see the benefit of this process.
Alternatively what I'll do is point out that their request is simply incomplete and I can help them create a better request, which I will then estimate (it's all still billable though).
It's critical to position yourself (or your company) to be able to bill for discovery (a separate project, or at least a day-rate engagement). You're a consultant working with the client to figure out what the project actually is. Amazingly, the client often doesn't really know, as can become apparent during that first gathering around a whiteboard. And if they don't know, you're well on your way to a "now that I see what you've done, it's not really what we had in mind" moment.
Also worthwhile is to consider the overall likelihood of success for projects you take on (and what it implies for future work, quality of life while working with the client, etc). I wasted a lot of time early in my career on projects that I knew were doomed just because someone agreed to pay my rate.
I give an estimate with an error margin, that is usually larger. eg something like "that will take about 3 weeks, plus or minus 2 months". The plus or minus bit reflects the uncertainty in requirements, things that could change etc.
Often it is the case the goal can already be achieved with existing functionality, but may not be as direct or elegant. That is why I include the "minus" bit covering this scenario where a solution is already there.
Yes, that was precisely the point with "Often it is the case the goal can already be achieved with existing functionality, but may not be as direct or elegant".
I am very serious. My clients respond very well because it helps them plan better. The problem with a traditional one number estimate is you don't see the error bars so they don't know how to treat "3 weeks".
My approach also helps start a discussion about how to reduce the size of the plus/minus bit, such as reducing scope, breaking into multiple pieces etc.
And the minus bit coming to negative time works well too. It becomes a decision about using what is already there even if not perfectly suited versus doing the new work, and how much of it to do.
Explain that the way to do estimates is to have a process that starts by breaking down the work into the components of the design, estimating each component, and adding it up, with small padding to each granular component. The more granular the breakdown, the more accurate the estimate. For a single top-line complete project guess (which is what you are being asked) you should use a large pad, as comments below say, double and shift. If you can afford the time to do the breakdown, you will get a closer and more realistic number, and have some of the project planning work done to boot.
At my company we use simple analysis on previous estimates and adjust future estimates based on past errors. We are usually about 70 to 100 percent accurate using this method. However, projects are usually not more than 7 days of work in my case.
This is something I wish I saw more talked about: how to get better at estimates. I wish more people dedicated time to reviewing their projection & mindset at the time, then look at the task timeline, then write down what caused it to go off schedule.
Sometimes it's an environment change. Are these consistent? Can they be modeled statistically?
Often, it's an inaccurate belief in the work to be done. Was it lack of requirements, that had to be expanded? Etc.
When I work in development, I spent up to 20% of my time on "overhead" tasks. These are of the estimate review type, not meetings. It's something a great manager will do without wasting his team's time, but as an independent, I spend a lot of time doing it myself, and it's amazing how much it helps to dedicate time and write things down.
A similar trick to help crystallize thinking about the project is to ask whether it's on the order of hours, days, weeks, or months. This mitigates the fear of speaking a number and having it be enshrined as a guarantee, while connecting the rumination process to the experience of past projects in all their underestimated splendor.
I like to say, "my gut says X days. But I'll have to get back to you." I think there's a lot value in having the conversation with the PM or manager or whomever in the moment. And being non-committal is frustrating for the other parties.
Not in my experience. I think it's about managing expectations. You just have to be clear about what you mean with that individual. If that PM takes it to mean that then you need to find another way to communicate your message.
I think what I intend to say with that can be, "I don't know how long it takes and it'll take more planning to figure it out, but I'm willing to participate in this conversation so that we can continue to effectively plan in the current context."
If the current context is "concrete" then you probably don't want to give a number. If the conversation is (and this was my assumption before) "here's a feature we want done but we're trying to figure out it's priority and feasibility" then giving a gut check adds value. You just need to be clear in your messaging.
And if someone uses a gut check number against you "but you said X days!" Then you need to call them on it and make sure you explained the discrepancy.
Just keep in mind that estimation != promise. Make estimations to help you plan your time, but never commit these to management who turns them into promises.
The problem with estimates is mostly requirement analysis.
When people say they need feature A, B and C and ask when I will have it done, I split it up in tasks that take below a day, estimate these tasks and add 50% for safety.
This works in most cases.
But often the reason why projects won't finish on time is, because people don't know what they want.
They ask for A, B, C, you estimate it right or probably too pessimistic and get done ahead of time, but still the project is late in the end because they forgot about D, E, F, G, H and J. In the end you can fit D, E and maybe F in the time you have left because you got A, B, C done quicker than estimated but G, H and J are still missing.
I'm not in the software industry so this sounds really odd to me. Don't you guys write proposals to go along with the quote that detail exactly what is in the scope of work? Do you not have a contractual statement of work?
If they come back and say, "oh, we forgot to say we also need D", then the response should be, "that's no problem, we estimate it will take an additional X hours or dollars to implement that. Would you like us to update the proposal?" Or, if you're under budget and you're charging hourly rates instead of a fixed cost, you might even say you can probably add it without going over budget.
PS: I just realized that you're probably talking about internal "clients" so there's no contract or even payment. But I think the same general idea applies. When they bring up extra features, you just say that it's going to take additional time.
edit 2: I guess this doesn't actually help with getting the project done on time. The initial time estimate is still going to be off for the reasons you mentioned. So this comment isn't really an argument against anything you wrote.
That's funny. No, for most developers who are asked to provide an estimate, the analogy is more like this:
PM: I want a bridge. When can we start using it?
Eng: Uh, ok, where is the bridge? What's is going over? What kind of traffic will it carry?
PM: I want a bridge. Why are you being difficult? Tell me right now when we can start using it! I've got to go promise my boss that you committed to getting it done by then! Oh, btw, it needs to be done next week.
More like they ask for a car ferry to cross the river but really want a bridge.
If you are lucky, you will recognize from their "requirements" (often just a verbal description) that they are describing the characteristics of a bridge (e.g. continuous bidirectional flow), not a ferry.
More typically, after you have built the car ferry, they ask why it doesn't implement continuous bidirectional flow despite never specifying that as a requirement.
I think that last description is more on point. People can usually describe why they want something, if asked, instead what they want. It's not too difficult to come up with a reasonable scope if either the dev or the manager knows how to start and scope a project.
Usually they both suck. The manager will send an unclear email. The developer will start developing right away. None have any idea what they are supposed to do, why or when.
This is why it's key to ask clients what problem they're trying to solve, rather than just delivering what they say they want. The classic, "If I'd asked my customers what they wanted, they'd have said 'a faster horse'." Especially don't let non-technical people constrain the range of possible solutions without very good reasons.
Outsourcing firms will do strict contracts as you said, to cover their asses. The history has proven that they are not enforceable and can usually backfire for the client.
A client wants something. In software, the something is usually complex and not well understood, a contract is done to do a something like that. During the project, they will invariably find new information and issues, the scope will change, sometimes dramatically.
For the comparison, consider a building. You want to construct a new building, it's easy, you say what land you have, the engineers will tell you how large and tall they can build on that, you get to pick the interiors among a few office/industrial layouts. You can all understand what is being discussed.
In software project, there is no physical limitations and no regulations. The limits are not clearly defined, noone will see or believe them. Worse, you have to integrate with clients/new/legacy systems that are not well defined and unknown ahead of time, nothing you can plan about that.
This is the most accurate description of reality I've seen in this thread. Most of my work is "just" discovery of what can be built and what should be built while building the thing. And that's OK since one of the greatest strengths of software is extreme flexibility.
> "that's no problem, we estimate it will take an additional X hours or dollars to implement that. Would you like us to update the proposal?
What happens in practice is they didn't realize they were asking for additional features -- they thought they were asking for D, E, F (etc) all along in the initial request. They just assumed "Email signup link" also meant spam handling, re-sending, prevention of double sending, etc. Good product managers account for these obvious things, and leave padding for the unexpected. But good product managers are rare. Most software developers I know have never even met one. So instead you have a political dance where you try to plan for the things they aren't telling you, along with all the _expected unexpected's_ that come up in normal software development, and set your deadline's appropriately. Definitely the worst part of software development.
I think there are a handful of problems unique to the software industry that make accurate estimation practically impossible. Mostly they're a sign of an immature industry. So, as a lot of comments here suggest, most people just give it their best guess, and then inflate the number a lot, and negotiate some kind of deal, and then work like hell to actually pull it off.
1. It's never repeatable. Nearly every single thing built in software is being built for the first time. Even if it's some government CRUD application, and even if you're leveraging sixteen popular software frameworks to build it, that application will still be in some way different from other applications in the same niche. There's no blueprint for building software; mechanics have a gigantic (sometimes accurate) database of the time required to do a variety of jobs, tract home builders are all building the same thing over and over ... the closest that software ever gets to that is themes or skins for frontend UI, and even those inevitably end up getting customized in some way.
2. Nobody wants to pay to build a blueprint. The only way in the software industry to figure out how complex a particular application is going to be is to build it. Developing an accurate spec is exactly as expensive as building the actual application. This is because the "engineering" part of software engineering is still practically nonexistent.
3. And while we're on the topic of engineering: there are no standards. Or, there are thousands of different standards, depending on how you look at it. That CRUD application could be built with one of many different interfaces -- should it emulate data entry terminals, with simple tab-navigable text fields? Should it use Google's "Material" UI? Bootstrap? There's no comprehensive study which has compared the amount of end user effort required to use each interface, and therefore there's no industry standard which says, "this is always the right way to build this thing given these constraints". Compare to something like a low voltage contractor's license, which expects you to be familiar with the different types of cable to use in different building environments.
4. It's buggy as hell. Ok, so you're experienced enough to navigate your way through everything else up above and not get burned too badly. You have a preferred toolkit, you specialize in a particular niche, cool. Well, today, Chrome just released an update that broke the way your application handles cached content. (Happened to me.) The vendor for one of your frameworks released an urgent security update that completely broke critical functionality in an unrelated part of the application. (Happened to me.) The two libraries you're using together, which don't use any of the same globals and should be entirely isolated from eachother, are still somehow interfering in a way that's not covered by the sparse documentation. (Happened to me.) There are millions upon millions of lines of code involved in every single interaction with a computer now, and they are imperfect. Many of them are written by amateurs, people who -- even though they're getting paid to write code -- have only been in the industry for a couple of years and they're still re-learning all of the same lessons that everyone else before them had to learn through trial and error.
5. Too many programmers like shiny new things. We could fix a lot of the above if we slowed down and "refactored" our industry -- started doing real usability studies, establishing standards for storing sensitive information, more rigorously separating security and feature updates. But, nobody really wants to do that. They want the new thing: the new programming language, the new framework, the new browser feature, the new toolchain. I think this is largely because while programmers spend a lot of their time talking to machines, they aren't machines themselves. They need a steady stream of new stuf...
> Many of them are written by amateurs, people who -- even though they're getting paid to write code -- have only been in the industry for a couple of years
Yeah, it's an interesting point, because it's not even "many", it's "most". There's a talk by Bob Martin where he goes into this. His statistic was that the number of working programmers is doubling every five years, which means that at any given time, around half the population has less than five years' experience.
If every software project was fully specced, we'd do nothing but spend all our time writing specs.
The best (and fastest) way to build something is with minimal specs, and modifying as you go along. As your clients (internal or otherwise) actually get to use features as they are built, they'll provide tons of input that they wouldn't ever have put in a spec.
This is also why you should never, ever, do fixed bid work in software development.
As a contractor, I'm always asked for an estimate. The client is worried not about the time, but the thought I'll milk the clock, so an estimate ends up being a flat rate.
The best way is to overshoot the time and offer the flat rate. If they bite, I can treble my usual rate, if they come back with a hard negation, I take that as an indicator that they will be difficult to work with.
Don't forget the budget. All clients wants a number so they can evaluate whether they actually have the money to pay for it and whether the project is worthwhile. Noone wants to engage in a 6 months contract thinking it only takes 1.
By the way, watch out when changing from daily rate to project. Legally, you have to deliver the project when you bill for a project, so you are forced to make a contract with well defined project scope to cover your ass. It's quite a lot of (billable) hassle.
Interesting you bring that up though. I had a client through a contracting company. After the project was done, they said they actually asked for something more, which was never even discussed. I ended up partly unpaid even though I was clearly correct.
The law is never in your favor as a single man or small shop. It's too expensive to go to court, then determining what should have been done or not done as per the contract is a lengthy and uncertain proceeding.
Clients who wanna pay per project basically want to move the risk over to you. They should pay a premium. Write a solid contract and increase the fees/duration. Most should give up when you start increasing the quote just to establish the contract, it's easier to start a POC and see what could be achieved.
I try to consistently use the word "target" to head off the ambiguous semantics around time estimates versus "deadlines". It's a little thing, but it can be psychologically easier if one explicitly controls the project language to allow "we need to revise our target" than to be implicitly seen as "not meeting a deadline".
174 comments
[ 3.5 ms ] story [ 232 ms ] threadMost people really suck at expectation management, much more than they suck at estimating work.
I underestimate everything and also forget interruptions, new "on fire" issues, life events (turkey day, etc) and everything else.
So instead of worrying about all the stuff I'm missing, I just give my best guess and say it's that. A guess. If you want to hold me to an estimate, I need weeks of planning to get back to you. In that time I can do the missing BA work, mock up what needs to be done and get a better estimate, which I'll still quadruple.
> Lt. Commander Geordi La Forge: Look, Mr. Scott, I'd love to explain everything to you, but the Captain wants this spectrographic analysis done by 1300 hours.
> [La Forge goes back to work; Scotty follows slowly]
> Scotty: Do you mind a little advice? Starfleet captains are like children. They want everything right now and they want it their way. But the secret is to give them only what they need, not what they want.
> Lt. Commander Geordi La Forge: Yeah, well, I told the Captain I'd have this analysis done in an hour.
> Scotty: How long will it really take?
> Lt. Commander Geordi La Forge: An hour!
> Scotty: Oh, you didn't tell him how long it would really take, did ya?
> Lt. Commander Geordi La Forge: Well, of course I did.
> Scotty: Oh, laddie. You've got a lot to learn if you want people to think of you as a miracle worker.
Sometimes there are legitimate reasons to come up with schedules that are as accurate as you can make them, understanding that stuff almost always happens. Sometimes a project or some dependencies don't make sense if they can't be done by a particular date.
But, yeah, it's good advice in general. For most of the work I do (not programming), I have a pretty good sense of the minimum time I need and I almost always significantly pad that to accommodate priority interrupts or just so I can take a bit of extra time if I feel the task warrants it.
LaForge was a competent engineering manager. The writers tended to focus more on the character's people management skills (and, sometimes, terrible personal relationship stuff). He kept things running smoothly. He was smart. You could imagine a detailed maintenance work log for every system on the ship, and there'd be no gaps in it ever since he took the position of chief engineer.
But while Scotty cared about his staff, it was the machinery that he knew inside and out. He never needed to conjure up a hologram of the designer of the warp engines, because he knew them as well as anyone else. His maintenance logs would have gaps because he'd know which things actually needed regular attention, and which ones were just bureaucratic nonsense. He didn't just understand all of it on the technician level, but on the theoretical level too, enabling him to pull off some unlikely saves.
He actually was a miracle worker and maybe one of the best fictional representations of an engineer, and that one scene was written to make him look a little more like a bureaucrat.
LaForge is competent, he's also newer in his overall career. Scotty has been around the block so many times he knows where every crack and seam is.
That was a moment where Scotty's wisdom was being handed down; it's OK to know what it will take, but stuff happens, and there's also the potential to hand tasks off to less senior staff that might not do it at perfect speed. Giving a padded estimate gives you options and room for corrections if there are issues.
(https://en.wikiquote.org/wiki/Star_Trek_III:_The_Search_for_...) see the first bit of dialogue in this page.
Also I don't think it makes him seem like a bureaucrat. I'm no manager. I'm an engineer at the bottom of the org chart, but I still need to deliver estimates to my supervisor. And even if I'm an exceptional engineer, I can choose to estimate the time required for a task as though I were merely average.
Ran it up the flagpole. It got cut in half. Fortunately I didn't need to change the schedule I had drawn up. I just said each box is now half a day. (This was before we used computers for this sort of thing.) As I recall, the job came in very close to my initial estimate.
Instead, try actually estimating the work. List components touched. List technologies involved. List screens needed. All of these are measurable. A point estimate is too easy to get wrong and impossible to question.
Elaborate work estimations are a fools errand. No one can argue that you can make far more accurate estimates by spending 50% of your time doing research for estimates instead of 2% of your time. But spending 2% of your time on estimates also means you'll finish every project twice as fast.
Now many people doing this use a "point system" where they escalate the allowed votes as you go up. This is a neat heuristic that accounts for some uncertainty. But it really only boils down the estimates to the easy to estimate tasks. Which are often not worth estimating.
I'm not claiming to be elaborate. But do realize if someone estimates a house project is large, I will have less faith in their estimate compared to someone estimating it will take about 300 square feet of tile, plus for gallons of paint and probably to gallons of mortar.
One shows they have thought and didn't gut shot it. Even better, we have something to actually burn down in the supplies. We can also gauge it with previous jobs to know how long it took to use that much supplies.
Software is tougher, because we don't have the same supplies. But this is why you don't list lines of code, but required screens, technical integrations, and general features. Agreed that you don't want to get too elaborate. But also don't give me some stupid t-shirt sizing or other nonsense.
Not the least of the reasons why, is negotiating a size is dumb. There are literally no reasons not to convince someone they estimated high. However, cutting an integration is a choice that has obvious downsides to account for any speed up in delivery. Even better, it is empowering to choose what you won't deliver, instead of having it chosen for you.
When I was referring to "points", I was talking about Agile development. It's planning process is far more accurate than time estimates because it uses points as a measure of the relative work in tasks. It's far easier to accurately say this task is a 3, and this other task is a 5, than to estimate their hours. In Agile planners don't need time estimates because they prioritize tasks based on relative work required (and customer benefit), and the team just works down the priorities until end of it's sprint, and then ships whatever got done.
In particular, those point estimates are not negotiable. Which is why they are less meaningful. Talking someone down from a 5 to a 3 just convinced them to agree they can do a task faster. Presumably at higher risk. Convincing them they don't have to do two of the tasks? That is a clear win, because it is work they don't have to do. Not work they have to try and do faster.
There is also the generative problem of software. As time on a project goes on, changes become slower. So, some task may be a 5 if done now, but a 12 if done later. Similarly, some tasks can piggy back of effort on others, such that they get cheaper in terms of work needed. Though, with time, they probably keep the high time cost.
All tasks should be pointed in Agile. If anything was left out it's on the team to make sure a story is written for it. For example lets say my PM wants us to estimate 10 stories for the next sprint. If I know I have to do some low level work to update the database to make those 10 stories feasible I'll say we need a story for that, and it needs to be our highest priority since all the other stories are dependent upon it.
But good Agile is done with a Kanban approach, we have a ranked board of tasks, and work on the highest priorities first. If we forgot a task, we add it to the board and point it, and our PMM/PM prioritize it. This can be done at any time in the schedule. When we reach the end of the sprint, we make sure all the completed tasks are done, bug free and we ship.
Then we start a new sprint, working on the remaining tasks, pointing new ones, and working from the top of the Kanban board again.
No one ever need to make a single time estimate, or waste more than a tiny fraction of development time doing any type of estimation.
My main concern here is precisely that it is not negotiated. I like the idea of making sure folks are on the same page. However, the idea of another meeting to discuss what we should instead be building is nauseating.
Instead, meetings that decide what we are not going to build are much more productive feeling. Which is why they should be negotiations. If there is just a list of tasks that has to be done, just keep the list up to date and let the team work on them. If there is concern on priority, make a choice and let the team weigh in. Don't add to their work by asking them to prioritize on behalf of stakeholders.
And kanban is good, but really needs handoff spots between teams. Otherwise, you are just in a constant swarm. Often rewarding the fastest workers. Which is fine, but risks alienating the slower ones that could contribute in other circumstances.
My viewpoint is let marketing estimate the value of features, my job is to give them a rough estimate of how hard each one is to build, and they can decide priorities based on those two things, and tell me what to work on first. If I go too fast and lesser developers feel left out, I'm happy to pair program and do code reviews with them to help them improve.
Yea, all my projects have to have a defined release process (code freeze, final bug fixing/deferall, final acceptance tests), you can't kanban your way through that, but you can kanban your way to it.
And I should be very clear, if what you are doing is working for you, that is by far the most important thing. I am decidedly not trying to convince you to stop and change.
Further, If it can be formulated and shared with others, I'd be interested in the results. However, I have come to find that I do not expect things to generalize between people nearly as often as my instincts would want them to.
I do take this as a challenge to how I've viewed stuff. Currently, I'm on leave for about a month, but when I get back I plan on paying more attention to the process. I'm hoping I don't miss any retrospectives on projects that were finishing up as I took leave. I'm not convinced people typically zero in on the important points. I am still fully convinced you should always try.
Execs view it as being able to make informed decisions about where to spend the company's time and money.
Let's say you have projects A and B you want to do, and you've estimated revenues from each at $11 MM and $4 MM. You really want to do A, since it would establish a business relationship with a new strategic partner, but estimates come back that A will take 16 weeks and B will take 4 weeks. You only have time to focus on one project at a time, so you choose B to try to capture a quick turnaround before moving onto A. Plus it would make an existing partner happy.
Execs know from experience that the engineering estimates usually suck. Sometimes they're really high and sometimes they're really low. Execs decide as long as B doesn't exceed 5 weeks, we're OK. They don't communicate this down the chain because if everyone knows 5 weeks is the "real" number they'll (grudgingly) accept, they are pretty sure someone somewhere in the chain will spend a week optimizing the test harness or refactoring the build process or something.
At week 3, engineering says they're gonna need 2 more weeks. Execs: Sure.
At week 4, engineering says they're gonna need 2 more weeks again. Execs are annoyed but now the work is mostly done, and everyone promises on their children's lives it will actually be done in 6 weeks, so they let it finish.
At week 5, we're still on track for week 6. Execs: Sure.
At week 6, we're gonna need another week. Execs: Whatever. At this point they've abandoned all faith in this engineering team. They re-task some other critical part of the business with beginning work on project A. Some part of the business suffers.
B actually takes 7 weeks when all is said and done. Execs wouldn't have bothered with it if they knew that up front. Everyone involved in B takes a reputation hit: the product manager, the project manager, the engineering manager, and the engineers themselves. The people who blame others get an even bigger reputation hit.
Now there's pressure from some sides of the exec table to make up time on project A and get the estimate below 16 weeks from a team that had nothing to do with B.
Happily for everyone but the first team, it turns out the other team is able to get project A out the door in just 12 weeks. Everyone on the first team takes another reputation hit.
You might say that the root problem was the estimate for B was too low. That is a problem, but the fact that the estimate for A was 25% is equally problematic. Had either estimate been more accurate, the company would have focuses on A first.
All of these execs are accountable to the CEO, who's accountable to the board. The CPO or CTO or whoever are not escaping unscathed from this, because at the next board meeting the CEO has to answer for why project A was delayed into Q2.
From the exec's standpoint, it looks like we can't be trusted. But the truth is the process can't be trusted. Never give out time estimates. Use Agile and point planning, and relative cost estimates.
In your example, the team should have said that we estimate A is 4x as much work as B. Pick one, we'll get it done as fast as possible.
You can have a fixed time schedule, or a fixed project feature set. You can't have both, without the project being late, and features being compromised.
You identified the first one correctly: they had the lead engineers estimate in isolation and not involve the team in any way until the end. Beyond being a bad way to estimate, it's a bad way to manage a team.
But then you said you should never provide a real estimate. That conclusion doesn't really follow. The leadership team has to come up with real estimates. "A is 4x as much work as B" is good information, but it's pretty useless when it comes to actual calendar planning.
The second issue, and the larger one, is that long, stop-the-world refactors (or rewrites) are the fastest way for the engineering team to lose all credibility in my experience. The business keeps moving even if new feature development doesn't. I've found incremental refactors to be the best way to go. And those are much easier to estimate.
As far as schedule vs. feature set, if you estimate properly and prioritize all of the least-needed features at the end, if you find yourself falling behind you just pop those tasks off. But you build that contingency in and communicate it up front.
In our case bogging down the entire team in time estimates would have wasted even more time and we would have accomplished even less. Had we kept to an agile process with sprints we could have met the fixed schedule expected, and delivered more benefits given that losing our leads in planning for weeks on end meant they contributed relatively little to the refactor.
What I said was no contradiction. How does one arrive at the finish date? Finger in the air? There are essential features and nice-to-haves, and to get a delivery date you need to estimate them. But it's an estimate and so you plan for contingencies.
At the end they required twice a day reporting meetings, the standup to say what you were doing, and an end of day status meeting to say what you did so it could be communicated to the exacs. Two more meetings where zero development gets done.
I am not really disagreeing with you. It's reasonable to do basic estimation in most projects. Usually marketing/management need to know when something can ship in order to plan the launch plan. So you figure out how long you need to be really confident the essentials will be finished, and use the nice-to-haves as padding that can be deferred if necessary to make the date.
But if you don't have to hit a specific date, say the product is already shipping and just needs regular constant improvement, just do it in fixed schedule sprints. Focus on the highest priorities every sprint, get as many done as possible during it, ship the new release, rinse and repeat. A well run sprint can be great because eschewing time estimates means more time spent actually building the product.
But if you are forced to give them, never give a single number. Use ranges. Your range for a confident estimate should be roughly 4x between the fastest and longest components. For example, 1 to 4 hours for a task you are very confident about. For things you have less confidence on, your range should be closer to 8X.
Why do this? Because wide ranges are the ONLY accurate software engineering time estimates. You need to communicate the inherent lack of precision in any estimate. For every time estimate you don't know
1) How many hours a day you'll be allowed to actually code, outside of meetings, standup, planning sessions, emergency bug fixes, etc, etc. 2) Which member of your team will actually end up doing it, the fastest one, the slowest one, or someone in between. 3) How many days will be lost to illness or personal issues. 4) How much the actual feature or story will change as it's developed and they realize the design is actual shit. 5) How much other code will need to be changed when you actually rip the lid off some of the older code it will interact with.
etc. etc.
Scrum has a similar technique only allowing Fibonacci numbers, I think the reasoning behind it is similar.
Obviously the receiver of the estimate should know that you are using this method, otherwise they might start questioning you thinking this estimate should only be 17 for example.
What decision are you trying to make using this estimate? Understanding context helps you to understand what sort of estimate is needed - or whether an estimate is needed at all.
The best thing for both the company and yourself is to try to be as high bandwidth as possible and discuss all the potential risks and tradeoffs. Find out what the real goals are so you can optimize a solution to provide for them. Many managers and customers get stuck early on with a particular approach, but if you apply some creative engineering you can come up with a solution which is easier, or folds in with some other goal you were already working on, leverages existing functionality etc.
its on you to express it in terms they can understand from their own position: risk, cost, headcount, schedule, dropping existing features.
if your organization doesn't want to engage in that kind of semantically meaningful discussion, or comes back and says 'I know you said 3 months, but get in don't in 2 weeks' then you need to start managing upward and looking for a new job. or you can just overestimate, and enjoy your free 2 hour lunches for as long as the money lasts.
The process of building software includes discussing what has the best impact to effort ratio. Both are estimates and will not come out as expected.
The logical response to managers that expect you to always hit deadlines is to inflate estimates enough that you will.
1. Why do you need the estimate? Is it to get budget approval, or because you're trying to meet an approved budget? If they don't have budget approval then I'm going to charge them something to gather requirements and deliver an estimate. No budget approval = low on priority list.
2. We charge a blended rate of $xxx per hour. If you have a detailed requirements list I can tell you if we can do this within your budget. If they don't have a detailed requirements list then I'm going to charge them to provide one. After I have the requirements list completed I'm asking for their budget.
3. Don't want to pay for a requirements list? I understand. Its okay. Something like this typically costs (low bid), but that will likely change if the requirements are different then my assumptions.
Going to hold onto these. Good phrasing/ideas.
Big projects can suffer from this because little architecture is involved (and we worked on already existing systems). Otoh, I’ve seen no big local project that wasn’t an architectural mess once it hit a deadline hard. It is better to not have one than to have a set of rules that you must obey, while other parts of system don’t even care of or contradict these.
Personally I don’t believe this “I’ll get back to you with an estimate” thing. It is just first iteration where you go relax and try to formulate at least 5-10 questions that you think you should ask foremost to start building an estimate. If you come back with an answer instead, then it is still wrong, maybe even worse than random, since you estimated a different thing. For us, estimates and requirements always moved, as they were functions of delays and business priorities that change unpredictably. I didn’t like to deliver the entirety on a deadline because my job was to support that for years, not throw a bill at them and run, as all integrators did.
Maybe, maybe these SE-users are speaking about coding phase, when all “business analysis” was done perfectly. But ime it is you coder who collects all the knowledge, because no one can. Or it is you who is non-coding analyst who must estimate, because non-analytic coders don’t have a clue what a big picture is.
https://meta.stackexchange.com/a/19514/154820
for medium to large size projects i like to give a confidence score to my estimate to give a sense of how much unknown there currently is.
for example: we think we can get this ticket done in 5 days, but there's a 50% chance it will take two weeks (because of complications in X, Y, and Z).
or the other way, my estimate is 3 days, but if this one thing works out that we're looking into, there's a chance we can finish it in 1 day.
what people are generally asking for with estimates is sizing and complexity, so providing some additional information helps.
i think we have phobias for estimates because of management repercussions that only happen when you miss an estimated delivery date, or for organizations that put external dates based on a best-case aggressive engineering estimate. then when the engineering team misses the date, there's a retro (when retros should happen for both good and bad sprints.)
i've never been clear why more organizations don't soft launch features or products, and then pick a marketing/external announcement date.
The only accurate time estimate is a range with at minimum a 4x difference between best and worst case, and preferably more the less certain the estimate is.
The reason for this is that in practice I've found estimates are an indirect way to define scope.
You need to ballpark the minimal viable product in a range from under a month, a month to 3 months, 3 to 6, 6 to 1 year, over a year.
After that, its just about downscoping all the "good to haves" based on if there's time left from the desired deadline or not. As long as you delivered the "must have" in your timeframe it should be good.
On the other hand: Do it. Laugh. Laugh in his face when you are called to help on a project that isn't ready after 1 year and started without you. When they show some wire-frames and ask you how long it takes without telling you about the actual technology used for the project.
Summer 2015 I was uninvited from a project meeting. My coworker was planning the project and discussed it a bit with me. He wasn't the one who decided that I shouldn't take part.
Summer/Fall 2016 and the project isn't going anywhere. I'm invited to help. I do my best, besides the other projects.
January 2017 and my coworker is gone. He gave his 3 months (+ x days) notice and told me December 2016. Shortly after it I gave my 4 1/2 months notice.
The new deadline for the project was May 2017.
They launched November 2017.
"Cool story, bro. What was it? Some revolutionary VR or AR application? AI supported super gadget? Brainwave reader and interpreter?"
…
It was the company's own website.
Any project like this one can often go off the rails if the right decision makers are not involved. This is not necessarily the fault of the project leader as it can be unintuitive in bureaucracies.
May of 2015: hired at startup that wants to use Natural Language Processing so salespeople can send text messages directly to their CRM (Salesforce, PipeDrive, etc). Very excited about the project. The app's initial UI design is a traditional one, with NLP helping to smooth the basic task. The full app would take about a year to build but we all agreed we could have an MVP by August.
August of 2015: without consulting anyone on the tech team, the Board Of Directors decides the app should get rid of all standard UI elements: no forms, no buttons, no links, no drop down menus. Instead, the interface should be a pure chat app. This makes the project much more ambitious, which I was very excited about, but which I felt would delay the project 2 or 3 months. Nobody was happy with my estimate.
September of 2015: the CEO was able to show a demo to the Board Of Directors. The demo was an illusion, as we had no error handling, and it only worked for some carefully planned examples, but I thought it was a good sign that we could show the Board that we were making progress. Unknown to me, the Board then asked the CEO for real feedback from real customers at the next meeting, a month later, in October. There was no possible way for us to finish the product, and find customers, and get feedback, all in a month, but the CEO was a bit of a coward, so he promised the Board we would do this.
October of 2015: obviously we did not have customer feedback by the next meeting of the Board. At this point the Board decides we have missed our schedule and they begin to panic. We are slowed by the fact that our "NLP expert" is inexperienced. Myself and our iPhone programmer tell management that we need to fire the current NLP guy and hire someone with real experience. Management initially agrees but then later changes its mind, for reasons unstated.
November of 2015: we get a basic demo working, and it has enough error checking that we can show it to potential customers, and not be entirely embarrassed. But the stress of the previous month has wrecked relationships inside the company, people are shouting at each other constantly. At this point I step away from the project, but I remain on friendly terms with the guy who is doing the iPhone programming.
January of 2016: company gets two trial customers, but these customers won't pay for the product. The NLP engine is improving, but at glacial speed, as the "NLP expert" is at the beginning of his career. He was a nice person, and I can believe he will eventually become a good programmer, but it didn't make sense that a company that wanted to move fast also remained so loyal to a guy who could not do what was needed. An established company could/should afford to have an apprenticeship program, but a fast moving startup can not offer an apprenticeship to someone working on a core technology.
April of 2016: the iPhone guy again asks management to find a new NLP expert, someone who can move the company forward. Management reacts by listing every bug ever discovered in the iPhone app, as a way of telling the iPhone guy to shut up.
May of 2016: the iPhone programmer quits and gets a job elsewhere.
June of 2016: the company is almost out of money, so it cuts back on spending, reduces the team.
summer and autumn of 2016: without much staff, the company crawls along at a glacial pace
Early 2017: the company gets more funding, begins moving forward again.
Summer of 2017: the company now has a few trial customers, but they are paying trial rates, which is to say, almost nothing. The company is not anywhere close to the breakeven point.
The company continues to burn money without making much progress. There are clearly some fundamental leadership issues that should be addressed, and one of those are the ways that estimates and budgets are created. Also, the company would be in much better shape if the leadership listened to feedback from the tech team.
I wrote about all of this in detail here:
1 hour becomes 2 days. 4 days becomes 5 weeks. And so forth.
I'm actually somewhat decent at estimating well defined work, so it actually annoys him when one day is one day; but I appreciate his work when some odd stumbling block (usually political; my estimating skills crash and burn when it involves working with humans) comes up and the time doubles or triples the estimate.
A year is an eternity anyway, you shouldn’t start a project with nothing coming out of it before a year, that’s suicide on so many plans.
So, 1 year is the time the first interns team will work on it, 10 year is the duration before the project is retired. Looks like both estimates were right.
My engineering manager at a games company I worked for told me that "three weeks" was the estimate that really meant they had no idea how long it would take, because the task was too hard to estimate. Anything longer than three weeks could, and did in fact, sometimes take six months.
Large projects can always be broken down into smaller, more well-understood tasks. You can't build something large all at once. Large estimates are just an indication of a lack of discipline in planning because they mean that the project hasn't been decomposed into its constituent tasks. Large estimates are lazy and can only be right out of sheer luck. In the team's I manage, I don't let anyone start a story with an estimate of more than two days.
Are you being serious? That only works if the people consuming these estimates don't have a clue and therefore can't call BS. This would get me aughed out of my job.
If I were to estimate things as suggested in this thread, it seems to me things would never get done. Ever. It simply boggles my mind people want and will fudge estimates on this scale.
It's only three days in to the project that you realize there's a mismatch between what you need and what the project provides. You start a conversation with the project's lead developer; but they're only able to have half of any given conversation in a day. How hard would it be to spend five days researching the cause and severity of the mismatch while discussing solutions with the remote developer?
There's 8 days.
You decide to write a patch for the open source project shepherd it through the internal and external approval process, burning up another two weeks of time.
18 days.
You finally get back and finish the original project, redoing a fair bit of work to account for the changes your patch introduced.
23 days.
You're interrupted throughout this process several times by fires in production that need to be fixed RIGHT NOW.
28 days.
Let's make the assumption your PM did their little trick, and slated 2 months for this project. Despite your own estimate being off (through no ill-will by anybody) by about 6x, your work is still completed before it's expected, and the teams depending upon your work are happy as clams. Even if it had finished in the original 5 days, everyone would be thrilled, and you'd get more work to do.
From the PM's point of view, it's a win-win situation. From your point of view, the worst you have to endure is a twinge of "don't you trust me?", offset by the relief of not having to answer "why isn't it done yet" every day from day 6 on.
While your example showed how project implementation can creep up, I already knew that and my problem is more my inability to give accurate estimates and people's willingness to stretch their estimates by a lot. I KNOW some of you will not agree with me, but I think that giving 4x as long estimates will lead to developers spending more time on the actual implementation without noticeable improvements in the delivery. I know my natural tendency is to do so.
You could deliver something in 1 hour, BUT, the customer/person that could answer you WHAT to do exactly will take days/weeks to be AVAILABLE.
This is not a joke.
For one of my very recent jobs, I need to wait 1 FULL YEAR TO MOVE ON. Then weeks after that. Despite already being paid in half!
--
Developer time is NOT ONLY the time the developer smash the keyboard.
Alternatively what I'll do is point out that their request is simply incomplete and I can help them create a better request, which I will then estimate (it's all still billable though).
Also worthwhile is to consider the overall likelihood of success for projects you take on (and what it implies for future work, quality of life while working with the client, etc). I wasted a lot of time early in my career on projects that I knew were doomed just because someone agreed to pay my rate.
Often it is the case the goal can already be achieved with existing functionality, but may not be as direct or elegant. That is why I include the "minus" bit covering this scenario where a solution is already there.
I can’t tell if you’re being serious. How do people respond to such a non-estimate?
My approach also helps start a discussion about how to reduce the size of the plus/minus bit, such as reducing scope, breaking into multiple pieces etc.
And the minus bit coming to negative time works well too. It becomes a decision about using what is already there even if not perfectly suited versus doing the new work, and how much of it to do.
Sometimes it's an environment change. Are these consistent? Can they be modeled statistically?
Often, it's an inaccurate belief in the work to be done. Was it lack of requirements, that had to be expanded? Etc.
When I work in development, I spent up to 20% of my time on "overhead" tasks. These are of the estimate review type, not meetings. It's something a great manager will do without wasting his team's time, but as an independent, I spend a lot of time doing it myself, and it's amazing how much it helps to dedicate time and write things down.
get a sense whether task is S, M, L, or XL from developers etc... and then multiply it by whatever factor you think S-XL should be in hours/days.
people can't seem to provide time-base estimates - but generally have a good sense of a complexity of a problem.
"Is this a hundreds of dollars or thousands of dollars problem" can work wonders.
I think what I intend to say with that can be, "I don't know how long it takes and it'll take more planning to figure it out, but I'm willing to participate in this conversation so that we can continue to effectively plan in the current context."
If the current context is "concrete" then you probably don't want to give a number. If the conversation is (and this was my assumption before) "here's a feature we want done but we're trying to figure out it's priority and feasibility" then giving a gut check adds value. You just need to be clear in your messaging.
And if someone uses a gut check number against you "but you said X days!" Then you need to call them on it and make sure you explained the discrepancy.
When people say they need feature A, B and C and ask when I will have it done, I split it up in tasks that take below a day, estimate these tasks and add 50% for safety.
This works in most cases.
But often the reason why projects won't finish on time is, because people don't know what they want.
They ask for A, B, C, you estimate it right or probably too pessimistic and get done ahead of time, but still the project is late in the end because they forgot about D, E, F, G, H and J. In the end you can fit D, E and maybe F in the time you have left because you got A, B, C done quicker than estimated but G, H and J are still missing.
If they come back and say, "oh, we forgot to say we also need D", then the response should be, "that's no problem, we estimate it will take an additional X hours or dollars to implement that. Would you like us to update the proposal?" Or, if you're under budget and you're charging hourly rates instead of a fixed cost, you might even say you can probably add it without going over budget.
PS: I just realized that you're probably talking about internal "clients" so there's no contract or even payment. But I think the same general idea applies. When they bring up extra features, you just say that it's going to take additional time.
edit 2: I guess this doesn't actually help with getting the project done on time. The initial time estimate is still going to be off for the reasons you mentioned. So this comment isn't really an argument against anything you wrote.
PM: I want a bridge. When can we start using it?
Eng: Uh, ok, where is the bridge? What's is going over? What kind of traffic will it carry?
PM: I want a bridge. Why are you being difficult? Tell me right now when we can start using it! I've got to go promise my boss that you committed to getting it done by then! Oh, btw, it needs to be done next week.
If you are lucky, you will recognize from their "requirements" (often just a verbal description) that they are describing the characteristics of a bridge (e.g. continuous bidirectional flow), not a ferry.
More typically, after you have built the car ferry, they ask why it doesn't implement continuous bidirectional flow despite never specifying that as a requirement.
Usually they both suck. The manager will send an unclear email. The developer will start developing right away. None have any idea what they are supposed to do, why or when.
For managers the value of a developer is the solutions they build divided by the amount of time and information they need to build it.
You have always the choice between delaying further or delivering something they like as soon as possible, risking they don't like it.
Normally you would need a prototype, throw it away after you saw the reactions to it and build something new.
But often there is no money for this :/
https://youtu.be/BKorP55Aqvg
A client wants something. In software, the something is usually complex and not well understood, a contract is done to do a something like that. During the project, they will invariably find new information and issues, the scope will change, sometimes dramatically.
For the comparison, consider a building. You want to construct a new building, it's easy, you say what land you have, the engineers will tell you how large and tall they can build on that, you get to pick the interiors among a few office/industrial layouts. You can all understand what is being discussed.
In software project, there is no physical limitations and no regulations. The limits are not clearly defined, noone will see or believe them. Worse, you have to integrate with clients/new/legacy systems that are not well defined and unknown ahead of time, nothing you can plan about that.
What happens in practice is they didn't realize they were asking for additional features -- they thought they were asking for D, E, F (etc) all along in the initial request. They just assumed "Email signup link" also meant spam handling, re-sending, prevention of double sending, etc. Good product managers account for these obvious things, and leave padding for the unexpected. But good product managers are rare. Most software developers I know have never even met one. So instead you have a political dance where you try to plan for the things they aren't telling you, along with all the _expected unexpected's_ that come up in normal software development, and set your deadline's appropriately. Definitely the worst part of software development.
1. It's never repeatable. Nearly every single thing built in software is being built for the first time. Even if it's some government CRUD application, and even if you're leveraging sixteen popular software frameworks to build it, that application will still be in some way different from other applications in the same niche. There's no blueprint for building software; mechanics have a gigantic (sometimes accurate) database of the time required to do a variety of jobs, tract home builders are all building the same thing over and over ... the closest that software ever gets to that is themes or skins for frontend UI, and even those inevitably end up getting customized in some way.
2. Nobody wants to pay to build a blueprint. The only way in the software industry to figure out how complex a particular application is going to be is to build it. Developing an accurate spec is exactly as expensive as building the actual application. This is because the "engineering" part of software engineering is still practically nonexistent.
3. And while we're on the topic of engineering: there are no standards. Or, there are thousands of different standards, depending on how you look at it. That CRUD application could be built with one of many different interfaces -- should it emulate data entry terminals, with simple tab-navigable text fields? Should it use Google's "Material" UI? Bootstrap? There's no comprehensive study which has compared the amount of end user effort required to use each interface, and therefore there's no industry standard which says, "this is always the right way to build this thing given these constraints". Compare to something like a low voltage contractor's license, which expects you to be familiar with the different types of cable to use in different building environments.
4. It's buggy as hell. Ok, so you're experienced enough to navigate your way through everything else up above and not get burned too badly. You have a preferred toolkit, you specialize in a particular niche, cool. Well, today, Chrome just released an update that broke the way your application handles cached content. (Happened to me.) The vendor for one of your frameworks released an urgent security update that completely broke critical functionality in an unrelated part of the application. (Happened to me.) The two libraries you're using together, which don't use any of the same globals and should be entirely isolated from eachother, are still somehow interfering in a way that's not covered by the sparse documentation. (Happened to me.) There are millions upon millions of lines of code involved in every single interaction with a computer now, and they are imperfect. Many of them are written by amateurs, people who -- even though they're getting paid to write code -- have only been in the industry for a couple of years and they're still re-learning all of the same lessons that everyone else before them had to learn through trial and error.
5. Too many programmers like shiny new things. We could fix a lot of the above if we slowed down and "refactored" our industry -- started doing real usability studies, establishing standards for storing sensitive information, more rigorously separating security and feature updates. But, nobody really wants to do that. They want the new thing: the new programming language, the new framework, the new browser feature, the new toolchain. I think this is largely because while programmers spend a lot of their time talking to machines, they aren't machines themselves. They need a steady stream of new stuf...
Yeah, it's an interesting point, because it's not even "many", it's "most". There's a talk by Bob Martin where he goes into this. His statistic was that the number of working programmers is doubling every five years, which means that at any given time, around half the population has less than five years' experience.
The best (and fastest) way to build something is with minimal specs, and modifying as you go along. As your clients (internal or otherwise) actually get to use features as they are built, they'll provide tons of input that they wouldn't ever have put in a spec.
This is also why you should never, ever, do fixed bid work in software development.
The best way is to overshoot the time and offer the flat rate. If they bite, I can treble my usual rate, if they come back with a hard negation, I take that as an indicator that they will be difficult to work with.
It's a a tough balance.
By the way, watch out when changing from daily rate to project. Legally, you have to deliver the project when you bill for a project, so you are forced to make a contract with well defined project scope to cover your ass. It's quite a lot of (billable) hassle.
Interesting you bring that up though. I had a client through a contracting company. After the project was done, they said they actually asked for something more, which was never even discussed. I ended up partly unpaid even though I was clearly correct.
The law works when the money is worth chasing.
The law is never in your favor as a single man or small shop. It's too expensive to go to court, then determining what should have been done or not done as per the contract is a lengthy and uncertain proceeding.
Clients who wanna pay per project basically want to move the risk over to you. They should pay a premium. Write a solid contract and increase the fees/duration. Most should give up when you start increasing the quote just to establish the contract, it's easier to start a POC and see what could be achieved.
Most of the time, you will give an estimate, and the manager will turn the sum of your estimates into a deadline. Don't fall into this trap.