Nicely complicated--- useful if the boss or client wants verbiage. If asked directly though, I usually say "If I've done it before, I know how long it takes; if not then not."
Re "environmental contributions", that was part of the discussion, but thinking about it - estimating total effort and elapsed calendar time are very different!
It's because developers are eternally optimistic. How else can you explain someone hitting a "compile" button a hundred times a day, hoping each time to get no errors?
This reminds me of assembly language in college... Each assignment we were told to end the program on a different type of ABEND ("for this assignment I want you to generate a SOC8, integer overflow" etc, etc...)
I prefer to take my advice from someone who has demonstrated the ability to repeatedly run projects and estimate them in advance to surprisingly high accuracy. I therefore recommend Software Estimation by Steve McConnell, available at http://www.amazon.com/Software-Estimation-Demystifying-Pract....
If you like Steve McConnell's work, you should take a look at http://www.LiquidPlanner.com, we built our entire product around the notion of ranged estimation so that your schedule actually captures some of uncertainty involved in estimation.
Good post Ash (good to see the discussion being consolidated)
IMHO - it's usually due to the lack of "up front planning" by Managers / Analysts.
If you compare building software to more general "building endeavors" (for example building the Olympic stadium) then, IMHO, developers are really the "construction workers" (the people actually doing the work). However no intelligent Olympic committee would countenance starting work without (for example) an architect making an incredibly detailed plan (and even a working replica model).
IMHO the problem we suffer from as developers is most often diving in too fast because:
a) we're the construction workers
b) when we do use Architects they are "construction workers too" (as opposed to being a trained with 'different but complimentary skills - it would be hard to imagine Sir Norman Foster laying bricks)
May be because software development is a creative process or the same reason why Artists won't tell you how long they will take to finish a painting or Authors repeatedly miss publisher deadline for books.
In my opinion, software development is exactly a creative process. Many times, you have to come up with a creative solution that fits within certain constraints. Also, in the beginning usually only a vague image is known. The artist (developer) has to fill this in and make sure it is what the client wants.
Also, many creative works are collaborative efforts as well these days. I'm not sure how that changes anything. If anything, it makes estimates harder.
You could equally say "Just because math is involved doesn't make it engineering".
But I suppose it's something in between art and engineering.
With regard to "sell weird color splotches for millions" - Isn't that the same with code? You can sell a simple game or app to millions of people and become rich. The amount of money you earn doesn't have any connection to complexity.
You are right, there is a sub-gendre of entertainment software that does have similarity to art. But even there you'll never buy a product because it crashes, while a work of Art can be valuable because it is ugly
On the other hand, in code styles, languages, idioms, a lot depends on taste. One person might find Lisp code ugly, another one might love it. Some person might find some library useful as a modular object oriented system, while someone else might find it a hellish heap of macaroni code.
Artists might like to experiment with different styles, so might coders.
Also, it's not like most artists try to sell people ugly things. That's also a very small niche. Most of them simply work for companies and design the things that they are told you. Just like coders.
We could debate about this for ages, but it seems very clear that there is both art and engineering involved.
No one will buy a skyscraper design that has no doors or windows and crumbles under a breeze. Architecture and industrial design are two examples of things that need to cater to practical needs yet are usually regarded as forms of art.
Of course there's also people who think that art is only art if it doesn't do anything and isn't "needed". This also creates problems (for example, can a movie be art? if yes what of the "jobs" needed to create the movie are art? who are the artists?)
I estimate by considering an optimistic, pessimistic and normal value for each task. I then have a spreadsheet that does a modicum of statistical analysis to give you times against probability that the task will be finished.
Some of the benefits I find are:
1. Getting into the pessimistic mindset helps produce better estimates
2. Tasks with a large spread are things you aren't really clear about, and are an obvious target to be attempted first
3. Giving a range of values to the person asking for the estimate seems to help them remember it is an estimate.
Need to include "tasks which will take significant time but I can't tell you what they are because we won't know until we get there". Managers can't seem to deal with the notion that 10-30% of a project is completely unknown when writing initial schedules.
If you give an estimate that's too far out, it'll be outside your client's planning scale. That means it's so far out, she doesn't know what will happen by then, she won't do the project. In my experience, this is scale is roughly 9-12 months, unless you're dealing with a MegaCorp.
So most estimates end up at 6-9 months, which then slip into 9-12 territory, and hopefully not much further =)
In my experience, coding time is usually easy to estimate, and that's what most estimates coming from developers should be taken to estimate. But then you have integration (nothing works), testing (everything is broken), and then realizing what the client meant when she said X (solved the wrong problem, optimized for wrong features), those are the tough ones.
You should take the estimate for coding time (coming from the developer), and multiply that at least by 2x. Of course, going back to my original point, that estimate may be outside the "human timescale", so you may not be able to tell the client that...
So, to answer your original question, developers can estimate primary development time up-front pretty well, but they suck at estimating integration, testing and shipping time.
A related thing I noticed is that if you ask a developer during the late(r) stages of the project when she'll be finished with the "next stable version", the answer usually hovers around "in 2 weeks", even if the actual answer is more like 3 months. The developer isn't lying, she genuinely believes that it can be done in 2 weeks.
That's why you want experienced engineers and not fresh college grads leading teams and giving estimates. They've gone through these experiences, have a lot of soft data metrics in their heads, and can come up with good multiplicative factors. I'd be curious if orgs like Google have a database of project data to aid planning.
she genuinely believes that it can be done in 2 weeks
This is where having good historical records is useful. If I have done 2 similar tasks before and they took a month each, then a 2-week estimate has no validity.
As far as general code estimation goes, lines of code and complexity taken together form a pretty good metric. If I estimate I have to write 50 lines of high-performance multithreading code, I might say it will take 8 hours. If it's 50 lines of fizzbuzz, estimate drops to 1 hour.
But like you say, that's just coding (and unit test). The full task probably requires (1) getting or understanding requirements, (2) at least some cursory thought about design and (2a) having a hallway review with somene if you're not sure. Then when we're done with code, unit test (don't forget to estimate how long it will take to create, debug and run the tests) and Code Review (how many reviewers do you need, how long will it take them to get around to inspecting your code, etc), someone has to estimate (4) how long the test team will take to write test cases and do the system test of the new feature, (5) tech docs has to update the user manuals, (6) sysadmin or the Release Master estimates how long deployment will take.
In short, at least in the environment I work in, that 50 lines of code that took eight hours to write, will easily require an entire week before any customer can see it. In my experience, it's often not so much that the developer didn't estimate accurately how long coding would take, it's that the developer never thought about all the ancillary tasks that have to be done (code review, unit test, documenting the design changes, etc), so they were not in the estimate.
You don't know the problems/limitations you will run into until you begin working on your project. We have immersed ourselves in a culture where it's expected to estimate the unknown.
Almost all bad estimates are due to a lack of information. In software development, the information that tends to be missing are the particulars of the task being estimated (which code needs to be modified, in what way). This missing information is highly unique and not to generally transitive from one programing assignment to the next. Additional information about the process is usually not the problem. Things like CMM tend to focus on the process information rather than the particulars of the coding problem at hand. Which, leads many developers to think: "You're doing it wrong."
Speaking from experience at a large multi-year project at a BigCo, I am quite familiar with the failures of estimation. Some information is absent because some things are simply unknown due to inexperience or inability to predict future changes or mistakes (reasonable). However, other things are lacking because there is no motivation or demand to produce them. For example, waterfall processes generally throw the software development lifecycle on a schedule for a system or subsystem: requirements, design, implementation, test. If the lifecycle stages are not divided into tasks upfront (which then should be further divided), then the complexity of each stage is basically not being factored into the estimates for each stage. This is setting the stage for failure unless it is a project that has been done repeatedly over many years in the organization.
Furthermore, continuous process improvement and monitoring is not done enough even in organizations that declare achievement of higher levels of CMMI. Our team had subsystems that overran their initial estimates by over 100%, causing nearly a year in delay. However, there were basically no penalties or major process changes despite the fact that multiple subsystems overran estimates multiple times. Process professionals, engineers, and managers are simply not aggressive in tackling these problems.
In addition, estimates tend to come from individuals. I imagine that a more collaborative team-based estimation approach would be better, factoring different levels of experience and sharing the burden of making estimations realistic. Also, recorded estimates need to be coupled with recorded justification.
agreed. we don't have a language to define the problem in the first place.
i am great at estimating - unfortunately i often go over because what i heard was being asked for is subtly different to what the client thought they were asking for
Lt. Commander Geordi La Forge: Look, Mr. Scott, I'd love to explain everything to you, but the Captain wants this spectrographic analysis done by 1300 hours.
[La Forge goes back to work; Scotty follows slowly]
Scotty: Do you mind a little advice? Starfleet captains are like children. They want everything right now and they want it their way. But the secret is to give them only what they need, not what they want.
Lt. Commander Geordi La Forge: Yeah, well, I told the Captain I'd have this analysis done in an hour.
Scotty: How long will it really take?
Lt. Commander Geordi La Forge: An hour!
Scotty: Oh, you didn't tell him how long it would really take, did ya?
Lt. Commander Geordi La Forge: Well, of course I did.
Scotty: Oh, laddie. You've got a lot to learn if you want people to think of you as a miracle worker.
This might work in StarFleet, but the problem is that if you promise a manager something repeatedly, by overestimating, they will start to assume you are padding your results.
Having worked on scrum teams before, I find its a good structured way to have a dialog about how much work is left, and how long you expect it to take.
So rather than saying an hour. You would say this is as easy as changing a diaper, or its as hard as navigating through an asteroid belt at the speed of light ( having or not having done it ).
I would appreciate it if managers ask:
Have you done this before? (yes, no)
How similar is this to something you have done before? ( very similar, kinda similar, totally new )
What concerns you about the task, or what risks do you see coming your way? ( detailed answer )
What can I do to get you the resources you need to help you deal with the risks you can foresee now?
> the problem is that if you promise a manager something
The real problem is thinking of "estimates" as "promises" when really they're rough guesses. If you go around holding people to a guess as if it were some kind of gospel biblical contract then yes, they will lie their ass off to protect themselves. Combine that with the fact that most organizations attempt to control the platform used, computers used, monitor size, operating system, IDE, editors, revision control tools, languages, documentation systems, testing methodology, meeting requirements, pairing, and nearly everything they can, and you start to see the real problem is....
managers who don't know programming, motherfucker. :-)
"At that meeting, SAP AG executives and engineers represented that the software was a mature solution and conducted a demonstration consisting of what they represented was the actual SAP Waste and Recycling software," the complaint states. The company later discovered that the software was a "mock-up version of that software intended to deceive Waste Management," according to the complaint. SAP has admitted to this in "internal documents," the complaint states.
"From the beginning, SAP assured Waste Management that its software was an 'out-of-the-box' solution that would meet Waste Management's needs without any customization or enhancements," the statement reads. "Unfortunately, Waste Management ultimately learned that these representations were not true."
Managers want promises because execs need promises because the board is critiquing their job performance based on when they ship the next product.
The board wants promises because they want to know when they get paid.
If you find it stressful that people are turning your vague estimate into a promise, imagine having to make promises on other people's vague estimates.
You are in part outlining the start of some standard methodology for project time estimates. That's the good news. The bad news is that, for cases of never done this before and this is totally new, the methodology makes no estimates!
That's why we were using complexity estimates in points (say 1 to 5) instead of actual time estimates in my last startup. Once you have some worklog, it's easy to calculate how many points the team can get done in a week (or whatever your estimation cycle is). This little abstraction made estimates much more reliable.
I guess some "agile" methodologies propose a similiar approach, but we didn't really follow any specific one, just figured out what works best as we moved along and kept the process as lean as possible.
Just to clarify what k7d is saying when he quotes "agile" methodologies:
Any prescriptive agile methodologies (first, do X. then, do Y) are really antithetical to agile. True agile processes are exactly what his own startup did. They used what worked. They implemented a practice (complexity points) that was relevant to their principles (accurate estimation). Of course, you should also be tweaking your process as you go along.
There's another way as well; specify a range of time within which you expect the task to be completed. This also communicates your uncertainty about the estimate at the same time.
The author makes the point nicely at the front: "because the nature of the work is creating."
I'm not a developer but I can confidently say that it's not just developers - NO ONE can estimate time when it comes to creative work. I use the term creative loosely, it could be anything from developing a business strategy, to crafting a story, to art directing an original visual, to writing code for a novel problem.
The problem is the same one that makes it difficult to say "when will we have a cure for Parkinson's," "when will we have fusion power," "when will we discover a viable alternative energy"? You often see estimates like 20 years, 40 years, 50 years - but properly interpreted, these aren't actual estimates of the amount of time it will take, but rather like a proxy for a much more probabilistic, uncertainty-oriented notion.
Hi hammock, thanks for the comment (I'm the author). For some reason you've reminded me of the motivation factors Dan Pink talks about. I wonder if trying to commit to estimates makes our creativity degrade in the same way as paying cash bonuses?
Want to hire the most experienced programmer out of a group of candidates? Ask them to estimate a programming task, then pick the one that gives the longest estimate.
I'd say hire the one that gives you the best justification for their estimate. It's important for people to understand what is and is not involved in an estimate as well as why they think something is hard.
Once you communicate that, you can uncover a lot of potential problems and misunderstandings. For instance just the other day my co-worker was asked to estimate a task that seemed to our boss to be pretty small. He estimated it as being 2-8 days, it wasn't until they actually started talking about _why_, that they both agreed on the requirements. The resulting task will probably take 1-3 hours.
So sure, as a developer learns more they may estimate tasks as being longer, but communicating your assumptions is more important than throwing out huge estimates to cover your ass ;)
> It's not just developers that are bad with estimates either. Everyone at some point is just winging it because it's something they've never done before and won't be able to successfully make a judgement until they have.
When my manager asks, "How long will this take," about half the time my honest answer is, "I don't know."
But from your manager's perspective, you probably do know. You know if it will take more than or less than a year. You know if it will take more than or less than a month. You have some idea of what needs to be done. You have some idea of what parts of it you don't know, etc.
One of the most important things I've learned about estimating is first asking how accurate an estimate is needed. Often the person just wants to know "can I have it tomorrow, or will it take 3 months?"
Depending on the task, it might not actually even be possible. Or, it's doable but not without rewriting major libraries or buying hundreds of servers.
There are 2 sides to this, one of which is detailed in most comments below, and I see it from a lot of devs I work with but mainly the less experienced ones, and that is the eternal optimism issue. That's been covered a lot already, so no point in belaboring that.
The OTHER side is that, at least given my experience in the financial software domain, managers don't WANT accurate estimates. They absolutely abhor them. And developers are punished for giving them. So they aren't given.
An accurate estimate, such that one can be made, is usually around a 70%-80% confidence interval. It includes many specifically unforeseen, but generally known issues such as problems with the environment, lacking specifications and time required to get them, including "making $#@! up" fudge factors, technical hurdles, cogitation and exploration time, etc.
But managers can't hear that. All they want to hear is something they can sell to their superiors, which is often the customer. An accurate estimate is almost always going to be larger than that, so they won't accept it. So, the development staff is forced to skimp on quality or features to make an artificial date. But that's ok! Why?
Well, it's partially an organization's willingness to accept "there's never enough time to do it right, but there's always enough time to do it twice" (or more), but that's not even half of the issue. The larger part is that NO ONE WANTS IT RIGHT THE FIRST TIME. I've seen this time after time.
It's a win/win to provide sub-par product. Why? It makes a manager look great to give low estimates (win), hold the developers to that, and deliver a product with less-than-promised quality, or scope, or both. That provides customers something to gripe about (which EVERY customer wants; it makes them appear "tough" or "thorough"; win), and it gives the managers something to "fix" and appear reactive to the customers' needs (win).
Who's holding the bag for all this win? The devs, and support. There are inevitable promises of being able to go back and fix things, but that never happens; once something is in production, no matter how crappy, it has the almighty momentum. If it's "working", almost no matter how fragile it is, there's no appetite to change something that works so that it can work better. That provides no revenue, and does impart considerable risk (with limited or no QA, having moved on to the next thing) and/or cost (keeping your QA around to regress fixes).
Although that mindset infuriates me, I don't honestly know that it's not the best way. It seems to have evolved, and companies that do it seem to do ok, so maybe it's the Darwinian process at work.
Well, yes, but too often "soft" requirements get lost, like performance or the ability to be supported by anyone who wasn't the author. I'm not IN support, but I've been there and man I feel for those guys. Getting crap dumped on them that has no documentation, architectural designs that were pulled from someones backside in the heat of the moment of getting it done quickly, inconsistent standards being applied, wheel reinvention, etc.
On the bright site, budgets for support are much higher, because application is in production and generates real money. So even though it's hard to support messy code, there is no risk that all these efforts will be wasted on something useless.
Prototype is really a dirty word in programming, but there's no reason you can't produce a small, well written app which does some of what's needed and build from there.
As opposed to a crappily written ball of mud which does most of what's needed, but poorly.
> but there's no reason you can't produce a small, well written app which does some of what's needed and build from there.
Depends on where you work, I guess. Part of my personal hell right now is our architecture (which was written "elsewhere" and is given to us from on high) prevents such luxuries. It's an all-in affair, and deviation is met with retribution. Unit testing is nigh impossible with it, so I try to unit test the best I can of MY stuff that gloms onto its bulbous, massive exterior.
I agree with your point; don't get me wrong; but sometimes there are external forces at play that make the "right way to develop" hard to impossible.
The 'right' answer there is to system test first (eg. set up a dev system and run things like selenium against it) and then you can start to pull stuff out into libraries and unit test it.
The standard way around the 'luxury' attitude is to start with fixing downtime, and grow from there.
I think that one of the major problems people have when giving estimates is that developers realize there is a lot of uncertainty add in all of the "'making $#@! up' fudge factors, technical hurdles, cogitation and exploration time, etc.", and then managers mentally strip it out.
The frustrating thing is that it means the uncertainty gets lost, and people just end of with a number, which managers may treat a fact, not an estimate. That's why I think it's a really good idea to give a range including both a low and a high estimate so you can let people know: "If everything goes right, it might take two hours", but warn them that: "It could take 4 days". Once people have that kind of information, they can make much more resilient plans.
That's why we always tell our CEO that it will take 2-4 hours or 4h-5d, and let me tell you, he appreciates knowing when we don't know, because a layer of false certainty is removed.
Actually, no. It's a win in that case too, as I mentioned. The customer gets something to complain about, so they win by appearing thorough and tough on the vendor, and the manager gets something to fix, and wins by appearing reactive to customer needs/wishes. I've seen this scenario so many times over my last 25 years of doing this. To date, I've only had 1 manager admit that's what's actually taking place.
Time estimations is an industrial way of thinking applied to a post-industrial world.
In the post industrial world time isn't the problem but rather project definition and scoping.
In the industrial world the problem was already solved (machine was built, market often established and output depended on a few factors that could be adjusted. Need more output add more of X)
In the post industrial world every project is about problem solving and scoping.
To put it into comparison.
If we apply post-industrial thinking to an industrial world. It means that each time a product needed to be done if not the factory, then the machines would have to be developed.
It will take many many years before time estimation will die, but it will happen.
So things like prototyping, UML, use cases, and Agile were not a "serious attempt" to answer that? I think it's also unwise to tell your manager "it'll be done when it's done."
Largely those are things done by developers, not by the business. Perhaps in cases where the business is on board with them (eg. Agile and rapid iterations), but mostly they'd rather handwave and hope for the best.
That is hard to say and it would be a book worthy to explain what could come next. All I know is that it is unsustainable.
The complexity is simply too high and it's not getting better. One of the reasons I think why you see the fail fast movement be so successful.
Once you accept that failure is part of the process, once you abandon the "zero mistake" policy that many large organizations instill internally and externally you will begin to approach projects differently.
The truth is that "zero mistake" organizations make as many mistakes as everyone else, they just have the financial strength to ignore them as long as economy of scale works in their favor.
I could write forever about projects that went wrong not because the developers where bad but because the premise that fuels product development is broken.
I blame primarily business schools and large parts of academia for this. But it could extend all the way into the way the stock market is structured.
If you buy my premise that post-industrial is different than industrial age. That project definition is primary and time is secondary today. Then it does put some doubt at least in me about whether the stock markets focus on growth and Q's is sustainable.
Nature seems to be doing a good job as pacing various processes. It takes nine months to give birth to a child. One cell at a time. But the process is ongoing.
Nature is the ultimate continues deployment strategy.
It sounds like you've thought deeply about this - have you written more on your ideas here elsewhere? Would be interested to hear how this would work practically as well as your ideas on how the focus on perpetual growth hinders successful project delivery.
I'd love to see you write more on this as well. You've clearly thought about it prior to this conversation in a way that I don't think a lot of us have.
I think you will find reading Hayek and Coase to be rewarding.
Hayek introduced the idea that no one individual planner can make flawless plans because the information to do so is widely dispersed; instead success or failure in the market gives rapid feedback about how to allocate resources. Indeed the market is a discovery process that unearths this information in a way that a planner never could[1].
Coase asked the question: if this is so, how do large firms emerge? He identified the cost of transactions to be the key. The higher the transaction costs, the higher the cost of loosely coupled economies. The size of the firm is thus based on the relative costs of transactions (searching, identifying, validating, negotiating etc) vs the inevitable waste caused by planning.
In essence, large companies are islands of command economics in the sea of the market.
[1] In a related argument, Mises said that even if a planner could know all the variables the resulting problem would simply be too large to solve rationally.
Instead of replacing estimation, I would say that you should embrace the problem in estimates - uncertainty.
Convey the uncertainty you have about a task to those you work with, and then you can start to factor in the risks of uncertain tasks.
I build project management software for a living at LiquidPlanner. Everything we do be it development, design, or marketing is based on ranged estimates. We don't always get the estimate right, but that's the beauty of a range, I it takes into account the fact that you will miss it some of the time.
The other thing that can replace an estimate is... lots of estimates. If you're planning something, update your estimates as you gain more knowledge about the problem.
I used to work for big oil, and they would have 100-200 million dollar projects. Not software projects, actually building plants. Similar to software development, these were basically prototypes. Yes a lot of the technology was known, but they weren't exactly building cookie-cutter houses.
So the first round of estimates would be something crazy like +/- 80%, both money and timeframe.
If that seemed good, then they'ed fork over the money to get more specifications and more details, and come up with a new estimate. Maybe +/- 50%. Then they'd re-evaluate the viability of the project. And do that a few more times, spending more money and time each iteration, before they actually committed to the project or dumped it. At that point, the estimates were quite accurate.
So basically, the software version. How long will that feature take?
An experienced developer can spitball an answer. About a week. Then they start working on it. After a day-and-a-half, they're going to have a better idea if that's actually a three day project or a two-weeker or if it ain't gonna happen.
Or maybe they need a day-or-two or a week to do a spike to be able to provide that estimate. All well and good if you let them do the spike. Not so good if you demand an estimate when the developer has clearly said he has no idea.
I absolutely agree that interval arithmetic is the best way to deal with this, accountants and their budget systems always require estimates to be reduced to fix figures.
What's needed may just be to solicit a wide confidence interval (e.g. "this task will take between 20 minutes and 1 month, and I'm right 90% of the time recently when making such estimates"). You can drop the lower-bound part, probably.
People absolutely should be able to provide a rough estimated amount of effort for a task. The trouble is in using a single point to describe the whole probability distribution. You may be right, in that nobody seems to have a great way of soliciting a probability distribution (or even a single probability) that makes sense. Something like a 70% confidence interval bracketing the amount of effort would be useful, but not sufficient.
I also agree that even if people could describe their beliefs about the required effort rigorously, you have to wonder how much planning/analysis they should spend trying to come up with a tight estimate. An 'outside view' - http://wiki.lesswrong.com/wiki/Outside_view - could give something reasonable in cases that aren't too novel.
I think estimates will never become useless. It's just that we may decide to replace them with "time by which we'll be late only 1/6 of the time", and provide feedback and incentives for people to correct this estimate so that it's, for someone who's experienced in the domain, eventually relatively unbiased.
Actual effort to complete can be quite surprising, but effort spent per day also varies tremendously. For example, when things are going well, there may be a burst of joyful effort.
When I needed to estimate a new project at my previous job I used to pretty much always say "3 months". I made sure clients knew I was talking about a MVP even if they were thinking about making something like Microsoft Excel in-house. Then we would just build on top of that, one block at a time, never more than 3 months for each block.
I'd say that people can't estimate time because it's an acquired skill and most people don't try to acquire it.
When I managed a team, I had people put their initial estimates in the tracking tool, and when closing out the item they'd also fill in how long it actually took. Then, some of the report generation tools would let them see how accurate they were. There were no review metrics associated with being accurate, but I found that within a couple of months in a new area, individuals started getting much more accurate without just lamely padding out schedules (their teammates would have called them out, and even I wasn't pointy-haired enough to be fooled that easily). An unexpected bonus is that people also got a lot better at describing their work and investigating the risky bits before throwing something onto the schedule for the sprint.
But, we didn't do this for purely investigative or experimental work (i.e. "try out a new immutable text region design in the editor"). We'd just timebox work like that and evaluate progress to decide whether to keep going or not.
I can't stand the rubbish advise to 'just double the estimate'. It just means you'll get an estimate 2^n times the size of the original guess from the guy n steps away from you.
133 comments
[ 2.3 ms ] story [ 117 ms ] threadMy post is definitely not suitable for most management, but in a discussion with a bunch of fellow geeks, it did fit in with the discussion.
Explaining this to management/clients is not easy. It's a problem that needs tackling, but it's a separate problem.
Also if anyone wants some more good books to read we have a list of some of our favorite project management books at http://www.liquidplanner.com/recommended-reading.
IMHO - it's usually due to the lack of "up front planning" by Managers / Analysts.
If you compare building software to more general "building endeavors" (for example building the Olympic stadium) then, IMHO, developers are really the "construction workers" (the people actually doing the work). However no intelligent Olympic committee would countenance starting work without (for example) an architect making an incredibly detailed plan (and even a working replica model).
IMHO the problem we suffer from as developers is most often diving in too fast because: a) we're the construction workers b) when we do use Architects they are "construction workers too" (as opposed to being a trained with 'different but complimentary skills - it would be hard to imagine Sir Norman Foster laying bricks)
In my opinion, software development is exactly a creative process. Many times, you have to come up with a creative solution that fits within certain constraints. Also, in the beginning usually only a vague image is known. The artist (developer) has to fill this in and make sure it is what the client wants.
Also, many creative works are collaborative efforts as well these days. I'm not sure how that changes anything. If anything, it makes estimates harder.
Art can break rules and you can sell weird color splotches for millions, but the output of your /dev/random will always be worthless.
But I suppose it's something in between art and engineering.
With regard to "sell weird color splotches for millions" - Isn't that the same with code? You can sell a simple game or app to millions of people and become rich. The amount of money you earn doesn't have any connection to complexity.
Artists might like to experiment with different styles, so might coders.
Also, it's not like most artists try to sell people ugly things. That's also a very small niche. Most of them simply work for companies and design the things that they are told you. Just like coders.
We could debate about this for ages, but it seems very clear that there is both art and engineering involved.
Of course there's also people who think that art is only art if it doesn't do anything and isn't "needed". This also creates problems (for example, can a movie be art? if yes what of the "jobs" needed to create the movie are art? who are the artists?)
Some of the benefits I find are:
1. Getting into the pessimistic mindset helps produce better estimates
2. Tasks with a large spread are things you aren't really clear about, and are an obvious target to be attempted first
3. Giving a range of values to the person asking for the estimate seems to help them remember it is an estimate.
If you give an estimate that's too far out, it'll be outside your client's planning scale. That means it's so far out, she doesn't know what will happen by then, she won't do the project. In my experience, this is scale is roughly 9-12 months, unless you're dealing with a MegaCorp.
So most estimates end up at 6-9 months, which then slip into 9-12 territory, and hopefully not much further =)
In my experience, coding time is usually easy to estimate, and that's what most estimates coming from developers should be taken to estimate. But then you have integration (nothing works), testing (everything is broken), and then realizing what the client meant when she said X (solved the wrong problem, optimized for wrong features), those are the tough ones.
You should take the estimate for coding time (coming from the developer), and multiply that at least by 2x. Of course, going back to my original point, that estimate may be outside the "human timescale", so you may not be able to tell the client that...
So, to answer your original question, developers can estimate primary development time up-front pretty well, but they suck at estimating integration, testing and shipping time.
A related thing I noticed is that if you ask a developer during the late(r) stages of the project when she'll be finished with the "next stable version", the answer usually hovers around "in 2 weeks", even if the actual answer is more like 3 months. The developer isn't lying, she genuinely believes that it can be done in 2 weeks.
That's why you want experienced engineers and not fresh college grads leading teams and giving estimates. They've gone through these experiences, have a lot of soft data metrics in their heads, and can come up with good multiplicative factors. I'd be curious if orgs like Google have a database of project data to aid planning.
This is where having good historical records is useful. If I have done 2 similar tasks before and they took a month each, then a 2-week estimate has no validity.
As far as general code estimation goes, lines of code and complexity taken together form a pretty good metric. If I estimate I have to write 50 lines of high-performance multithreading code, I might say it will take 8 hours. If it's 50 lines of fizzbuzz, estimate drops to 1 hour.
But like you say, that's just coding (and unit test). The full task probably requires (1) getting or understanding requirements, (2) at least some cursory thought about design and (2a) having a hallway review with somene if you're not sure. Then when we're done with code, unit test (don't forget to estimate how long it will take to create, debug and run the tests) and Code Review (how many reviewers do you need, how long will it take them to get around to inspecting your code, etc), someone has to estimate (4) how long the test team will take to write test cases and do the system test of the new feature, (5) tech docs has to update the user manuals, (6) sysadmin or the Release Master estimates how long deployment will take.
In short, at least in the environment I work in, that 50 lines of code that took eight hours to write, will easily require an entire week before any customer can see it. In my experience, it's often not so much that the developer didn't estimate accurately how long coding would take, it's that the developer never thought about all the ancillary tasks that have to be done (code review, unit test, documenting the design changes, etc), so they were not in the estimate.
Furthermore, continuous process improvement and monitoring is not done enough even in organizations that declare achievement of higher levels of CMMI. Our team had subsystems that overran their initial estimates by over 100%, causing nearly a year in delay. However, there were basically no penalties or major process changes despite the fact that multiple subsystems overran estimates multiple times. Process professionals, engineers, and managers are simply not aggressive in tackling these problems.
In addition, estimates tend to come from individuals. I imagine that a more collaborative team-based estimation approach would be better, factoring different levels of experience and sharing the burden of making estimations realistic. Also, recorded estimates need to be coupled with recorded justification.
i am great at estimating - unfortunately i often go over because what i heard was being asked for is subtly different to what the client thought they were asking for
[La Forge goes back to work; Scotty follows slowly]
Scotty: Do you mind a little advice? Starfleet captains are like children. They want everything right now and they want it their way. But the secret is to give them only what they need, not what they want.
Lt. Commander Geordi La Forge: Yeah, well, I told the Captain I'd have this analysis done in an hour.
Scotty: How long will it really take?
Lt. Commander Geordi La Forge: An hour!
Scotty: Oh, you didn't tell him how long it would really take, did ya?
Lt. Commander Geordi La Forge: Well, of course I did.
Scotty: Oh, laddie. You've got a lot to learn if you want people to think of you as a miracle worker.
Having worked on scrum teams before, I find its a good structured way to have a dialog about how much work is left, and how long you expect it to take.
So rather than saying an hour. You would say this is as easy as changing a diaper, or its as hard as navigating through an asteroid belt at the speed of light ( having or not having done it ).
I would appreciate it if managers ask:
Have you done this before? (yes, no)
How similar is this to something you have done before? ( very similar, kinda similar, totally new )
What concerns you about the task, or what risks do you see coming your way? ( detailed answer )
What can I do to get you the resources you need to help you deal with the risks you can foresee now?
It makes no sense if only the developers and designers are doing scrum when the rest of the company is expecting deadline deliveries.
The real problem is thinking of "estimates" as "promises" when really they're rough guesses. If you go around holding people to a guess as if it were some kind of gospel biblical contract then yes, they will lie their ass off to protect themselves. Combine that with the fact that most organizations attempt to control the platform used, computers used, monitor size, operating system, IDE, editors, revision control tools, languages, documentation systems, testing methodology, meeting requirements, pairing, and nearly everything they can, and you start to see the real problem is....
managers who don't know programming, motherfucker. :-)
Manager: "How quick can you get this done?"
MFer: "3months (assuming price is no object)"
Two weeks pass...
Manager: "Our budgets changed. How much $$ can we trim out of this project?"
MFer: "We can cut it in half, but that will triple our timeline."
Manager: "cut in half! Great!"
3months pass:
Manager: "Hey why are we behind? The big guys want an explanation why you aren't meeting your targets!"
Of course, then you run into some of the same situations as described.
shrug
http://www.itworld.com/waste-management-sues-sap-080327
"At that meeting, SAP AG executives and engineers represented that the software was a mature solution and conducted a demonstration consisting of what they represented was the actual SAP Waste and Recycling software," the complaint states. The company later discovered that the software was a "mock-up version of that software intended to deceive Waste Management," according to the complaint. SAP has admitted to this in "internal documents," the complaint states.
"From the beginning, SAP assured Waste Management that its software was an 'out-of-the-box' solution that would meet Waste Management's needs without any customization or enhancements," the statement reads. "Unfortunately, Waste Management ultimately learned that these representations were not true."
The board wants promises because they want to know when they get paid.
If you find it stressful that people are turning your vague estimate into a promise, imagine having to make promises on other people's vague estimates.
I guess some "agile" methodologies propose a similiar approach, but we didn't really follow any specific one, just figured out what works best as we moved along and kept the process as lean as possible.
Any prescriptive agile methodologies (first, do X. then, do Y) are really antithetical to agile. True agile processes are exactly what his own startup did. They used what worked. They implemented a practice (complexity points) that was relevant to their principles (accurate estimation). Of course, you should also be tweaking your process as you go along.
I'm not a developer but I can confidently say that it's not just developers - NO ONE can estimate time when it comes to creative work. I use the term creative loosely, it could be anything from developing a business strategy, to crafting a story, to art directing an original visual, to writing code for a novel problem.
The problem is the same one that makes it difficult to say "when will we have a cure for Parkinson's," "when will we have fusion power," "when will we discover a viable alternative energy"? You often see estimates like 20 years, 40 years, 50 years - but properly interpreted, these aren't actual estimates of the amount of time it will take, but rather like a proxy for a much more probabilistic, uncertainty-oriented notion.
http://www.youtube.com/watch?v=u6XAPnuFjJc
Once you communicate that, you can uncover a lot of potential problems and misunderstandings. For instance just the other day my co-worker was asked to estimate a task that seemed to our boss to be pretty small. He estimated it as being 2-8 days, it wasn't until they actually started talking about _why_, that they both agreed on the requirements. The resulting task will probably take 1-3 hours.
So sure, as a developer learns more they may estimate tasks as being longer, but communicating your assumptions is more important than throwing out huge estimates to cover your ass ;)
> It's not just developers that are bad with estimates either. Everyone at some point is just winging it because it's something they've never done before and won't be able to successfully make a judgement until they have.
When my manager asks, "How long will this take," about half the time my honest answer is, "I don't know."
One of the most important things I've learned about estimating is first asking how accurate an estimate is needed. Often the person just wants to know "can I have it tomorrow, or will it take 3 months?"
The OTHER side is that, at least given my experience in the financial software domain, managers don't WANT accurate estimates. They absolutely abhor them. And developers are punished for giving them. So they aren't given.
An accurate estimate, such that one can be made, is usually around a 70%-80% confidence interval. It includes many specifically unforeseen, but generally known issues such as problems with the environment, lacking specifications and time required to get them, including "making $#@! up" fudge factors, technical hurdles, cogitation and exploration time, etc.
But managers can't hear that. All they want to hear is something they can sell to their superiors, which is often the customer. An accurate estimate is almost always going to be larger than that, so they won't accept it. So, the development staff is forced to skimp on quality or features to make an artificial date. But that's ok! Why?
Well, it's partially an organization's willingness to accept "there's never enough time to do it right, but there's always enough time to do it twice" (or more), but that's not even half of the issue. The larger part is that NO ONE WANTS IT RIGHT THE FIRST TIME. I've seen this time after time.
It's a win/win to provide sub-par product. Why? It makes a manager look great to give low estimates (win), hold the developers to that, and deliver a product with less-than-promised quality, or scope, or both. That provides customers something to gripe about (which EVERY customer wants; it makes them appear "tough" or "thorough"; win), and it gives the managers something to "fix" and appear reactive to the customers' needs (win).
Who's holding the bag for all this win? The devs, and support. There are inevitable promises of being able to go back and fix things, but that never happens; once something is in production, no matter how crappy, it has the almighty momentum. If it's "working", almost no matter how fragile it is, there's no appetite to change something that works so that it can work better. That provides no revenue, and does impart considerable risk (with limited or no QA, having moved on to the next thing) and/or cost (keeping your QA around to regress fixes).
Although that mindset infuriates me, I don't honestly know that it's not the best way. It seems to have evolved, and companies that do it seem to do ok, so maybe it's the Darwinian process at work.
As opposed to a crappily written ball of mud which does most of what's needed, but poorly.
Depends on where you work, I guess. Part of my personal hell right now is our architecture (which was written "elsewhere" and is given to us from on high) prevents such luxuries. It's an all-in affair, and deviation is met with retribution. Unit testing is nigh impossible with it, so I try to unit test the best I can of MY stuff that gloms onto its bulbous, massive exterior.
I agree with your point; don't get me wrong; but sometimes there are external forces at play that make the "right way to develop" hard to impossible.
The standard way around the 'luxury' attitude is to start with fixing downtime, and grow from there.
The frustrating thing is that it means the uncertainty gets lost, and people just end of with a number, which managers may treat a fact, not an estimate. That's why I think it's a really good idea to give a range including both a low and a high estimate so you can let people know: "If everything goes right, it might take two hours", but warn them that: "It could take 4 days". Once people have that kind of information, they can make much more resilient plans.
That's why we always tell our CEO that it will take 2-4 hours or 4h-5d, and let me tell you, he appreciates knowing when we don't know, because a layer of false certainty is removed.
Only if sub-par means limited scope. If sub-par means buggy, you might damage your reputation.
In the post industrial world time isn't the problem but rather project definition and scoping.
In the industrial world the problem was already solved (machine was built, market often established and output depended on a few factors that could be adjusted. Need more output add more of X)
In the post industrial world every project is about problem solving and scoping.
To put it into comparison.
If we apply post-industrial thinking to an industrial world. It means that each time a product needed to be done if not the factory, then the machines would have to be developed.
It will take many many years before time estimation will die, but it will happen.
"You want it know, or when it's done?"
My personal experience is that most deadlines are primarily perceptual.
They follow the principle of "The best way to predict the future is to invent it"
so yes you are totally right about that.
The complexity is simply too high and it's not getting better. One of the reasons I think why you see the fail fast movement be so successful.
Once you accept that failure is part of the process, once you abandon the "zero mistake" policy that many large organizations instill internally and externally you will begin to approach projects differently.
The truth is that "zero mistake" organizations make as many mistakes as everyone else, they just have the financial strength to ignore them as long as economy of scale works in their favor.
I could write forever about projects that went wrong not because the developers where bad but because the premise that fuels product development is broken.
I blame primarily business schools and large parts of academia for this. But it could extend all the way into the way the stock market is structured.
If you buy my premise that post-industrial is different than industrial age. That project definition is primary and time is secondary today. Then it does put some doubt at least in me about whether the stock markets focus on growth and Q's is sustainable.
Nature seems to be doing a good job as pacing various processes. It takes nine months to give birth to a child. One cell at a time. But the process is ongoing.
Nature is the ultimate continues deployment strategy.
Edit: Fixed lame false modesty in the end!
Hayek introduced the idea that no one individual planner can make flawless plans because the information to do so is widely dispersed; instead success or failure in the market gives rapid feedback about how to allocate resources. Indeed the market is a discovery process that unearths this information in a way that a planner never could[1].
Coase asked the question: if this is so, how do large firms emerge? He identified the cost of transactions to be the key. The higher the transaction costs, the higher the cost of loosely coupled economies. The size of the firm is thus based on the relative costs of transactions (searching, identifying, validating, negotiating etc) vs the inevitable waste caused by planning.
In essence, large companies are islands of command economics in the sea of the market.
[1] In a related argument, Mises said that even if a planner could know all the variables the resulting problem would simply be too large to solve rationally.
Convey the uncertainty you have about a task to those you work with, and then you can start to factor in the risks of uncertain tasks.
I build project management software for a living at LiquidPlanner. Everything we do be it development, design, or marketing is based on ranged estimates. We don't always get the estimate right, but that's the beauty of a range, I it takes into account the fact that you will miss it some of the time.
The other thing that can replace an estimate is... lots of estimates. If you're planning something, update your estimates as you gain more knowledge about the problem.
I used to work for big oil, and they would have 100-200 million dollar projects. Not software projects, actually building plants. Similar to software development, these were basically prototypes. Yes a lot of the technology was known, but they weren't exactly building cookie-cutter houses.
So the first round of estimates would be something crazy like +/- 80%, both money and timeframe.
If that seemed good, then they'ed fork over the money to get more specifications and more details, and come up with a new estimate. Maybe +/- 50%. Then they'd re-evaluate the viability of the project. And do that a few more times, spending more money and time each iteration, before they actually committed to the project or dumped it. At that point, the estimates were quite accurate.
So basically, the software version. How long will that feature take?
An experienced developer can spitball an answer. About a week. Then they start working on it. After a day-and-a-half, they're going to have a better idea if that's actually a three day project or a two-weeker or if it ain't gonna happen.
Or maybe they need a day-or-two or a week to do a spike to be able to provide that estimate. All well and good if you let them do the spike. Not so good if you demand an estimate when the developer has clearly said he has no idea.
People absolutely should be able to provide a rough estimated amount of effort for a task. The trouble is in using a single point to describe the whole probability distribution. You may be right, in that nobody seems to have a great way of soliciting a probability distribution (or even a single probability) that makes sense. Something like a 70% confidence interval bracketing the amount of effort would be useful, but not sufficient.
I also agree that even if people could describe their beliefs about the required effort rigorously, you have to wonder how much planning/analysis they should spend trying to come up with a tight estimate. An 'outside view' - http://wiki.lesswrong.com/wiki/Outside_view - could give something reasonable in cases that aren't too novel.
I think estimates will never become useless. It's just that we may decide to replace them with "time by which we'll be late only 1/6 of the time", and provide feedback and incentives for people to correct this estimate so that it's, for someone who's experienced in the domain, eventually relatively unbiased.
[1] http://www.mountaingoatsoftware.com/books/1-agile-estimating...
When I managed a team, I had people put their initial estimates in the tracking tool, and when closing out the item they'd also fill in how long it actually took. Then, some of the report generation tools would let them see how accurate they were. There were no review metrics associated with being accurate, but I found that within a couple of months in a new area, individuals started getting much more accurate without just lamely padding out schedules (their teammates would have called them out, and even I wasn't pointy-haired enough to be fooled that easily). An unexpected bonus is that people also got a lot better at describing their work and investigating the risky bits before throwing something onto the schedule for the sprint.
But, we didn't do this for purely investigative or experimental work (i.e. "try out a new immutable text region design in the editor"). We'd just timebox work like that and evaluate progress to decide whether to keep going or not.
Excellent idea. We do this and it really works well to see how far off your estimates were. Feedback is essential to controlling any process!