Once upon a time, I offered cost estimates on a per-feature basis. It didn't take a lot of time for this strategy to bite me. Over the last few years I have established more stable contracts and now I tend to give estimates for larger blocks of work. My estimates per feature are still often wrong, but this risk is mitigated by spreading the estimated effort across a large set of features.
I usually provide an estimate of the time it would take me to come up for an estimate of the whole project.
Pushing for a 2 level estimation is definitely not easy, and if the person on the other side of the negotiation refuses this, then I ask him/her to stop playing the "Price is right" and spit out his time requirement already.
It is agreeable to be reminded of the fact that psychology influences us and our craft disciplines. We're so occupied with math, algorithms and automation that this may slip our minds.
There is a popular psychology book, “Games People Play” by Eric Berne that shows and explains many of those game mechanics. Everybody uses them, not just PMs and bit crunchers.
This book! A great introduction to transactional analysis and a very useful tool for disarming and defusing interpersonal strife.
This book helped me figure out that, in personal relationships especially, I could raise the discussion by one level and talk about the game I was playing in an effort to figure out why and get somewhere useful, emotionally. As in, "I think I'm just playing 'Why Don't You - Yes, But'. I need you to just listen to me while I vent."
Can't recommend enough.
EDIT: I should say that the edition I read seemed a bit sexist? I just ordered the new edition so we'll see if any of that text changed.
"...It is our belief that over the 30 plus years of commercial computing has developed a series of sophisticated political games that have become a replacement for estimation as a formal process..."
No. You are confusing the natural inclination for people to negotiate with some kind of slippage into "politics" (The word "politics" is most often used to describe "system of people stuff that I do not understand and that does not make sense to me")
For thousands of years, people have had a natural way of dealing with commitments where incomplete information is present. It's called "negotiation", and it's done by various techniques and tactics such that when all is done both sides are happy. We've also had ways of numerically determining tangible things. It's called math or science.
These are completely different things. What I see all the time is scientific, technical people flummoxed at how business-minded people are treating them. And the reverse is equally true. The reason is that both sides are working inside a completely different universe.
I like this article because it shows how multiparty negotiations can lead to disaster, but I do not like it because it confuses the true nature what's going on.
Politics can be understood in a narrow sense (members of political parties performing kabuki dances suggesting a form of governance), or in a broader sense, as what happens when multiple humans try to make a decision.
I tend to think of politics in the broader sense, and suspect that the author of the piece does as well. It's the process of negotiation between people with different spheres of influence, when a single decision has to be made. Client negotiations are absolutely political.
This is a nice taxonomy of estimation games, but stopping the games won't address the root problem: Software development is complex enough to be fundamentally unpredictable. The games are just about hedging against people not wanting to hear that.
I think you're painting it a bit too black. Having both sides agree to the fact that it's an estimate (and not an exact prediction) does help. And if you make the client understand the contingencies and unknown factors, you may get around the expectation to make an accurate prediction — which nobody can give anyway, except when doing the same thing the umpteenth time. Communicating often on progress that you actually do make can still give the client a good sense of confidence and control on what's happening.
> Software development is complex enough to be
> fundamentally unpredictable.
This is false.
The weather is an extremely complex system, yet we use weather forecasts all the time. What's different between weather and software is that everyone knows that weather is extremely complex, and we account for that when we think about the weather in the future. ("Maybe I'll carry an umbrella today.")
The equities market is an extremely complex system, yet we use market forecasts all the time. What's different between equities market and software is that everyone knows that equities market is extremely complex, and we account for that when we think about the price of an equity in the future. ("Maybe I'll diversify my investments.")
Forecasting the completion date of a software project is no more complex[1] than predicting the weather or the future price of an equity. In no way is it "fundamentally unpredictable." What's different is that people don't regard project estimates as, well, _estimates_.
> "What's different between weather and software is that everyone knows that weather is extremely complex, and we account for that when we think about the weather in the future."
Weather is governed by impersonal physical forces which can be described by mathematics. Software is built by groups of people who most defitely cannot be modeled as mathematical entities. Weather is a bad analogy here.
> "The equities market is an extremely complex system, yet we use market forecasts all the time. What's different between equities market and software is that everyone knows that equities market is extremely complex, and we account for that when we think about the price of an equity in the future. ("Maybe I'll diversify my investments.")"
Those market foreasts are often wrong. Even a company's own forecasts are frequently wrong, despite having the optimal perspective from which to make those forecasts.
Furthermore, diversifying investments is easy to do because the price mechanism makes those investments fungible (in a liquid market, anyway). This is not the case with software. Each piece of software is unique. You may "diversify" your software by investing in multiple projects, but ultimately you have a business need for a piece of software which does task X. "Diversifying" by funding software that does tasks Y and Z will not help you if the software "investment" in task X fails. "Diversifying" in three software projects, all intended to do task X, will roughly triple your costs for doing X. Again, equities are a bad analogy.
Software development is a process which can't be easily summarized by analogies to existing systems. In fact, the term "software development" can describe many different systems.
The real reason schedule estimation in software development is so hard is because, regardless of which system of software development is used, the problem of simultaneously optimizing to maximise a desired quality while minimizing the cost/schedule is inherently hard in a mathematical sense. Changes which promote one oppose the other, and the sensitivity of a software development system to unknown and unpredictable future changes cannot, obviously, be predicted in advance.
Your point about hedging one's bets is valid, but there are costs to doing so which are not as simple as carrying an umbrella or splitting up a pile of money across multiple equities.
> Software is built by groups of people who most defitely
> cannot be modeled as mathematical entities.
Putting aside whether mathematics is up to modeling human behavior, why does a project schedule estimate need to be modeled using people?
> Weather is a bad analogy here.
I was using weather as an example of complex things that can be estimated, not as an analogy for software development itself. Sorry if I wasn't clear about this.
> Those market forecasts are often wrong.
Useful forecasts/estimates are seldom binary, and "wrong" versus "right" is a bad way to evaluate their usefulness. Estimates have both precision and accuracy. For example if you want perfect precision (as in the wrong/right example), you may in turn get a very low accuracy, i.e. the estimate will often be incorrect. If, on the other hand, you can live with less precision, you can often get higher accuracy.
> The real reason schedule estimation in software
> development is so hard is because, regardless of which
> system of software development is used, the problem of
> simultaneously optimizing to maximise a desired quality
> while minimizing the cost/schedule is inherently hard in
> a mathematical sense. Changes which promote one oppose
> the other, and the sensitivity of a software development
> system to unknown and unpredictable future changes
> cannot, obviously, be predicted in advance.
Again, this is true only if you need very high precision and very high accuracy. If you can live with lower accuracy and precision, estimation becomes quicker, easier and cheaper.
So I stand by my original assertion that software estimation is not impossible.
(edit: To remain consistent, I should say that software schedules are certainly predictable, just not perfectly predictable. Most teams can get by just fine with good enough predictions.)
"...It is our belief that over the 30 plus years of commercial computing has developed a series of sophisticated political games that have become a replacement for estimation as a formal process..."
If there was a planning process that was consistently reliable for new, complex software I think people would use it. And I think the adoption of agile methodologies in software development is an example of people trying to implement an improved planning process. Still, there is a lot of uncertainty.
So people negotiate, incentivize, and test conviction.. I guess that is what the author means by politics. Those same games happen in most complcated transactions and relationships. I think some programmers think this stuff happens to them because they are deeply misunderstood. But it happens to building contractors, real estate agents, lawyers, etc. too.
> If there was a planning process that was consistently reliable for new, complex software I think people would use it. And I think the adoption of agile methodologies in software development is an example of people trying to implement an improved planning process.
There are such processes. But they require understanding of the problem, which takes time. I am not sure "agile" is really attempt at an improved planning process, it's at best not doing planning and just marching onto the problem, seeing what happens at regular intervals in the hope of better understanding it, and at worst, things like "planning poker" where you put a bunch of people into a room and let everybody make a (possibly educated but usually not very much) guess from the top of their heads.
This is an excellent article. It's a great synopsis of what's really behind those estimates, whether in naked form or couched in that "story points" bullshit of the "Agile" cult. I fucking hate all of it.
Here's my take. Programmers don't underestimate because they're afraid of getting fired or yelled at. In this market, a good developer is unlikely to be fired for honestly estimating. Yelled at, yes. Fired in worse economic circumstances (e.g. 2003)? Possibly. Fired now? Doubtful. Rather, it's about a desire not to be jerked around. Implicit in the demand for an estimate is the threat to jerk the employee around-- either imposing stupid new micromanagement ("Agile") or changing project priorities. No one wants to work for the boss who jerks her around and never lets her finish anything, because soon enough she gets to 3 years and while she's fixed bugs and added a few features, she doesn't have the coherent project that would make the case for promotion or transfer to her target group... because her boss kept jerking her away from projects whenever his hindbrain misfired, he interpreted it as the "this is taking too damn long" impulse, he got impatient, and pulled support. It's a fight-or-flight reaction in the boss when that happens, and managers tend to favor flight because fighters tend to get themselves fired. If you work for a hair trigger "flighter", you can very easily get to 3 or even 10 years without ever actually finishing a project. And flighters love estimates.
A good middle manager isn't one who exacts and reports reliable estimates. A good manager goes out, wines and dines the executives, and gets sufficient credibility for his people that no one ever has to give an estimate.
Anyway, programmers give misleadingly optimistic estimates-- not dishonest, just aggressively optimistic-- because they want to be left alone so they can actually produce something and move along to a better project. This is what "Agile" doesn't get. The thing that motivates us is delivering a major project that gives us enough credibility that we never have to make estimates again; not delivering small cantrips that anyone could do, but in record time. I think that "Agile" provides some structure for junior programmers, but it doesn't provide an exit. Under the Agile Ideology, even the senior programmers in R&D and architectural rules have to structure their work in terms of these dumbass "iterations". Implicit to Agile is the absence of personal progress for an engineer: no matter what you do, you'll always be working on feature-level "user stories" instead of real projects where you call the shots, and you'll always be subordinate to the business.
I've been doing this for almost 10 years and I'm good at it, so what being asked for an estimate communicates to me is that whatever I'm doing isn't really important. I will do what is necessary to meet a real, hard deadline (they're rare, but they exist) such as a bid deadline for a government agency (where people can go to jail for corruption if they grant an exception to someone who's a day late) but the silly deadlines that business usually creates are just resource limits, and (sorry but) I'm too good to work on something so unimportant that I'm not even trusted with a month or few of my own time on it. If you won't accept delivery on my time, then it's not important enough to be doing in the first place, because I will work very hard to make sure it's done reasonably quickly and well, and if you don't trust me on that much, then you should just fire me.
In other words, the difference between "4 weeks" and "2 weeks" is going to mean that I either don't get to do it or have to relegate it to extra hours, then (unless there is a hard deadline, and not some silly target like "Q3 KPIs") it's ju...
I assume you were down-voted for going on a tear about Agile methods. I can't speak to that since I've never been in an Agile shop (read about it plenty).
But I think what you have written is applicable to most developers in general, particularly this gem, which I loved:
"The thing that motivates us is delivering a
major project that gives us enough credibility
that we never have to make estimates again..."
I also agree that the article was wonderful. I laughed out loud several times because, like all of the best comedic material, it is firmly grounded in the painful truth.
In the end, developers really do just want to build great stuff and I think most of us _are_ acutely aware of the realities of costs and time in real business. I sympathize with management's desire for estimates (and even the various methods to somehow get a reign on development). But the unfortunate truth is that the attempts often only serve to irritate and chain down people whose only desire was to work like hell making quality software in the first place.
Who says putting time pressure on programmers is more effective? What I have seen is that it leads to delivering low quality work, followed by a lot of "putting out fires" work.
When you work with motivated engineers (most programmers are in my experience), they set their own pressure and deliver quality work.
If you have a project supervisor that doesn't understand how quality software is made, leave. Putting in 80-100 hours? Nuts! Your estimates will slip even further when you do that.
There have been a number of posts recently about software estimates, including an emerging hashtag #noestimates.
I'd say that developers do pretty clearly prefer working in an environment where they aren't required to provide estimates and meet arbitrary deadlines (nobody likes hard deadlines either, but there's a difference between a window to launch a rocket to study halley's comet and a manager who figured that saying "march" rather than "may" might get the team to work a little harder).
This does't mean that developers don't like to work hard or produce good work, I think they just greatly prefer to take on a task, work diligently at it, and show progress rather than be pressed for estimates based on limited knowledge that quickly morph into hard deadlines before the problem is adequately understood.
Yeah, I'm stating the obvious. But thing is, it isn't a complete pipe dream. I do know developers who have quite a bit of autonomy. While the do have to answer for what they've been up to, they develop software and show the value of that software. They don't take specs, give estimates which become deadlines, and get reviewed based on how well they met those deadlines.
It's all about aspirations. I have a few for myself. I'd like to get to the point in my career where my interview focuses more on my existing work and contributions (github, blogs, speaking at conferences) rather than my performance on finding cycles in linked lists at the whiteboard. I'd like to work in a quiet office (happily shared) with a door that closes rather than a loud, open office with back visibility. And yeah, now I can add in that I'd like an environment where I can work steadily on a challenging problem and be evaluated by what I have accomplished.
Just to be clear, nobody owes me a job on my terms, I have to earn it. But, ahem, the loudest people in our industry seem to be the ones who think the world owes them a talented developer on their terms. If you're having trouble attracting and/or retaining talent, you might want to consider how developers like to work...
Just like how most people who ask for estimates are actually asking for commitments, most articles that state they are about estimates are actually about commitments, including this one.
Estimates and commitments are opposites - with a commitment, the giver of the commitment is under pressure to deliver. With an estimate (an actual estimate, not "I'm using the word 'estimate' but I really want a commitment"), the receiver is under pressure to use that estimate to control the project to meet its targets. That's all an estimate is good for, so there is no other reason to ask for it... which is why hardly anyone does.
By definition, you can't negotiate an estimate. The subject of a negotiation is always a commitment, so if you know that the number you're going to bring to the table is going to be negotiated, make sure it's a commitment you bring, not your personal estimate. The time-honored tradition of "doubling and adding some" is simply the process of converting an estimate into a commitment before bringing it to the table.
I had many long discussions with a previous boss about estimates. What it boiled down to is this:
* He is on the hook for certain things to be done, and is being asked when they will be done.
* I don't have enough data to tell him in such a way that the organization can plan around it. Anything I tell him is purest moonshine and unicorn hair. Sometimes it relates to reality.
* If I give an estimate, I am essentially passing dishonesty through the organization. I am lying to him.
This did not go over well. We compromised: I gave a order of magnitude WAG. Which, at the least, gives meaningful information and probably isn't wrong.
I'm a development manager and so am obviously asked to give estimates all the time. I found many years ago that no-one reads the qualifications that go with estimates, they just want a date or a number of weeks' effort and assume that it's a commitment.
I now play The Quality Game, which works every time. An example from a few weeks ago:
I gave an estimate for a piece of work which has plenty of contingency. For any number of reasons, when I gave the date, I expected a sharp intake of breath from the client --- so I quickly added: "The reason for this is that it's a complex piece of work - I've allocated 2 developers and one DBA for 6 weeks, and then one tester for 4 weeks. It's complex and important, and there is no way that the team wants to compromise on quality. That's why it's taking so long. Obviously, if we can do it faster, we will. How does that sound?" No problem.
I've played the Quality Game when telling clients that they need to upgrade to a later version, despite their having been told they wouldn't have to. Upgrading takes time and effort. Again, when quality is at the center of what you do, the logic is unassailable. ("WE could do it without you upgrading - of course - but our strong recommendation is that you upgrade. We're not saying this to piss you off, we're saying it because we want you to have the best quality product possible.").
Dev teams and clients never want to compromise on quality. Sales people don't care, they have to hit their figures, but it's 2 against 1. We win.
A game I play that is not mentioned in the article: give an estimate in the form of "I will not be done with this before %time." I'll often add in something about giving a better time estimate for the entire thing when that period of time elapses.
I usually build that time estimate by imagining the whole process going smoothly. The vast majority of surprises are negative, and of unknown size, so however long the project takes, it's going to be something bigger than the everything-goes-right scenario.
26 comments
[ 3.5 ms ] story [ 73.7 ms ] threadPushing for a 2 level estimation is definitely not easy, and if the person on the other side of the negotiation refuses this, then I ask him/her to stop playing the "Price is right" and spit out his time requirement already.
There is a popular psychology book, “Games People Play” by Eric Berne that shows and explains many of those game mechanics. Everybody uses them, not just PMs and bit crunchers.
This book helped me figure out that, in personal relationships especially, I could raise the discussion by one level and talk about the game I was playing in an effort to figure out why and get somewhere useful, emotionally. As in, "I think I'm just playing 'Why Don't You - Yes, But'. I need you to just listen to me while I vent."
Can't recommend enough.
EDIT: I should say that the edition I read seemed a bit sexist? I just ordered the new edition so we'll see if any of that text changed.
No. You are confusing the natural inclination for people to negotiate with some kind of slippage into "politics" (The word "politics" is most often used to describe "system of people stuff that I do not understand and that does not make sense to me")
For thousands of years, people have had a natural way of dealing with commitments where incomplete information is present. It's called "negotiation", and it's done by various techniques and tactics such that when all is done both sides are happy. We've also had ways of numerically determining tangible things. It's called math or science.
These are completely different things. What I see all the time is scientific, technical people flummoxed at how business-minded people are treating them. And the reverse is equally true. The reason is that both sides are working inside a completely different universe.
I like this article because it shows how multiparty negotiations can lead to disaster, but I do not like it because it confuses the true nature what's going on.
I tend to think of politics in the broader sense, and suspect that the author of the piece does as well. It's the process of negotiation between people with different spheres of influence, when a single decision has to be made. Client negotiations are absolutely political.
The weather is an extremely complex system, yet we use weather forecasts all the time. What's different between weather and software is that everyone knows that weather is extremely complex, and we account for that when we think about the weather in the future. ("Maybe I'll carry an umbrella today.")
The equities market is an extremely complex system, yet we use market forecasts all the time. What's different between equities market and software is that everyone knows that equities market is extremely complex, and we account for that when we think about the price of an equity in the future. ("Maybe I'll diversify my investments.")
Forecasting the completion date of a software project is no more complex[1] than predicting the weather or the future price of an equity. In no way is it "fundamentally unpredictable." What's different is that people don't regard project estimates as, well, _estimates_.
[1] It's probably a lot simpler.
Weather is governed by impersonal physical forces which can be described by mathematics. Software is built by groups of people who most defitely cannot be modeled as mathematical entities. Weather is a bad analogy here.
> "The equities market is an extremely complex system, yet we use market forecasts all the time. What's different between equities market and software is that everyone knows that equities market is extremely complex, and we account for that when we think about the price of an equity in the future. ("Maybe I'll diversify my investments.")"
Those market foreasts are often wrong. Even a company's own forecasts are frequently wrong, despite having the optimal perspective from which to make those forecasts.
Furthermore, diversifying investments is easy to do because the price mechanism makes those investments fungible (in a liquid market, anyway). This is not the case with software. Each piece of software is unique. You may "diversify" your software by investing in multiple projects, but ultimately you have a business need for a piece of software which does task X. "Diversifying" by funding software that does tasks Y and Z will not help you if the software "investment" in task X fails. "Diversifying" in three software projects, all intended to do task X, will roughly triple your costs for doing X. Again, equities are a bad analogy.
Software development is a process which can't be easily summarized by analogies to existing systems. In fact, the term "software development" can describe many different systems.
The real reason schedule estimation in software development is so hard is because, regardless of which system of software development is used, the problem of simultaneously optimizing to maximise a desired quality while minimizing the cost/schedule is inherently hard in a mathematical sense. Changes which promote one oppose the other, and the sensitivity of a software development system to unknown and unpredictable future changes cannot, obviously, be predicted in advance.
Your point about hedging one's bets is valid, but there are costs to doing so which are not as simple as carrying an umbrella or splitting up a pile of money across multiple equities.
So I stand by my original assertion that software estimation is not impossible.
(edit: To remain consistent, I should say that software schedules are certainly predictable, just not perfectly predictable. Most teams can get by just fine with good enough predictions.)
Assuming estimating 100 tasks, how many can we safely go beyond the estimate? 20%? 1%? 0.001%
Then give the estimate based on some probability curve considering the known unknowns.
If there was a planning process that was consistently reliable for new, complex software I think people would use it. And I think the adoption of agile methodologies in software development is an example of people trying to implement an improved planning process. Still, there is a lot of uncertainty.
So people negotiate, incentivize, and test conviction.. I guess that is what the author means by politics. Those same games happen in most complcated transactions and relationships. I think some programmers think this stuff happens to them because they are deeply misunderstood. But it happens to building contractors, real estate agents, lawyers, etc. too.
There are such processes. But they require understanding of the problem, which takes time. I am not sure "agile" is really attempt at an improved planning process, it's at best not doing planning and just marching onto the problem, seeing what happens at regular intervals in the hope of better understanding it, and at worst, things like "planning poker" where you put a bunch of people into a room and let everybody make a (possibly educated but usually not very much) guess from the top of their heads.
Here's my take. Programmers don't underestimate because they're afraid of getting fired or yelled at. In this market, a good developer is unlikely to be fired for honestly estimating. Yelled at, yes. Fired in worse economic circumstances (e.g. 2003)? Possibly. Fired now? Doubtful. Rather, it's about a desire not to be jerked around. Implicit in the demand for an estimate is the threat to jerk the employee around-- either imposing stupid new micromanagement ("Agile") or changing project priorities. No one wants to work for the boss who jerks her around and never lets her finish anything, because soon enough she gets to 3 years and while she's fixed bugs and added a few features, she doesn't have the coherent project that would make the case for promotion or transfer to her target group... because her boss kept jerking her away from projects whenever his hindbrain misfired, he interpreted it as the "this is taking too damn long" impulse, he got impatient, and pulled support. It's a fight-or-flight reaction in the boss when that happens, and managers tend to favor flight because fighters tend to get themselves fired. If you work for a hair trigger "flighter", you can very easily get to 3 or even 10 years without ever actually finishing a project. And flighters love estimates.
A good middle manager isn't one who exacts and reports reliable estimates. A good manager goes out, wines and dines the executives, and gets sufficient credibility for his people that no one ever has to give an estimate.
Anyway, programmers give misleadingly optimistic estimates-- not dishonest, just aggressively optimistic-- because they want to be left alone so they can actually produce something and move along to a better project. This is what "Agile" doesn't get. The thing that motivates us is delivering a major project that gives us enough credibility that we never have to make estimates again; not delivering small cantrips that anyone could do, but in record time. I think that "Agile" provides some structure for junior programmers, but it doesn't provide an exit. Under the Agile Ideology, even the senior programmers in R&D and architectural rules have to structure their work in terms of these dumbass "iterations". Implicit to Agile is the absence of personal progress for an engineer: no matter what you do, you'll always be working on feature-level "user stories" instead of real projects where you call the shots, and you'll always be subordinate to the business.
I've been doing this for almost 10 years and I'm good at it, so what being asked for an estimate communicates to me is that whatever I'm doing isn't really important. I will do what is necessary to meet a real, hard deadline (they're rare, but they exist) such as a bid deadline for a government agency (where people can go to jail for corruption if they grant an exception to someone who's a day late) but the silly deadlines that business usually creates are just resource limits, and (sorry but) I'm too good to work on something so unimportant that I'm not even trusted with a month or few of my own time on it. If you won't accept delivery on my time, then it's not important enough to be doing in the first place, because I will work very hard to make sure it's done reasonably quickly and well, and if you don't trust me on that much, then you should just fire me.
In other words, the difference between "4 weeks" and "2 weeks" is going to mean that I either don't get to do it or have to relegate it to extra hours, then (unless there is a hard deadline, and not some silly target like "Q3 KPIs") it's ju...
But I think what you have written is applicable to most developers in general, particularly this gem, which I loved:
I also agree that the article was wonderful. I laughed out loud several times because, like all of the best comedic material, it is firmly grounded in the painful truth.In the end, developers really do just want to build great stuff and I think most of us _are_ acutely aware of the realities of costs and time in real business. I sympathize with management's desire for estimates (and even the various methods to somehow get a reign on development). But the unfortunate truth is that the attempts often only serve to irritate and chain down people whose only desire was to work like hell making quality software in the first place.
When you work with motivated engineers (most programmers are in my experience), they set their own pressure and deliver quality work.
If you have a project supervisor that doesn't understand how quality software is made, leave. Putting in 80-100 hours? Nuts! Your estimates will slip even further when you do that.
I'd say that developers do pretty clearly prefer working in an environment where they aren't required to provide estimates and meet arbitrary deadlines (nobody likes hard deadlines either, but there's a difference between a window to launch a rocket to study halley's comet and a manager who figured that saying "march" rather than "may" might get the team to work a little harder).
This does't mean that developers don't like to work hard or produce good work, I think they just greatly prefer to take on a task, work diligently at it, and show progress rather than be pressed for estimates based on limited knowledge that quickly morph into hard deadlines before the problem is adequately understood.
Yeah, I'm stating the obvious. But thing is, it isn't a complete pipe dream. I do know developers who have quite a bit of autonomy. While the do have to answer for what they've been up to, they develop software and show the value of that software. They don't take specs, give estimates which become deadlines, and get reviewed based on how well they met those deadlines.
It's all about aspirations. I have a few for myself. I'd like to get to the point in my career where my interview focuses more on my existing work and contributions (github, blogs, speaking at conferences) rather than my performance on finding cycles in linked lists at the whiteboard. I'd like to work in a quiet office (happily shared) with a door that closes rather than a loud, open office with back visibility. And yeah, now I can add in that I'd like an environment where I can work steadily on a challenging problem and be evaluated by what I have accomplished.
Just to be clear, nobody owes me a job on my terms, I have to earn it. But, ahem, the loudest people in our industry seem to be the ones who think the world owes them a talented developer on their terms. If you're having trouble attracting and/or retaining talent, you might want to consider how developers like to work...
Estimates and commitments are opposites - with a commitment, the giver of the commitment is under pressure to deliver. With an estimate (an actual estimate, not "I'm using the word 'estimate' but I really want a commitment"), the receiver is under pressure to use that estimate to control the project to meet its targets. That's all an estimate is good for, so there is no other reason to ask for it... which is why hardly anyone does.
By definition, you can't negotiate an estimate. The subject of a negotiation is always a commitment, so if you know that the number you're going to bring to the table is going to be negotiated, make sure it's a commitment you bring, not your personal estimate. The time-honored tradition of "doubling and adding some" is simply the process of converting an estimate into a commitment before bringing it to the table.
* He is on the hook for certain things to be done, and is being asked when they will be done.
* I don't have enough data to tell him in such a way that the organization can plan around it. Anything I tell him is purest moonshine and unicorn hair. Sometimes it relates to reality.
* If I give an estimate, I am essentially passing dishonesty through the organization. I am lying to him.
This did not go over well. We compromised: I gave a order of magnitude WAG. Which, at the least, gives meaningful information and probably isn't wrong.
I now play The Quality Game, which works every time. An example from a few weeks ago:
I gave an estimate for a piece of work which has plenty of contingency. For any number of reasons, when I gave the date, I expected a sharp intake of breath from the client --- so I quickly added: "The reason for this is that it's a complex piece of work - I've allocated 2 developers and one DBA for 6 weeks, and then one tester for 4 weeks. It's complex and important, and there is no way that the team wants to compromise on quality. That's why it's taking so long. Obviously, if we can do it faster, we will. How does that sound?" No problem.
I've played the Quality Game when telling clients that they need to upgrade to a later version, despite their having been told they wouldn't have to. Upgrading takes time and effort. Again, when quality is at the center of what you do, the logic is unassailable. ("WE could do it without you upgrading - of course - but our strong recommendation is that you upgrade. We're not saying this to piss you off, we're saying it because we want you to have the best quality product possible.").
Dev teams and clients never want to compromise on quality. Sales people don't care, they have to hit their figures, but it's 2 against 1. We win.
Thing is: it's not a game, it's just the truth.
(Great article BTW, will share with colleagues!)
I usually build that time estimate by imagining the whole process going smoothly. The vast majority of surprises are negative, and of unknown size, so however long the project takes, it's going to be something bigger than the everything-goes-right scenario.