I’m a big fan of the Donald Reinertsen approach: measure queue length.
Simply track the time to complete each task in the team queue on average, then multiply that by the number of tasks remaining in the queue.
Each team will habitually slice things into sizes they feel are appropriate. Rather than investing time to try and fail at accurately estimating each one, simply update your average every time a task is complete.
The bonus with this approach is that the sheer number of tasks in the queue will give you a leading indicator, rather than trailing indicators like velocity or cycle time.
Strongly seconding this. For anyone still hesitant, I further recommend the following experiments:
----
Sample a few activities your team has completed. Check how long the 90 % smallest activities are on average, and compare it to the average of the biggest 10 %. Or the median compared to the maximum, or whatever. You'll probably find the difference is about an order of magnitude or less. In the grand scheme of things, every activity is the same size. You can estimate it as exp(mean(log(size)) and be within an order of magnitude almost every time.
Once your team has accepted that something is "an" activity and not a set of activities, don't bother estimating. For all practical intents, size is effectively constant at that point. What matters is flow, not size.
----
For the above sample, also study how long passed between the "go" decision on the task and when it was actually released to customers. In a stable team, this number will be eerily close to the theoretical based on Little's law referenced in the parent comment.
Oh, and you shouldn't focus on man-hours. Work with calendar days. Not only does that simplify mental arithmetic for everyone, it's also the only thing that matters in the end. Your customer couldn't care less that you finished their functionality in "only" 6 man-hours if it took you six weeks to get it through your internal processes.
----
Fun follow-up to the size experiment: now ask someone intimately familiar with your customers to estimate the dollar value of each activity. You might find that while all activities are practically the same size, they'll have very different dollar values. That's what you ought to be estimating.
This. I've found that, whatever the project is, the velocity of a team in terms what they define as a "task" is pretty constant, surprisingly so even. In the end, just counting the outstanding tasks proved to be a good estimator of where we'd end up at the deadline.
What do you do when your future tasks are unknown or ambiguous?
For example, at my day job my task is to implement banking. The day to day tasks change... day to day. There aren't a "number of tasks remaining in the queue," since whatever I'm doing is what I'm doing.
One could say this is poor planning. But due to the nature of Big Banks, each task is usually blocking the next one -- in other words, it's not possible to discover or plan what you need to do next, until you've finished currently.
An example of this is when we realized we didn't need to run our banking API using $BigBank's test environment. Their test environment was ... uh ... well, let's just say, when we realized that we could simply switch on "production mode" and bypass their test environment altogether, we collectively facepalmed while rejoicing.
It wouldn't be possible to add "switch to the production environment" into the queue several days ago, because we didn't discover that we could do that until yesterday during our biweekly sync call.
I'm sympathetic to your writeup, and I like your recommended approach. But I just wanted to point out a realistic case of it failing. But in fairness, I think every estimation approach would fail us, so don't feel singled out. :)
Perhaps your approach will work in most cases though, and I'm merely stuck in a twilight zone special case.
What happens when you complete your work before you know what you need to do next?
If this never happens, then you have some invisible queue, as you do have things to do next.
As far as your example, that's a great example of a task that seemed like it would take long, and ended up being very very short. Can you describe why this would be bad to add into your task system?
- Add Task: Run banking API in $BigBank test environment
- Start work time clock.
- Find out we don't need to do it, and switch to prod mode
- switch to prod mode
- Close task, and time clock
This is now data for your estimates of future tasks, as this will probably happen randomly from time to time in the future.
Switching to prod mode takes 5 to 7 business days, because we have to order certs from DigiCert and then upload them to $BigBank, whose team requires 5 to 7 business days to activate said certs.
We expected to turn on prod once testing was finished. But we ended up discovering that prod was the only correct test environment, because their test environment is rand() and fork()ed to the point that it doesn't even slightly resemble the prod environment. Hence, "prod am become test, destroyer of estimates."
So for 5 to 7 business days, we'll be building out our APIs by "assuming a spherical cow," i.e. assuming that all the test environment brokenness is actually working correctly (mocking their broken responses with non-broken responses.) Then in 5 to 7 business days, hopefully we'll discover that our spherical-cow representation is actually closer to the physical cow of the real production environment. Or it'll be a spherical cow and I'll be reshaping it into a normal cow.
By the way, if you've never had the pleasure of working with a $BigBank like Scottrade, Thomson Reuters, or $BigBank, let's just say it's ... revealing.
Maybe I'm missing your point. It seems you're attempting to answer the wrong question: is this task accurate, given all the changes that have happened? This is irrelevant for large scale estimation.
The question for scheduling prediction is: what distribution of time will it take to mark any task in this queue as FIXED/INVALID/WONTFIX/OBSOLETE/etc? The queue can have any amount of vagueness you want in it.
Regardless of the embedded work, regardless of whether or not it changes, becomes invalid, doesn't exist, etc - these are all probability weights for any given task/project.
> The day to day tasks change... day to day. There aren't a "number of tasks remaining in the queue," since whatever I'm doing is what I'm doing.
What you are describing is not ambiguity, it's total variability. If your future is 100% random, it is, by definition, impossible to predict. Such a state would also mean a total absence of direction/vision. Predicting dates is not only impossible but not a question you can ask, since you don't know what's next.
What I'm going to challenge is that you're effectively in such a case, because I don't think it's true.
> One could say this is poor planning. [...] in other words, it's not possible to discover or plan what you need to do next, until you've finished currently. [...] because we didn't discover that we could do that until yesterday during our biweekly sync call.
The example you're giving *is* poor planning. You're going into execution without validating base assumptions. That you discover the specifics of a dependency that late into the game is that you're going into it without a plan. I'm not judging, in your case maybe no-one is asking for any sort of accountability, and just executing is the best recourse with the lower overhead. But the fact that you can't estimate isn't due to the environment, it's due to the fact that you don't have a plan. Some of the companies I worked with are fine with that, most are not.
It's like this for day-day operations when leadership is absent and no CSI ever gets prioritized over new development. You can say the org is dysfunctional, but there's less leverage workers can use to change such situations. Especially when efficiency measures get rewarded with layoffs.
This is great for a simple monte carlo simulation!
Choose a finished tasks time randomly, once per remaining task in the queue, and add this up to be a single estimate. Do this 1000 times or so and get an estimated distribution of completion times for the current queue.
This type of thing is covered extensively in Evidence Based Scheduling[0], and is one of the reasons I still think FogBugz ' power is misunderstood.
Nice, I just recently left a company where this approach would have been tremendously useful and fairly easy to build - we had all the required data already, but were just looking at averages of past performance to budgets and using that as a multiplier on the schedule rather than going through a distribution.
That being said, I also despise tracking time!
…Suppose you could move that technique to story points (or whatever unit of measurement) though you would lose a ton of precision.
This is interesting. A lot of machine learning works exactly like this [1], in the sense that you're biasing your prediction of each ticket time towards the average, except when there is strong evidence that ticket is special.
(You'd arrive at exactly this method if you found that there were no easy to find characteristics of tickets that could distinguish them as extra-long or extra-short)
Probably this works for a large team of tens or hundreds of developers.
On a team of 4 or 5 people, where people go sick, take vacations, leave the company, new members join... each of these events has a big impact on those metrics. Which again, becomes a lot of work and effort wasted.
But yes, probably this is a viable option on larger teams.
I really dislike posts in this vein, because they use statements like
"This post will teach you how to replace estimations and guesses with a Monte Carlo simulation."
as though the output of a Monte Carlo simulation were not an estimate, but some higher truth. The process of simulating many different outcomes with a touch of randomness gives the layperson the impression that we are really _doing_ something, when in fact we could have obtained the exact result (no confidence intervals needed) of Monte Carlo by quadrature (although admittedly that could be tedious).
In the end, the method outlined in the link is an estimate, like any other, and is not necessarily better or worse than any other estimate. It erects a straw person and then misuses mathematical terminology throughout.
I use Monte Carlo simulations because I am not that great at doing complex statistics.
It sounds fancier to say modeled using a Monte Carlo simulation, but in reality it is much easier than working with joint and conditional probabilities.
I sort of agree with you although for different reasons. The problem with the MC approach IMO is not really that it is also an estimate, estimates are fine. However doing a Monte Carlo simulation is only reasonable if we have a good model. So now we have moved from we can't estimate (intuitively I guess) to how to find a good model, that is not trivial and the time spend on finding and verifying the model might take more time than what you are actually trying to do (just an (gu-)estimate on my part)
A Monte Carlo simulation is infinitely better than all the conventional "to estimate, take your first estimate and multiply it with 10x". And even better, your manager can do that without your help and this learn better how much you tend to over/under estimate projects without trying to force you to change your estimates to fit what he think it should be.
It also ensures that the final estimate becomes on average correct, which is much better than most projects which tends to under estimate the effort it will take. Although, under estimating projects is often a feature managers wants since it makes it easier to sell it to customers, ie "My engineers estimated we can remake Twitter in a weekend, I think we should focus on that!".
Anyway, the point is that if your manager complains about your estimations being on average off then he is just incompetent and should have applied a Monte Carlo simulation on it instead of asking you to perform an easily automatable task.
Indeed, "estimations" might not be the best word as you're still, in some way, estimating with the Monte Carlo approach. I thought of this differentiation because I consider the MC approach to yield a forecast, while an "estimation" is the typical term used in agile settings for "guessing how long/much work it will take.". I'll think of ways of making this distinction clearer.
As for the misused mathematical terminology I guess you're referring to "confidence intervals"? If so, I understand it also has a clear definition within statistics, but I used it in the broader sense. I'd be grateful if you could point out which terms you'd recommend replacing and what you think I could replace it for.
> Indeed, "estimations" might not be the best word as you're still, in some way, estimating with the Monte Carlo approach.
Not in some way -- literally. The samples are averaged, and as the number of samples goes to infinity, the average converges to some true value. In the case of one of the histograms you plot, you are averaging an indicator function .
> I thought of this differentiation because I consider the MC approach to yield a forecast, while an "estimation" is the typical term used in agile settings for "guessing how long/much work it will take.". I'll think of ways of making this distinction clearer.
It seems to me the distinction you care for is to provide not a single value ("estimate"), but a distribution of values. That's not specific to Monte Carlo.
> As for the misused mathematical terminology I guess you're referring to "confidence intervals"
That's certainly one of them. I even misunderstood what you were doing and said you don't need confidence intervals, but in the way you're using them those again can be obtained by quadrature.
My main gripe remains the language, and the presentation of the method. You could most likely obtain your results by hand, i.e. with pen and paper, and you would still get a distribution at the end of the day. The advantage of running Monte Carlo is that it simplifies this process. You could illustrate this by getting the distribution of the sum of two independent uniform random variables in two different ways: by doing a convolution, and by MC. Wow, MC was so much easier, anyone can do it, and it can handle arbitrarily complicated distributions (in principle).
While a different application than shown here, my project management class taught me one of the biggest benefits of Monte Carlo simulation - estimating uncertainty.
Traditionally, net present value calculations are done with single point estimates. For example, analyzing a rental property we want to buy, we'd estimate the vacancy rate, interest rate, property appreciation, maintenance expenses, and do all of that on a cash flow time-adjusted basis.
There's variability on every aspect of the through, and understanding how much can easily swing your decision.
Traditional methodology is to pull the trigger on the action if the net present value is greater than 0. We put in a variety of estimated factors, ran it through Crystal Ball (simulation software), and found a 75% chance the NPV would be negative. It suddenly isn't the safe investment it looked like originally.
---
In practice, it's tough to get people in business to understand uncertainty. They gravitate towards "the number", which is almost surely wrong all the time.
The real danger is not understanding that this "uncertainty" estimate is a function of your assumptions. How you model the distribution of your inputs is huge, and often not stated clearly.
GIGO, the first thing I learnt, as I entered the industry 20 years ago. This was from a 60 year old engineer who told me that experience is only a nice name for "all the @#$% I made I will try not to make again".
A very nice thing about Monte Carlo simulation is that at the end your distribution of results are all within the feasible range. If you do error propagation using uncertainty on your parameters, you can get non-sense results.
For example, suppose you have a pendulum, you simulate it with error propagation and you have a non 0 probability of having an increase of the energy in the system over time.
It is easy to spot this in such constrained example but with more complex models, this is sometimes pretty hard to control.
> A very nice thing about Monte Carlo simulation is that at the end your distribution of results are all within the feasible range.
I guess Monte Carlo helps provide conservative estimates behind a façade of rigour, but the truth of the matter is that in the end it's still GIGO.
Any empirical distribution only reflects the empirical measures that were used to generate it. If you bundle everything from the time it took employee A to walk the dog while allocated to project Foo to the time it took employee Z to fix a nasty Eisenbug while allocated to project Bar, and Foo and Bar used totally different tech stacks and team members and even approaches to project planning, that distribution is meaningless in estimating, say, how much time it will take employee G to implement a React widget.
I’m smiling a bit while reading your comment. You are completely correct that that estimate is virtually meaningless. And yet… that doesn’t mean it’s completely useless!
I’ve been using a somewhat meaningless Monto Carlo inspired approach to project planning for a while in my consulting company. The vast majority of the projects I do have only a small amount in common with previous projects, so when I’m estimating I’m informed by past estimate vs actual numbers, but not really relying on much other than intuition and gut feelings from past work.
My basic approach is to estimate 2- or 3-sigma upper/lower estimates for each task, modelled (incorrectly) as a Normal distribution in hours, and then sum the random variables to come up with a final distribution (whose variance is quite a bit smaller than what you see on the original bag of tasks). From there, if I am making a quote, I’ll quote at +3-sigma x hourly rate as a high-end effort estimate. If it seems like a very meeting-heavy client, I’ll either add meetings in as development tasks or just pad it out by a %age or fixed hours/week.
This technique has worked amazingly well for me, and it’s been quite rare that I blow the estimate, and in the… one time I can think of, we missed by very little and there were tasks that we hadn’t thought of when we did the estimate.
To your point though (with walking the dog), there’s a really key thing that this process doesn’t capture, somewhat on purpose: the resulting estimate is in effort-hours, not delivery date. While modelling per-task effort hours as Normal is suspect, modelling task delivery dates as Normal is completely irrecoverably wrong: delivery dates, in my experience, only ever slide in one direction. People get sick and the project slips a week; people don’t ever get super healthy and effectively knock out 80 or 120 hours worth of tasks in a week.
I honestly haven’t found a good way to estimate delivery dates very well. At one point I did put together some regression on my “actual billed hours per week” based on my billing, but ran into the same problem. “Oh, my dog died and my mother-in-law got sick during that project.”
Someone who’s better at statistics might have a better way to model that as a high-skew distribution, but when I tried doing that myself I ended up with a distribution that didn’t feel like it did a good job of capturing the non-negligible long tail of things that slow down calendar estimates without burning hours.
The traditional distribution for costs and dates is Beta-PERT. Douglas W. Hubbard uses log-normal distribution. I am a pessimist, so I like Beta-PERT with a fat, fat tail.
> If you do error propagation using uncertainty on your parameters, you can get non-sense results.
Only if you do it wrong, generally. Events which are impossible under some hypothesis should have zero probability under a model for it and not be sampled.
"Just take a guess - we have to have a number in this box by the end of the meeting". I was literally just told that yesterday. No one else who would look at that number will know it was a totally random guess pulled out of thin air to satisfy ceremonial box-filling. They will treat it as an estimate, and make plans accordingly.
And if you make up a number that is too obviously small or exceeds some unspoken upper bound, you'll be asked to re-estimate anyway. Sometimes the best way to respond to that is to finesse the discussion into coming up with a number that, while it will have no relationship to the actual effort, at least reflects what the stakeholders hope and desire it to be. At that point the team, if they are smart, will examine scope and re-plan to come up with some level of effort that they are confident can be done in the time hoped for.
In other words, get the people who want a number to tell you what number they want, then use your best efforts to scope the effort to one you can be pretty confident will fit.
> This simple example shows how the net present value may lead the firm to take unnecessary risk, which could be prevented by real options valuation.
Even with a monte carlo on NPV, you may still make a more risky decision than needed; as decisions are rarely just "do or dont" but can be sequenced pending more information.
It's not just business people. I have yet to find a task tracking system (Jira etc.) that lets you assign a range of points to a task.
People try and use nonsense like Fibonacci numbers to imply uncertainty, but then just add up all the numbers to get a number with no uncertainty measure.
Story points are a curse on the software development industry. However much people say "they're indicative, and don't map to hours", someone, somewhere, will map them to hours.
The most accurate project plan I was ever involved in had only 3 values that could be assigned to a piece of work during the early estimation phase: hours, days, and weeks. Each of those was then turned into a range of possible hours they could represent, with that range expanding as you got into larger units. You could then slice and dice the numbers however you choose to get anywhere between the most pessimistic timeline to the most optimistic timeline. Probably unsurprisingly the project was delivered somewhere between the two.
That seems reasonable, but I would say you don't have to resort to such crudeness. In my experience I know the difference between a task that will definitely take weeks and a task that might take a day or might take weeks. Just let me write that down!
Even if you don't have any idea about the uncertainty we already have a crude way of measuring it - planning poker! Just record everyone's guesses instead of throwing away the uncertainty information. There's a huge difference between everyone guessing 5, and some people guessing 1 and others guessing 20.
I agree about points being stupid though - there's simply no way to avoid it being converted to/from time, because that's the actual unit of work.
Sorry, reading back I definitely come across as disagreeing - I would kill someone for task tracking software which supports giving the degree of uncertainty on an estimate.
One of the concepts that many people don’t get is differentiation between risk and uncertainty. These same people will then attempt to “model risk” and end up with (ironically) an “uncertain estimate”
They define risk as having statistical noise or a varying parameter in the estimate, while uncertainty is lacking information about the target. Doesn't the presence of noise in an estimate imply that there's information you aren't accounting for? I don't see the importance of the distinction.
Estimating uncertainty is preferred, but if you have people who insist on a single number, then teach them that it should be the median instead of the mean. Means tend to overvalue "moonshots" where there's a 99% chance you lose money, but if the payoff in that 1% where you win is large enough, it can still result in a positive mean. And a mean can often be an outcome that isn't actually possible, but falls between several options, while a median is always an outcome that can actually happen, and there's ~50% chance you'll get a better one.
> Monte Carlo simulation - estimating uncertainty.
By definition, "uncertainty" is the thing that does not have a PDF ... You can quantify risk based on assumptions you make about underlying probability distributions from which you are drawing. Far too often, even in Monte Carlo, people decide to work with the easy distributions instead of the most appropriate distributions.
The "easy" distributions tend to admit actual mathematical solutions which means Monte Carlo is helpful if you can't do the math but not strictly necessary.
Monte Carlo shines when you cannot get nice solution and there is your opportunity to not be constrained by the need to do so: Draw from appropriate distributions.
Also, know the difference between LLN and CLT and when to appeal to which to justify your methods.
I would think that you could have "uncertainty" even if you do have a PDF. Maybe there is a formal definition of uncertainty that I am not aware of. The PDF can describe the uncertainty for every outcome.
Risk is what you can put a probability on (i.e., quantify). Uncertainty is what you can't. Uncertainty encompasses unknown unknowns. See Knight[1] and Keynes[2].
Good rule of thumb even if it sounds overly simplified.
After many decades we still discuss and wonder how work amount estimates should be done and why they are always wrong. The most brilliant software engineers and project managers have developed countless different methods and nothing works.
For me it tells that nature of work is such that we just can not estimate work amounts. We are just pretending that it can be done.
It depends on the type of work. Generally and counterintuitively, the more individual random variables the work contains the easier it is to estimate. This is because random variations cancel each other out when they are in large quantities.
With programming, one random variable can add days, weeks or even months of work.
I think FogBugz used this method to estimate how long open tickets would take to get closed. Or at least I remember Joel talking about it. I wonder if there are any other issues tracking apps out there making use of this technique
Yeah, FogBugz had a very similar feature 15 years ago.
It was a hybrid between the approach proposed in the article, and the "always multiply by N" technique.
Basically, FogBugz always asked you for your own best estimate. It tracked how much longer the actual time to completion was compared to your best estimate, and learned a simplified statistical distribution of your personal multipliers (e.g. in 80% of cases your estimate was 20% shorter than actual time to completion, in the remaining 20% of cases it was 10% longer).
FogBugz used the learned distribution to perform Monte Carlo simulations, and presented you with estimates of actual time to completion (e.g. "in 95% of simulations the project didn't take longer than 30 work days"). The great thing about this method is that it automatically accounts for chronic optimists, who always think it'll be ready to ship soon, and underestimate difficulty. The learned multipliers tell the algorithm that the estimates made by the optimist tend to be underestimates, so that it can present you with a "debiased" estimates.
E.g. if something always takes at least twice as long as Developer Anderson says it will, then FogBugz would tell you: "Anderson estimated 10 days, but based on past data I think it'll take at least 20, and it took less than 15 only in 5% of the simulations".
It had a very simple algorithm at its core, but the developers were knowledgable ,and had tons of data about project management. This allowed them to add many empirically verified "embellishments" (e.g. if the algorithm didn't have enough data about your specific multipliers, it assumed that your multipliers were about the average of your coworkers'), which made the algorithm surprisingly accurate in the end. Certainly better than any estimates most people could produce by hand.
I never tried FogBugz but I heard good things about it, similar to what you've written. It's surprising it didn't catch on more if it's that good? I mean, I suppose it could still catch on, the product still exists as far as I know.
I’ve written about why FogBugz failed before, but it basically came down to Atlassian having vastly better marketing, being much more willing to support every edge case instead of being opinionated, and having a better pricing model.
Unrelated, the new owners of Kiln and FogBugz are asshats, I moved my stuff off ages ago, and I’d encourage anyone else still on them to do the same.
pip install carlo
carlo "d(20)+randint(2, 5)+int(random()*20)"
This will continually generate samples and show them in a self-updating histogram.
This is harder than it looks, because there's too many samples to store them all, and we don't know the range of values at start. But I'm quite happy with the results.
That is pretty neat! I have a dice stats python script I use specifically for DnD. It has command line flags and basically the same syntax as the discord dice bot. Very useful when you are considering strategy, even though my group plays fast and loose with the DnD rules.
It's fascinating how poor this area currently is. We are able to capture so many tiny, horrifying details about engagement and optimise shit out of ad clicking but we don't do basic things like using calendar or task dependency for internal delivery estimation.
Looking at what we're already doing on outside, improvements inside seem trivial. Good estimator should know that estimation spanning holiday season will be impacted. It should know serial dependency will span wider in time. It should know that set of unevenly distributed tasks (A-many + B-not-many) for small-team-A and larger-team-B will flow slower in time than distribution matching weighted max flow for team A and B. It should know multiple dependency on team X will have lower flow because they're maxed out.
It seems to me that monte carlo projection on timeline from inputs we already have mapped onto graph is really not a rocket science and it should have been solved problem by now. But it's not. Weird.
Not to hijack the conversation, but I though some people interested in replacing estimations could find this useful.
In order to replace estimation, we are trying Basecamp's Shapeup language and techniques [1]
In a nutshell the thinking is reversed, instead of asking:
- "how long would it take to implement X?"
you ask:
- "how much appetite (in weeks and people involved) do I have for this feature ?"
Then you work with a fix "dead line" and variable scope.
Instead of user stories, they call it Bets, and they give themselves 2 weeks to shape bets and 6 to implement them (these two cycles can happen in parallel), during those weeks is about leaving the design at the Mockup/sketch level, and identify with help of business and technical people all the possible risks, so the Bet safe becomes a safer Bet.
We are still in the first cycle, and still a lot of questions and doing our own little customisations, but everyone response seems to be positive with the way of working, curious if anyone else has tried succeed and fail.
Flipping the question within a restricted variable is my favorite hack.
Overlords - “How much money do you need for this project?”
Me - “How much money can we spend?” / “How much time should we spend?”
Friend- “Look at how much money that CEO earns!”
Me - “What will you give up in your lifestyle to do what the CEO does?”
The only challenge though in these situations is that most people aren’t open enough to do “Think in bets”. They either avoid the question or don’t even attempt to come an answer (despite them not spending a single dime!).
Shape up is a powerful framework, it takes a while to get right but it’s very worth doing. We are about a year in, still improving at it. Our main focus now is doing things more by the book and getting rid of the customizations we added in the early days - so try and not make too many! :)
Email is in profile if you want to chat more on trade notes on it.
We've been doing Shape Up since July 2020 and have yet to fail to ship what we shape for each 6 week cycle, and while there's sometimes a bit of a crunch in the last few days before a release, everyone is generally working at a pretty relaxed pace.
My favorite side-effect of working this way as a leader is that by batching 6 weeks worth of planning effort, the team (and myself) can just be generally left alone to get into it without having to decide on the next priority every single week. Combined with async check-ins (that the team provide themselves every couple of days via Basecamp) instead of daily/weekly stand-ups, everyone generally has an empty calendar at all times.
We've been using ShapeUp at Zinc Learning Labs for two years now and it has done wonders to our pace and sanity. I wrote about our experience with ShapeUp recently, specifically about moving away from estimation - https://world.hey.com/karthikc/wrestling-with-the-unknown-be...
Shape Up is really powerful and cuts straight to the meat of things which is getting things done as opposed to the traditional way of planning and processes greater than actual results. We’ve been using Shape Up for a little over a year at the Sailscasts Company and love every bit of idea in it. I think personally is a lean mean way of getting things done with tangible and measurable results!
I'd love to hear a post mortem blog post on a project that was ran in this way in real life & what the pros/cons at different stages were.
The author first talks about the waste of time that is upfront analysis/planning or adding a buffer for contingency and how Monte Carlo simulation fixes for that, but then the caveats seem to be based on an established team, of a fixed size doing similar style work.
For me, this stage of a project has never faced big hiccups in estimation & our teams are normally quite good at knowing "what's a 2 vs a 5 point story" — if there _is_ a big discussion in a planning meeting at that stage, it's normally because one person has overlooked a key requirement (or the other is assuming a requirement that doesn't exist) so it's a good opportunity to solve for that.
To use the analogy, we're probably quite good in established teams at estimating how long it'll take to write 10 blog posts. The trouble comes when someone says "The latest blog posts aren't generating enough interest. Can you estimate how long it'd take for you to write 10 poems instead? We want a mix of iambic pentameter and maybe some trochaic tetrameter (TBC). We've gotten budget approval to increase the size of the team, so we'll have 5 new people (off-shore) joining next week so we can go faster. Also, some of the old blog posts seem to be getting lots of visitors in South America, so we may need to translate these to Portuguese/Spanish too. When can you give me an estimate for all that?"
The book "How to Measure Anything" by Douglas W. Hubbard has a chapter on using Monte Carlo simulations to do project planning and gather estimates. It's successfully been used for project planning large complex projects like building nuclear power plants, or projects that NASA or the Navy or similar have done.
The approach is slightly different then the article above describes. Instead it has each engineer go through calibration exercises until they can fairly accurately produce 5th and 95th confidence interval estimates. Then each engineer provides 5th and 95th confidence interval estimates for each item that needs to be worked on. Those confidence intervals are kept separate. You can then run a Monte Carlo simulation where each piece of work is weighted randomly assigned to each engineer that provided estimates for that particular item, and randomly picks a number based off their provided confidence interval estimate on how long it ends up taking them to complete the item for this particular simulation.
I was on a small team that used the above technique. We were in a large company trying to launch a new product, and given manufacturing lead time, and seasonality of the market demand, it was very important that we could provide a good estimate to the business when we could have the software portion of the MVP completed. The business provided us with what they thought the MVP features were, we further added in engineering tasks that weren't business facing, but needed to be completed. We confidence interval estimated those, and then also confidence interval estimated our personal vacation days, sick days, as well as a bucket of "unidentified work". The 80th percentile Monte Carlo simulation put us out a little more then a year. Our actual delivery was off by only a week from the Monte Carlo tp80; I don't remember in which direction, but it wasn't consequential to the business.
There is also a short chapter on estimation tools using e.g. Monte Carlo simulations. One tip which is relevant here and that is also more or less presented in the article and probably obvious :) "TIP #65: Don't treat the output of a software estimation tool as divine revelation. Sanity-check estimation tool outputs just as you would other estimates."
Some other books I can recommend to read alongside McConnell's are Waltzing With Bears by DeMarco and Lister (of Peopleware fame) and How to Measure Anything by Douglas W. Hubbard.
For those who like to do things a little more visually I recommend the open-sourced Argo[0] for microsoft excel.
There's a lot you can do with a monte carlo model. Tornado plots are something I highly recommend for understanding risk profiles around any complex task
A little bit of knowledge about stats and the appropiate language helps make this exercise easier. The R language is good at this kind of simulation. It comes with a whole bunch of random number generators out of the box. Here is how to estimate frequency of 7 from sums of two d6 dice rolls in R in two lines:
We want two six sided dice so we sample a random uniform (continous) distribution and round to a whole number:
`round(runif(n,1,6))`
This will make a vector of 1 million integers representing random "rolls" of the dice. In R it is really easy to pair-wise add two vectors of the same length so to get two dice we do:
`round(runif(n,1,6))+round(runif(n,1,6))`
Now we have a million rolls of summed pairs of dice. We want the frequency count of 7s. We can get this by getting the frequency count of every dice with `table()`:
`table(round(runif(n,1,6))+round(runif(n,1,6)))`
Which will return a print out of frequencies for every number that appears in the vector.
We can select 7s by pulling the 6th index and then dividing by the `n` dice rolls to get the frequency:
A Monte Carlo simulation can hide a lot of assumptions about the the actual system being modelled and the distribution of random variables (both on a univariate basis and as a dependency).
Its flexibility and power of estimating (otherwise maybe untractable) metrics comes at the expense of transparency, potential simulation noise and difficulty in estimating accurate "what if" scenarios. The risk is that one might get out simply what one assumes.
Here a list of questions you can ask to minimize the associated risks and steer your development:
* Do I have a good, self contained, description of the system that I want to simulate? Is it even possible to define it in practical terms?
* Do I have historical data that can pin down its stochastic behavior? Can I estimate a statistical model reliably?
* Does the uncertainty around model estimation justify retaining the full model or could I possibly simplify it and obtain semi-analytic results?
A more “back of the envelope“ approach to handling uncertainty for this sort of thing is the “three point estimate” approach, where you give a best, worst, and most likely estimate for each sub task. These points are implicitly used to parametrise a distribution, and then you analytically find the overall uncertainty, rather than through simulation.
We used a different three point system. 1 point means we know exactly how to accomplish this work. 2 means there is uncertainty, but we are confident it will completed. 3 means there are unknowns that require more digging. A 3 pointer is a good candidate for breaking the story/task up into smaller chunks.
This is a very naive approach for a number of reasons:
1). You're only sampling a tiny fraction of the space of possibilities. The example using two dice having 36 outcomes makes it seem like you're in a very well behaved world. But the total number of possible samples is n_possible_values ^ n_dice, so with three dice you have 216 possible outcomes, with 4 dice 1296, with 10 60466176. The growth is exponential in the number of variables, with predictable results.
2). The distribution of delivery times is not normal, experience shows it's much closer to a power law with a small number of tasks exploding beyond any reasonable expectations.
3). No critical path. A task can't be completed before it's critical path is completed. Since you allow subtasks to vary your critical path needs to be calculated for each run. With 1). and 2). it basically means that you have no idea if you've gotten a good representation of the sample space of critical paths.
If you run a simulation that samples a tiny fraction of your probability space you have no idea what monsters lurk in the background when the problem space is one prone to explosions.
In short: beware of tap dancing in minefields when blindfolded.
I used this method for a while on one of my teams. It was useful and helped provide more accurate estimates.
Some tips I have for people interested:
* Get about two months of velocity data in place first.
* Run the simulation on a regular basis. Update your estimates every week/sprint.
* It helps to have individual work units be "sized" as consistent as you can. (E.g. each ticket is 2-3 days, a consistent pointing system, etc.) It's ok if anomalies happen from time to time.
* Dates will get more accurate the further you get into a project.
Some things I've learned from doing this exercise:
* When work units are sized to be 2-3 days of effort, engineers complete about 4 units/wk on average.
* Don't shoot for the perfect estimate at the start of a project. Plan as much scope as you know and enough to get started.
* As new scope is found, update your project plan and estimates.
If you already have ~2 months worth of data, why not just use that to plot a distribution curve? Seems to me that the monte carlo simulation that the article describes is really just used to "guess" the likely distribution curve. Seems unnecessary if you already have ~2 months worth of data.
A nice exposition! I think Omniplan does this [1], and it certainly helped me in the past get a better idea on variability of long-term project planning. Of course, it's only as good as the assumptions that go into it!
For anybody who wants to try it out in a spreadsheet environment and/or is looking for a useful abandoned project to duplicate, I recommend Risk Engine for Mac. Unfortunately it hasn’t been maintained for a long time and has been abandoned, but I keep using it.
161 comments
[ 2.8 ms ] story [ 282 ms ] threadSimply track the time to complete each task in the team queue on average, then multiply that by the number of tasks remaining in the queue.
Each team will habitually slice things into sizes they feel are appropriate. Rather than investing time to try and fail at accurately estimating each one, simply update your average every time a task is complete.
The bonus with this approach is that the sheer number of tasks in the queue will give you a leading indicator, rather than trailing indicators like velocity or cycle time.
----
Sample a few activities your team has completed. Check how long the 90 % smallest activities are on average, and compare it to the average of the biggest 10 %. Or the median compared to the maximum, or whatever. You'll probably find the difference is about an order of magnitude or less. In the grand scheme of things, every activity is the same size. You can estimate it as exp(mean(log(size)) and be within an order of magnitude almost every time.
Once your team has accepted that something is "an" activity and not a set of activities, don't bother estimating. For all practical intents, size is effectively constant at that point. What matters is flow, not size.
----
For the above sample, also study how long passed between the "go" decision on the task and when it was actually released to customers. In a stable team, this number will be eerily close to the theoretical based on Little's law referenced in the parent comment.
Oh, and you shouldn't focus on man-hours. Work with calendar days. Not only does that simplify mental arithmetic for everyone, it's also the only thing that matters in the end. Your customer couldn't care less that you finished their functionality in "only" 6 man-hours if it took you six weeks to get it through your internal processes.
----
Fun follow-up to the size experiment: now ask someone intimately familiar with your customers to estimate the dollar value of each activity. You might find that while all activities are practically the same size, they'll have very different dollar values. That's what you ought to be estimating.
For example, at my day job my task is to implement banking. The day to day tasks change... day to day. There aren't a "number of tasks remaining in the queue," since whatever I'm doing is what I'm doing.
One could say this is poor planning. But due to the nature of Big Banks, each task is usually blocking the next one -- in other words, it's not possible to discover or plan what you need to do next, until you've finished currently.
An example of this is when we realized we didn't need to run our banking API using $BigBank's test environment. Their test environment was ... uh ... well, let's just say, when we realized that we could simply switch on "production mode" and bypass their test environment altogether, we collectively facepalmed while rejoicing.
It wouldn't be possible to add "switch to the production environment" into the queue several days ago, because we didn't discover that we could do that until yesterday during our biweekly sync call.
I'm sympathetic to your writeup, and I like your recommended approach. But I just wanted to point out a realistic case of it failing. But in fairness, I think every estimation approach would fail us, so don't feel singled out. :)
Perhaps your approach will work in most cases though, and I'm merely stuck in a twilight zone special case.
If this never happens, then you have some invisible queue, as you do have things to do next.
As far as your example, that's a great example of a task that seemed like it would take long, and ended up being very very short. Can you describe why this would be bad to add into your task system?
This is now data for your estimates of future tasks, as this will probably happen randomly from time to time in the future.We expected to turn on prod once testing was finished. But we ended up discovering that prod was the only correct test environment, because their test environment is rand() and fork()ed to the point that it doesn't even slightly resemble the prod environment. Hence, "prod am become test, destroyer of estimates."
So for 5 to 7 business days, we'll be building out our APIs by "assuming a spherical cow," i.e. assuming that all the test environment brokenness is actually working correctly (mocking their broken responses with non-broken responses.) Then in 5 to 7 business days, hopefully we'll discover that our spherical-cow representation is actually closer to the physical cow of the real production environment. Or it'll be a spherical cow and I'll be reshaping it into a normal cow.
By the way, if you've never had the pleasure of working with a $BigBank like Scottrade, Thomson Reuters, or $BigBank, let's just say it's ... revealing.
The question for scheduling prediction is: what distribution of time will it take to mark any task in this queue as FIXED/INVALID/WONTFIX/OBSOLETE/etc? The queue can have any amount of vagueness you want in it.
Regardless of the embedded work, regardless of whether or not it changes, becomes invalid, doesn't exist, etc - these are all probability weights for any given task/project.
I don't expect my team to know everything about a project on day 1. I just want them to know enough to start and provide a "good enough" estimate.
I do expect my team's estimate to get more accurate the further they get into a project.
What you are describing is not ambiguity, it's total variability. If your future is 100% random, it is, by definition, impossible to predict. Such a state would also mean a total absence of direction/vision. Predicting dates is not only impossible but not a question you can ask, since you don't know what's next.
What I'm going to challenge is that you're effectively in such a case, because I don't think it's true.
> One could say this is poor planning. [...] in other words, it's not possible to discover or plan what you need to do next, until you've finished currently. [...] because we didn't discover that we could do that until yesterday during our biweekly sync call.
The example you're giving *is* poor planning. You're going into execution without validating base assumptions. That you discover the specifics of a dependency that late into the game is that you're going into it without a plan. I'm not judging, in your case maybe no-one is asking for any sort of accountability, and just executing is the best recourse with the lower overhead. But the fact that you can't estimate isn't due to the environment, it's due to the fact that you don't have a plan. Some of the companies I worked with are fine with that, most are not.
Choose a finished tasks time randomly, once per remaining task in the queue, and add this up to be a single estimate. Do this 1000 times or so and get an estimated distribution of completion times for the current queue.
This type of thing is covered extensively in Evidence Based Scheduling[0], and is one of the reasons I still think FogBugz ' power is misunderstood.
[0]: https://www.joelonsoftware.com/2007/10/26/evidence-based-sch...
That being said, I also despise tracking time!
…Suppose you could move that technique to story points (or whatever unit of measurement) though you would lose a ton of precision.
https://en.m.wikipedia.org/wiki/Burn_down_chart
(You'd arrive at exactly this method if you found that there were no easy to find characteristics of tickets that could distinguish them as extra-long or extra-short)
[1] https://en.wikipedia.org/wiki/Ridge_regression
On a team of 4 or 5 people, where people go sick, take vacations, leave the company, new members join... each of these events has a big impact on those metrics. Which again, becomes a lot of work and effort wasted.
But yes, probably this is a viable option on larger teams.
"This post will teach you how to replace estimations and guesses with a Monte Carlo simulation."
as though the output of a Monte Carlo simulation were not an estimate, but some higher truth. The process of simulating many different outcomes with a touch of randomness gives the layperson the impression that we are really _doing_ something, when in fact we could have obtained the exact result (no confidence intervals needed) of Monte Carlo by quadrature (although admittedly that could be tedious).
In the end, the method outlined in the link is an estimate, like any other, and is not necessarily better or worse than any other estimate. It erects a straw person and then misuses mathematical terminology throughout.
It sounds fancier to say modeled using a Monte Carlo simulation, but in reality it is much easier than working with joint and conditional probabilities.
It also ensures that the final estimate becomes on average correct, which is much better than most projects which tends to under estimate the effort it will take. Although, under estimating projects is often a feature managers wants since it makes it easier to sell it to customers, ie "My engineers estimated we can remake Twitter in a weekend, I think we should focus on that!".
Anyway, the point is that if your manager complains about your estimations being on average off then he is just incompetent and should have applied a Monte Carlo simulation on it instead of asking you to perform an easily automatable task.
Indeed, "estimations" might not be the best word as you're still, in some way, estimating with the Monte Carlo approach. I thought of this differentiation because I consider the MC approach to yield a forecast, while an "estimation" is the typical term used in agile settings for "guessing how long/much work it will take.". I'll think of ways of making this distinction clearer.
As for the misused mathematical terminology I guess you're referring to "confidence intervals"? If so, I understand it also has a clear definition within statistics, but I used it in the broader sense. I'd be grateful if you could point out which terms you'd recommend replacing and what you think I could replace it for.
Thanks a lot.
Not in some way -- literally. The samples are averaged, and as the number of samples goes to infinity, the average converges to some true value. In the case of one of the histograms you plot, you are averaging an indicator function .
> I thought of this differentiation because I consider the MC approach to yield a forecast, while an "estimation" is the typical term used in agile settings for "guessing how long/much work it will take.". I'll think of ways of making this distinction clearer.
It seems to me the distinction you care for is to provide not a single value ("estimate"), but a distribution of values. That's not specific to Monte Carlo.
> As for the misused mathematical terminology I guess you're referring to "confidence intervals"
That's certainly one of them. I even misunderstood what you were doing and said you don't need confidence intervals, but in the way you're using them those again can be obtained by quadrature.
My main gripe remains the language, and the presentation of the method. You could most likely obtain your results by hand, i.e. with pen and paper, and you would still get a distribution at the end of the day. The advantage of running Monte Carlo is that it simplifies this process. You could illustrate this by getting the distribution of the sum of two independent uniform random variables in two different ways: by doing a convolution, and by MC. Wow, MC was so much easier, anyone can do it, and it can handle arbitrarily complicated distributions (in principle).
I'll take some time to digest and think through changes, and either edit, or write a clarifying appendix.
Just wanted to appreciatively note your use of inclusive, non-gendered language here.
Traditionally, net present value calculations are done with single point estimates. For example, analyzing a rental property we want to buy, we'd estimate the vacancy rate, interest rate, property appreciation, maintenance expenses, and do all of that on a cash flow time-adjusted basis.
There's variability on every aspect of the through, and understanding how much can easily swing your decision.
Traditional methodology is to pull the trigger on the action if the net present value is greater than 0. We put in a variety of estimated factors, ran it through Crystal Ball (simulation software), and found a 75% chance the NPV would be negative. It suddenly isn't the safe investment it looked like originally.
---
In practice, it's tough to get people in business to understand uncertainty. They gravitate towards "the number", which is almost surely wrong all the time.
GIGO
A very nice thing about Monte Carlo simulation is that at the end your distribution of results are all within the feasible range. If you do error propagation using uncertainty on your parameters, you can get non-sense results.
For example, suppose you have a pendulum, you simulate it with error propagation and you have a non 0 probability of having an increase of the energy in the system over time.
It is easy to spot this in such constrained example but with more complex models, this is sometimes pretty hard to control.
I guess Monte Carlo helps provide conservative estimates behind a façade of rigour, but the truth of the matter is that in the end it's still GIGO.
Any empirical distribution only reflects the empirical measures that were used to generate it. If you bundle everything from the time it took employee A to walk the dog while allocated to project Foo to the time it took employee Z to fix a nasty Eisenbug while allocated to project Bar, and Foo and Bar used totally different tech stacks and team members and even approaches to project planning, that distribution is meaningless in estimating, say, how much time it will take employee G to implement a React widget.
I’ve been using a somewhat meaningless Monto Carlo inspired approach to project planning for a while in my consulting company. The vast majority of the projects I do have only a small amount in common with previous projects, so when I’m estimating I’m informed by past estimate vs actual numbers, but not really relying on much other than intuition and gut feelings from past work.
My basic approach is to estimate 2- or 3-sigma upper/lower estimates for each task, modelled (incorrectly) as a Normal distribution in hours, and then sum the random variables to come up with a final distribution (whose variance is quite a bit smaller than what you see on the original bag of tasks). From there, if I am making a quote, I’ll quote at +3-sigma x hourly rate as a high-end effort estimate. If it seems like a very meeting-heavy client, I’ll either add meetings in as development tasks or just pad it out by a %age or fixed hours/week.
This technique has worked amazingly well for me, and it’s been quite rare that I blow the estimate, and in the… one time I can think of, we missed by very little and there were tasks that we hadn’t thought of when we did the estimate.
To your point though (with walking the dog), there’s a really key thing that this process doesn’t capture, somewhat on purpose: the resulting estimate is in effort-hours, not delivery date. While modelling per-task effort hours as Normal is suspect, modelling task delivery dates as Normal is completely irrecoverably wrong: delivery dates, in my experience, only ever slide in one direction. People get sick and the project slips a week; people don’t ever get super healthy and effectively knock out 80 or 120 hours worth of tasks in a week.
I honestly haven’t found a good way to estimate delivery dates very well. At one point I did put together some regression on my “actual billed hours per week” based on my billing, but ran into the same problem. “Oh, my dog died and my mother-in-law got sick during that project.”
Someone who’s better at statistics might have a better way to model that as a high-skew distribution, but when I tried doing that myself I ended up with a distribution that didn’t feel like it did a good job of capturing the non-negligible long tail of things that slow down calendar estimates without burning hours.
Could you maybe provide an example with figures?
Thanks
Only if you do it wrong, generally. Events which are impossible under some hypothesis should have zero probability under a model for it and not be sampled.
One of my favorite quotes is one that was made famous by Will Rogers and Rita Mae Brown:
"Good judgment comes from experience. Experience comes from bad judgment."
I have seen this "attributed to Nasrudin," which means no one really knows who came up with it, the first time.
Unfortunately, that seems to be the norm in almost every place that I'm familiar with.
In other words, get the people who want a number to tell you what number they want, then use your best efforts to scope the effort to one you can be pretty confident will fit.
> This simple example shows how the net present value may lead the firm to take unnecessary risk, which could be prevented by real options valuation.
Even with a monte carlo on NPV, you may still make a more risky decision than needed; as decisions are rarely just "do or dont" but can be sequenced pending more information.
People try and use nonsense like Fibonacci numbers to imply uncertainty, but then just add up all the numbers to get a number with no uncertainty measure.
The most accurate project plan I was ever involved in had only 3 values that could be assigned to a piece of work during the early estimation phase: hours, days, and weeks. Each of those was then turned into a range of possible hours they could represent, with that range expanding as you got into larger units. You could then slice and dice the numbers however you choose to get anywhere between the most pessimistic timeline to the most optimistic timeline. Probably unsurprisingly the project was delivered somewhere between the two.
Even if you don't have any idea about the uncertainty we already have a crude way of measuring it - planning poker! Just record everyone's guesses instead of throwing away the uncertainty information. There's a huge difference between everyone guessing 5, and some people guessing 1 and others guessing 20.
I agree about points being stupid though - there's simply no way to avoid it being converted to/from time, because that's the actual unit of work.
https://en.m.wikipedia.org/wiki/Knightian_uncertainty
By definition, "uncertainty" is the thing that does not have a PDF ... You can quantify risk based on assumptions you make about underlying probability distributions from which you are drawing. Far too often, even in Monte Carlo, people decide to work with the easy distributions instead of the most appropriate distributions.
The "easy" distributions tend to admit actual mathematical solutions which means Monte Carlo is helpful if you can't do the math but not strictly necessary.
Monte Carlo shines when you cannot get nice solution and there is your opportunity to not be constrained by the need to do so: Draw from appropriate distributions.
Also, know the difference between LLN and CLT and when to appeal to which to justify your methods.
Good rule of thumb even if it sounds overly simplified.
[1]: https://www.econlib.org/library/Knight/knRUP.html
[2]: https://www.jstor.org/stable/4538116
For me it tells that nature of work is such that we just can not estimate work amounts. We are just pretending that it can be done.
With programming, one random variable can add days, weeks or even months of work.
It was a hybrid between the approach proposed in the article, and the "always multiply by N" technique.
Basically, FogBugz always asked you for your own best estimate. It tracked how much longer the actual time to completion was compared to your best estimate, and learned a simplified statistical distribution of your personal multipliers (e.g. in 80% of cases your estimate was 20% shorter than actual time to completion, in the remaining 20% of cases it was 10% longer).
FogBugz used the learned distribution to perform Monte Carlo simulations, and presented you with estimates of actual time to completion (e.g. "in 95% of simulations the project didn't take longer than 30 work days"). The great thing about this method is that it automatically accounts for chronic optimists, who always think it'll be ready to ship soon, and underestimate difficulty. The learned multipliers tell the algorithm that the estimates made by the optimist tend to be underestimates, so that it can present you with a "debiased" estimates.
E.g. if something always takes at least twice as long as Developer Anderson says it will, then FogBugz would tell you: "Anderson estimated 10 days, but based on past data I think it'll take at least 20, and it took less than 15 only in 5% of the simulations".
It had a very simple algorithm at its core, but the developers were knowledgable ,and had tons of data about project management. This allowed them to add many empirically verified "embellishments" (e.g. if the algorithm didn't have enough data about your specific multipliers, it assumed that your multipliers were about the average of your coworkers'), which made the algorithm surprisingly accurate in the end. Certainly better than any estimates most people could produce by hand.
Unrelated, the new owners of Kiln and FogBugz are asshats, I moved my stuff off ages ago, and I’d encourage anyone else still on them to do the same.
https://github.com/boppreh/carlo
This will continually generate samples and show them in a self-updating histogram.This is harder than it looks, because there's too many samples to store them all, and we don't know the range of values at start. But I'm quite happy with the results.
Looking at what we're already doing on outside, improvements inside seem trivial. Good estimator should know that estimation spanning holiday season will be impacted. It should know serial dependency will span wider in time. It should know that set of unevenly distributed tasks (A-many + B-not-many) for small-team-A and larger-team-B will flow slower in time than distribution matching weighted max flow for team A and B. It should know multiple dependency on team X will have lower flow because they're maxed out.
It seems to me that monte carlo projection on timeline from inputs we already have mapped onto graph is really not a rocket science and it should have been solved problem by now. But it's not. Weird.
In order to replace estimation, we are trying Basecamp's Shapeup language and techniques [1]
In a nutshell the thinking is reversed, instead of asking:
- "how long would it take to implement X?"
you ask:
- "how much appetite (in weeks and people involved) do I have for this feature ?"
Then you work with a fix "dead line" and variable scope.
Instead of user stories, they call it Bets, and they give themselves 2 weeks to shape bets and 6 to implement them (these two cycles can happen in parallel), during those weeks is about leaving the design at the Mockup/sketch level, and identify with help of business and technical people all the possible risks, so the Bet safe becomes a safer Bet.
We are still in the first cycle, and still a lot of questions and doing our own little customisations, but everyone response seems to be positive with the way of working, curious if anyone else has tried succeed and fail.
- [1] https://basecamp.com/shapeup
Overlords - “How much money do you need for this project?”
Me - “How much money can we spend?” / “How much time should we spend?”
Friend- “Look at how much money that CEO earns!”
Me - “What will you give up in your lifestyle to do what the CEO does?”
The only challenge though in these situations is that most people aren’t open enough to do “Think in bets”. They either avoid the question or don’t even attempt to come an answer (despite them not spending a single dime!).
Email is in profile if you want to chat more on trade notes on it.
We've been doing Shape Up since July 2020 and have yet to fail to ship what we shape for each 6 week cycle, and while there's sometimes a bit of a crunch in the last few days before a release, everyone is generally working at a pretty relaxed pace.
My favorite side-effect of working this way as a leader is that by batching 6 weeks worth of planning effort, the team (and myself) can just be generally left alone to get into it without having to decide on the next priority every single week. Combined with async check-ins (that the team provide themselves every couple of days via Basecamp) instead of daily/weekly stand-ups, everyone generally has an empty calendar at all times.
The author first talks about the waste of time that is upfront analysis/planning or adding a buffer for contingency and how Monte Carlo simulation fixes for that, but then the caveats seem to be based on an established team, of a fixed size doing similar style work.
For me, this stage of a project has never faced big hiccups in estimation & our teams are normally quite good at knowing "what's a 2 vs a 5 point story" — if there _is_ a big discussion in a planning meeting at that stage, it's normally because one person has overlooked a key requirement (or the other is assuming a requirement that doesn't exist) so it's a good opportunity to solve for that.
To use the analogy, we're probably quite good in established teams at estimating how long it'll take to write 10 blog posts. The trouble comes when someone says "The latest blog posts aren't generating enough interest. Can you estimate how long it'd take for you to write 10 poems instead? We want a mix of iambic pentameter and maybe some trochaic tetrameter (TBC). We've gotten budget approval to increase the size of the team, so we'll have 5 new people (off-shore) joining next week so we can go faster. Also, some of the old blog posts seem to be getting lots of visitors in South America, so we may need to translate these to Portuguese/Spanish too. When can you give me an estimate for all that?"
The approach is slightly different then the article above describes. Instead it has each engineer go through calibration exercises until they can fairly accurately produce 5th and 95th confidence interval estimates. Then each engineer provides 5th and 95th confidence interval estimates for each item that needs to be worked on. Those confidence intervals are kept separate. You can then run a Monte Carlo simulation where each piece of work is weighted randomly assigned to each engineer that provided estimates for that particular item, and randomly picks a number based off their provided confidence interval estimate on how long it ends up taking them to complete the item for this particular simulation.
I was on a small team that used the above technique. We were in a large company trying to launch a new product, and given manufacturing lead time, and seasonality of the market demand, it was very important that we could provide a good estimate to the business when we could have the software portion of the MVP completed. The business provided us with what they thought the MVP features were, we further added in engineering tasks that weren't business facing, but needed to be completed. We confidence interval estimated those, and then also confidence interval estimated our personal vacation days, sick days, as well as a bucket of "unidentified work". The 80th percentile Monte Carlo simulation put us out a little more then a year. Our actual delivery was off by only a week from the Monte Carlo tp80; I don't remember in which direction, but it wasn't consequential to the business.
There is also a short chapter on estimation tools using e.g. Monte Carlo simulations. One tip which is relevant here and that is also more or less presented in the article and probably obvious :) "TIP #65: Don't treat the output of a software estimation tool as divine revelation. Sanity-check estimation tool outputs just as you would other estimates."
There's a lot you can do with a monte carlo model. Tornado plots are something I highly recommend for understanding risk profiles around any complex task
[0] https://github.com/boozallen/argo/releases
If you don't want to go through the process of logging in via Jira there is a demo where you can manually input the data needed.
See: https://agilytics.leanloop.co.uk
I agree with the author that agile world zero effort estimates are pointless though.
"... engineers determine it by licking the tip of their finger and putting it up in the air.
Unfortunately, the only way to win at the estimation game is not to play it." "
n <- 1e06
table(round(runif(n,1,6))+round(runif(n,1,6)))[6]/n
To break this down:
`n <- 1e06` assigns 1 million to n.
We want two six sided dice so we sample a random uniform (continous) distribution and round to a whole number:
`round(runif(n,1,6))`
This will make a vector of 1 million integers representing random "rolls" of the dice. In R it is really easy to pair-wise add two vectors of the same length so to get two dice we do:
`round(runif(n,1,6))+round(runif(n,1,6))`
Now we have a million rolls of summed pairs of dice. We want the frequency count of 7s. We can get this by getting the frequency count of every dice with `table()`:
`table(round(runif(n,1,6))+round(runif(n,1,6)))`
Which will return a print out of frequencies for every number that appears in the vector.
We can select 7s by pulling the 6th index and then dividing by the `n` dice rolls to get the frequency:
`table(round(runif(n,1,6))+round(runif(n,1,6)))[6]/n`
We can do the exact same oneliner in python using numpy and the Counter() function from collections:
`Counter(np.around(np.random.uniform(1,6,size=10*6))+np.around(np.random.uniform(1,6,size=10*6)))[7]/10*6`
Will return the frequency of 7s in python3.
Its flexibility and power of estimating (otherwise maybe untractable) metrics comes at the expense of transparency, potential simulation noise and difficulty in estimating accurate "what if" scenarios. The risk is that one might get out simply what one assumes.
Here a list of questions you can ask to minimize the associated risks and steer your development:
* Do I have a good, self contained, description of the system that I want to simulate? Is it even possible to define it in practical terms?
* Do I have historical data that can pin down its stochastic behavior? Can I estimate a statistical model reliably?
* Does the uncertainty around model estimation justify retaining the full model or could I possibly simplify it and obtain semi-analytic results?
* Can I validate my estimates out-of-sample
[0] - https://www.getguesstimate.com/
1). You're only sampling a tiny fraction of the space of possibilities. The example using two dice having 36 outcomes makes it seem like you're in a very well behaved world. But the total number of possible samples is n_possible_values ^ n_dice, so with three dice you have 216 possible outcomes, with 4 dice 1296, with 10 60466176. The growth is exponential in the number of variables, with predictable results.
2). The distribution of delivery times is not normal, experience shows it's much closer to a power law with a small number of tasks exploding beyond any reasonable expectations.
3). No critical path. A task can't be completed before it's critical path is completed. Since you allow subtasks to vary your critical path needs to be calculated for each run. With 1). and 2). it basically means that you have no idea if you've gotten a good representation of the sample space of critical paths.
If you run a simulation that samples a tiny fraction of your probability space you have no idea what monsters lurk in the background when the problem space is one prone to explosions.
In short: beware of tap dancing in minefields when blindfolded.
Some tips I have for people interested:
* Get about two months of velocity data in place first. * Run the simulation on a regular basis. Update your estimates every week/sprint. * It helps to have individual work units be "sized" as consistent as you can. (E.g. each ticket is 2-3 days, a consistent pointing system, etc.) It's ok if anomalies happen from time to time. * Dates will get more accurate the further you get into a project.
Some things I've learned from doing this exercise:
* When work units are sized to be 2-3 days of effort, engineers complete about 4 units/wk on average. * Don't shoot for the perfect estimate at the start of a project. Plan as much scope as you know and enough to get started. * As new scope is found, update your project plan and estimates.
[1] https://www.omnigroup.com/omniplan/features/#Monte-Carlo-Sim...
https://www.macupdate.com/app/mac/30431/risk-engine