OT: I like the "thinking systems" infographic but the fact that the green areas are few pixels off is annoying me to no end. I hope it's not a new trend.
In my experience, deadlines set by higher-ups long removed from (or never having) the technical chops to be determining the deadlines in the first place. VP's and Directors are often the ones dictating the direction, which is great, but then also introducing deadlines, with helpful input from Directors who ALSO haven't touched code in many years.
Generally that results in one of two things:
* Product delivered on-time with massive technical debt.
* Product delivered late with massive technical debt.
Frankly I don't know if adding front-line engineers to the deadline decisions is going to make the issue better or worse, but fundamentally having non-technical or formerly-technical people defining deadlines definitely doesn't work.
I'm a software developer, and I know this is a bit heretical to say, but I think sometimes we need to set goals that are overly ambitious so that we can work hard to meet them, even if we end up being late. I've been a contractor to a couple companies that didn't have real goals and there is this kinda easy going attitude that results in many things taking 4x as long as they could and often a lot more office politics as people spend more time jockeying for position than cranking out features or squashing bugs.
We need to set realistic goals so that we can work hard to meet them. The mindset you are describing is part of the reasons that we have these problems in the first place.
My experience, a ‘my goal’ gets quickly interpreted as ‘you committed to this’. ‘I can try’ projected as ‘You said you will get this done’. I’m all in for trying my level best and I have seen the rare slip ups penalized, criticized and highlighted to such a degree that it is just irrational for any dev to ‘try’ or be ‘ambitious’.
And they spoke among themselves, saying, "It is a crock of shit, and it stinketh."
And the workers went unto their Supervisors and said, "It is a pail of dung, and none may abide the odour thereof."
And the Supervisors went unto their Managers, saying, "It is a container of excrement, and it is very strong, such that none may abide by it."
And the Managers went unto their Directors, saying, "It is a vessel of fertiliser, and none may abide its strength."
And the Directors spoke amongst themselves, saying one to another, "It contains that which aids plant growth, and it is very strong."
And the Directors then went onto the Vice Presidents, saying unto them, "It promotes growth and is very powerful."
And the Vice Presidents went unto the President, saying unto him, "This new plan will actively promote the growth and vigour of the company; with powerful effects."
And the President looked upon the Plan, and saw that it was good.
Even with technical higher-ups, you're not guaranteed a fair fight.
I've worked with Engineering VP's/CTO's that are pressured by deadlines and promise unrealistic features. The key element needed in this is a 'leader' that can push back and manage expectations.
Having worked in the trenches as a software engineer and also as a CEO has given me respect for the VP level as well as the engineers. When the VP sets a deadline, it’s driven by the survival of the company - aka release by June or we can’t make our numbers and have to do layoffs. When the engineers push back, they often do so without an appreciation of the business reality, aka those numbers pay their salaries.
Engineers hate releasing poor quality products, but the reality is that customers often will buy and use something laden with defects rather than nothing at all. And in any case, after the initial botched release, you can always fix stuff. See Apple.
I agree with you in part. However, engineers are more than capable to also take into account for planning the survival of the company if management provides the proper data to them. I think this is one of the problems, management usually hides information from the people they manage and without that data certainly plans can go in the wrong direction.
That's a fine argument for a bootstrapped startup with one product, but isn't meaningful when talking about the tech titans that suffer from the exact same problems.
This is the same type of argument that makes short-term profit take precedence over long-term success. Why in the world are we getting a deadline of 4 months? You, as an executive, had to know burn rate was blowing through cash a lot earlier than that.
Engineers hate releasing poor quality products because we know we'll eventually have to fix them and we will likely be given almost no leeway time-wise to fix the actual debt, as time will always favor ugly hacks that "work" over well-written code that prevents the problem in the future.
> the tech titans that suffer from the exact same problems
The tech titans aren't monoliths with a collective hive-mind. Each business unit and team is responsible for projecting revenue targets and then hitting them, same as any other kind of business. And, yes, if the business unit managers fail to hit their financial goals too many times, their project will be cut and the team reassigned or laid off, same as a startup running out of financial runway.
Have you considered maybe you're treating your engineers like children? Why aren't they cognizant of the business realities? If they aren't on the same page as the business side, that's poor leadership on your part. I think everybody can appreciate that in order to keep the lights on sometimes you need to be in crisis mode, but if the company perpetually operates in crisis mode, then maybe the engineers aren't the problem.
Here's the reality, and maybe you've forgotten: The engineers are the ones who are going to get dumped on, and you and the VP's are the ones who are going to reap the benefits. Never forget that.
Ideally, yes, a deadline is set for a specific business reason. But I cannot count the number of times a deadline has been set for a project that is just utterly arbitrary, proven by the project releasing successfully on the deadline and then being immediately abandoned and forgotten about. Feature launches without marketing, website redesigns without fanfare, architecture refactoring that sits on the shelf for a month before actually being deployed.
Not that I'm claiming that all executive decisions are clueless, most people are just trying to do a good job with limited information, and mistakes happen. But you run into enough executives without respect that runs in the other direction, and you start to question all deadlines.
(Which isn't necessarily a bad thing. In a healthy environment, questioning the feasibility of a deadline can lead to appropriate descoping of overreaching features, reduction of risk, etc. But the situations that lead to this kind of suspicion are rarely healthy to begin with.)
> When the VP sets a deadline - it’s driven by the survival of the company
This is everything that is wrong with software.
What is software really? It is translation of business process and need into code - I spent the first 10 years of my work life mastering code, I will spend the rest of it understanding whatever business I'm working for.
Assuming you have some experienced engineers (not mangers) on your staff then having the vp set a date (and defacto choose the project) is absolutely backwards.
Bring the problem to your engineers (money) and don't look for a moon shot look for 10 smaller projects that get you 1/2 way to the goal in 1/10th the time. Don't think of it as a "hack day" think of it as a hack week or hack month.
If you have a large project and it is to "save the company" (or push it forward) and your treating it like a must do rather than a foundation (and taking care) your creating debt and starting a cycle that you will never pay off.
Director who still writes code here. You're correct, but there is a vicious cycle here: when a project is delivered late, some managers respond (incorrectly) by artificially tightening the next deadline, hoping that when the team invariably slip, it'll at least get delivered by the original deadline. This never happens. The team does hasty planning and dives into implementation as soon as possible (to save time) and ends up delivering even later.
What worked for us is what's basically mentioned in the article: taking a more structured, slightly painstaking approach to estimation. It's more tedious than gut feel story point estimation, but a lot less painful towards the end of the project.
Software is up there as the most complex thing ever created. People have been putting up houses for thousands of years, And modern big bridges for hundreds.
Big software is ~40-60 years old. Its hard to have rules of thumb and best practice that will deliver the 'on time outcome'. Give it another 100 years.
Most dev work right now is still just attempting to deliver correct software. let alone on time or on budget.
Also, generally, much less need. The Shuttle is a physical system, and can be directly observed and components either individually (unit) or collectively (integration) tested.
"Debuggers" are the monitoring, telemetry, sensing, and forensics equipment, data, and processes applied to each test, launch, and component.
We have debuggers because code is otherwise a black box. Debuggers are a visualisation technology.
In addition, as requirements get more complex and users expect more polished products, we end up relying on more and more open source/reusable infrastructure and libraries. Those tools allow us to not reinvent the wheel and accomplish more in the same amount of time, but they also add to the complexity of any software system.
House building has become somewhat more complicated over the years as building codes are made stricter and new techniques are developed, but for the most part it is just a switch to a new way of doing things vs. in the software world where we will continue to pile on more and more levels of abstraction, but can't reasonably expect to have at least some kind of understanding of the underlying layers.
Nonsense. Software is not more complex as other engineering jobs. It is much less complex than e.g. architecture, which is a generalization over multiple technical jobs. In general you cannot trust a young architect with less than 20 years of experience. You can appreciate the end quality because of the ambition, but with the time and budget you need to be careful.
But other typical engineering jobs also, which are not as simple and logical as software.
I worked professionally in a couple of high-tech engineering jobs, as architect, surveyor, city planner, civil engineering, construction, stage design, movies, automotive engineering (F1), games and at internet service providers. (but no airplanes, nuclear power plants or rockets, sorry. Lots of hospitals still.)
The SW development jobs compared to that were always the easiest, most creative, most rewarding, and best paid also.
The problem with SW planning is pure lack of experience and poor engineering practices. A good SW engineer can hold his estimates, just as an experienced architect can hold his date and costs. In practice it rarely happens, I only know a couple of successful projects with smaller companies, but the good and big companies mostly deliver on time and budget.
As you mentioned houses and bridges. With houses you don't change engineering paradigms every 5 years as it might fancy you. You never try out new materials which are not tested over 10 years at least. These things need to stay up for 50 years at least. With bridges you cannot really plan them properly. Well, you can. But a typical bridge engineer multiplies the safety factor with 10, leading up to 10x more material as needed, as you can hardly estimate the dynamic peaks and resonance frequencies. With highly dynamic forces such as e.g. in a F1 gearbox, a huge spring with max 4000 Nm forces on your shafts. There you cannot apply any 10x rule. This thing needs to be perfect. You plan to avoid the peaks at all, avoiding the dangerous frequencies. Comparable to wind forces on a bridge. But in practice you look at best practices. You build dynamic physical models and simulations. And measure. And apply safety margins. E.g. with the big steel bridge next to my home, the famous "Blaues Wunder"in Dresden, the general public and politicians never trusted their engineers with this fancy new steel design, so it had to made 10x bigger and heavier, and even on opening day they invited the whole town to test it live, with lots of heavy trucks, a tramway and pedestrians walking over the bridge and jumping on it all at once. It stills stands today, but could have been made much simpler and elegant by todays knowledge.
Now compare that to modern SW engineering. You got a fancy new framework and language, looking for engineers with at least 10 years experience in that, with the framework existing for 2 years. You got massively overhyped tools which rarely deliver on their hype, complete lack of security, memory safety, type safety or concurrency safety, completely outdated API's like blocking IO or POSIX or threads, and undesigned monsters like C or C++. Good luck with that. That's of course a death march.
Still with proper tools and proper management it's much easier than everything else.
In automotive everything was properly planned, with traditional waterfall. Everything worked as planned. Even if it was hugely complex, much more complex than a simple OS.
In my own SW I never miss my milestones or overdo my budget.
In architecture we always did. With huge costs up into billions. Only once we went under costs by 200 millions with the biggest building construction project in Austria. This was a miracle then.
In my experience talking to hundreds of software engineering managers, running a software project management business, and being involved on software dev teams for 20+ years, there are two reasons why teams don’t ship or ship poor software.
1. Poor engineering planning. The way to do this well is to break down a feature until they’re at about half-day sizes tasks. Identify any obvious risks or ambiguities with the technical approach and communicate them to Product and figure out how to de-risk them. Communicate costs to Product so PMs can make cost/benefit trade offs (eg “if we built it the way you specced it it’ll take two weeks. But if you make these trade offs it’ll take two days.” As a Product Manager I might not want the feature if it costs two weeks but do if it costs less than 5 days. Without costing the features in dev days I can’t makr this trade off).
2. Failure to make trade offs that drive towards shipping software. The way you ship is by declaring it’s done, even when it’s not really done. When code is being written unexpected issues always arise. It’s Product’s job to make hard trade offs that drive towards shipping. Eg “that’s an edge case bug, let’s punt it to v2” Or “let’s cut that nice to have feature and squash the showstopper bugs and ship”.
People bring up impossible deadlines set by management. In my experience most deadlines are movable and a strong PM/Eng team can convince management to push a deadline back. And if a deadline can’t be moved, it’s all the more important to know how much features will cost and make hard trade offs to get the product out the door.
Wholeheartedly agree with #1. I'm fond of saying, "If it takes longer than four hours, you probably haven't broken it down enough. Find the monsters hiding inside your big tasks."
Well, they're just checkboxes inside the main tasks. You just think through how you're going to perform the task (which you'll anyway have to do when you get close to actually doing it), and note down the sequence of steps. Then you try to estimate each step. Sounds simple, but it works.
Why bother estimating? you have the checkboxes, now just get to work. If the pace of checking those boxes is slow then you can start cutting things out. Estimates are for managers, much better to just ship it already.
Because if you haven't built it yet, you can't ship it already.
But you are right that estimates are for managers. Managers should watch their reporter's progress and use that as evidence to build evidence. If the first task of 20 took 1 week, it's a 2year project. Update estimate as time and progress and work remaining increases.
There are literally hundreds of methods and tools; it doesn’t matter which one(s) you pick.
I’m quite serious.
The thing you need to realize is that you’ll get it wrong, no matter the tool... until you invest time in reviewing past estimates and plans.
People think because they’re engineers they can just magically make estimates... but its a skill. If you don’t have a feedback loop to measure your progress and adjust your actions, you’ll literally never improve at it.
...that’s why even ‘Senior’ developers and ‘technical leads’ are often terrible at this; they’ve simply never bothered to seriously make an effort to learn how to do it.
You break it down (effectively doing the work) as part of planning and after that you pretend that you have a working system. Make sure not to go into details and/or “take it offline” until the developer is tired/annoyed enough to just agree with you pipe-dream estimates /s
I see, so you simply hire superhumans who finish the project before UX mockups are even available. Simple stuff. I think I'm ready to start my own consulting business
It wouldn't be easier if we didn't have to keep learning silly new words for everything. I used to do prototypes and proofs of concept, now I have to do spikes.
Spikes are not necessarily proofs of concept or prototypes. A spike is simply a task aimed at gathering information.
Spending 30 minutes researching four different libraries to see which one to would be a spike, for instance, but it doesn't necessarily involve building a proof of concept.
> How do you estimate how much time a task will take when it's a novel idea
In my experience, they just pretend it doesn't exist and remove it from the "plan". Where I see this the most is in figuring out how things work. For some reason, non-programmers have it in their head that once you've finished college, you ought to know (even though they don't) how everything that goes on a computer works instantaneously. We're going to use Apache Kafka! Ok, need to spend some time figuring out how that works. Um, ok, that should take like an hour, right? Because there's no time in the budget to learn anything. If you push back, they say, "ok, never use anything new that you have to learn. Only use the things you already know". That doesn't work either, because the things that we have to integrate with don't work with the old things. Or, "you don't need to know how that works, Bob knows how that works, just ask him. But don't let it take any time, because there's no time in the plan for that" And Bob is sick of all your questions that you'd know the answer to if you'd just spent a two or three days reading the online documentation and installing a test system and testing it out.
You hit on one of my favorite gems, "Bob has done something like that before, just ask him how to do it." The manager is envisioning that the team already possesses the knowledge necessary to get the task done, so there should be zero learning time involved. What this is missing though is that Bob isn't going to be solving the problem for the current task AND/OR Bob didn't even solve a similar problem, he just used the same tool or library that is now being used for the current task. If we had a Matrix like ability to instantanously transfer knowledge then this approach might work, but we aren't there yet.
I'll also wholeheartedly agree with this comment—my company has a long history of shipping on time and under budget. The caveat is that we closely follow #2: we aggressively strike requirements from the overall target if it means preserving the integrity of the project overall, with a commitment to return later.
I'll add this: a product team that ships earns 1) more credibility, and 2) that credibility is reciprocated towards the team in the form of greater flexibility and freedom to set their own targets.
Breaking a task down into meaningful half day task is almost as hard is just implementing it, but usually this is attempted in a committee setting. This is not a recipe for success.
Agreed. Think of how ridiculous it would be to make a half-day task for figuring out X -- sometimes I've spent days, even weeks, trying to figure out how a module works.
In practice, being able to break things down meaningfully and accurately to such a minute level implies that a lot of time was spent in grooming, which means we're just back to waterfall again. (Or the tasks at hand really are just that simple and/or the people involved have a lot of expertise.)
I agree. Saying, "Break it down until it's in half-day size tasks" is fine. Some of the things that might interfere:
1. Are you sure you can do that in half a day?
2. You forgot about this one part of the task that you thought would take half a day.
3. There was a bug in the plugin that you were going to use to do this task in half a day, and you spent three days fixing it.
4. This task will actually take longer, but management won't buy that. Looks like it should be half a day on paper, so that's what we'll say.
5. I've never even done this one thing before, but it sounds like it should just take a couple of hours.
I understand the concept - trying to say something will take a week is even more error prone. But I think most of us are still pretty bad at estimating a budget for tasks even at such a small scale.
When I read that in OPs comment I immediately had to think of "How to draw an owl" (the meme version, not one of the serious online lessons -- https://pbs.twimg.com/media/DcC1YEyVQAA9Lkw.jpg). "You simply break the task down into all its parts, then you just do it. Projects are easy."
Not to mention that that approach only works if you produce another version of something you already produced many times. There certainly are such development efforts, for example, I think anything data- and form based, like data entry + database + some reporting, what used to be paper cards a long time ago, and then punch cars, those type of apps can probably be made using that method.
So that's when they had to build the freight version of the Boeing 747. Then they thought "that was easy", and started the Dreamliner, something very different that they had not done before... (and it even still is an airplane, so not even that outrageously different as different pieces of software can be)
I think the "break it down" approach also faces the problem of the "coastline length measurement".
I'm also reminded of an anecdote from my early days with computers: I had to write an assembler routine (as part of an 8 bit assembler program) that seemed too hard, so I outsourced it to someone else. He didn't know what to do either, so I explained it to him. When I was done I realized I had just implemented that difficult function - I wrote it down in half an hour and that was that. Planning well, finding out what to do, is harder than the actual programming. Again and again, in all projects, I find that the >90% part is finding out what to do. There are stories from projects that lost all their code, years of work, and they thought they were done and finished (business kaputt) - but when they rewrote the software they fond it took them only weeks or instead of years. When you already know what you want and how the end result is supposed to look like (and feel and behave) then writing it in code is the easy part.
> Planning well, finding out what to do, is harder than the actual programming. Again and again, in all projects, I find that the >90% part is finding out what to do. There
So you've advocating an approach where you... don't plan? I'm confused.
Perhaps you're getting too hung up on the half day part? That measure is a useful forcing function that ensures the dev team has thought through the approach in enough detail to identify where the risks are and what smart trade-offs can be made.
I'll go against all the other answers and say that I have seen that in practice. I'm even doing that in practice, every day of the week. A half-day task would even be a big task, most of our tickets are a couple hours maximum.
We practice a lean-laced Scrum with 1-week sprints. And yes, it works. And yes, we go fast. So yes, it can work (thought we have several years of refining our methodology and have a big culture of pragmatism and problem-solving).
Do you mind putting out a number of hours (per dev team) that get spent in Backlog curation, story generation, and other budget planning every week? I think perhaps an important missing ingredient for getting budgets right is actually spending the time to do so. The smaller the company, the harder it seems to justify.
Indeed, for a lot of us in smaller companies, getting our clients to actually consider taking an agile approach at all is a stretch.
For a team of 3 devs + 1 scrum master:
- Backlog refinement/grooming: 1.5 to 2h, 1 dev + SM (+ PO)
- Technical refinement: 1.5h, all 3 devs
- Sprint review + retrospective + planning: 4h, all 3 dev + SM (+ PO)
Story generation is done by the PO, that's most of his job (though we split stories during backlog refinement if needed).
I don't know if it sounds like a lot (I feel like it could), but turns out it's actually efficient, every minute of each meeting is productive because we know exactly what we're going in for, how it will go and what you want to get out of it!
Have you ever been in a situation where communicating and breaking things down takes longer than the actual work?
IMHO most disasters I’ve seen occur when ‘Product’ tries to strictly control the ‘Developers’. It just does not work for certain kinds of projects. It also diminishes the developers’ involvment and leads to a culture of mediocrity and CYAs.
Also, a person that does not have to do the work will always be waaaay more optimistic about how much it takes to do the work.
How do you fix it? Trust and ownership. Product and Dev need to have an interface - a boundry if you will - and the discussions and talk are about defining this interface and setting expectations. Product trusts Dev on implementation. Dev owns the delivery and makes it happen.
This is why agile tries to push the business stakeholders to define -problems- rather than -solutions-. Developers then try and come up with a solution, and pitch it to the business (effectively).
Almost every issue I've had with the business/developer dichotomy (and I use that instead of product because different orgs place product differently) has been due to the business trying to pitch solutions to dev that invariably end up not actually addressing the problem (even simple things like "that's the wrong color" or similar, are because they believe they're responsible for the solution, and they have a very clear idea of what the solution should be, without actually being able to articulate or recognize the problem. Does the color actually matter? If you have a problem that implies it does, yes, if you just came up with a solution on your own and in your head that solution was yellow, no).
Maybe, maybe not. I’m very suspicious of things that are pitched as a panacea for everything.
The real world is a messy place where even with a good team, a good plan and sensible expectations you can fail spectacularly.
If there is one thing that I really like is aggressive validation of the assumptions that are being made + keeping the product as small and lean as humanly possible (and the latter should apply across the board to everything you do).
Complexity and Entropy are preparing the coffin for your work even before you have the idea. You can give ‘em a run for their money, but in most cases you’re going to end up 6 feet under. As soon as you accept this and are genuinely open to failure you will be ready for the opposite.
Oh, I'm not saying it's a panacea for everything. Just that most issues I've seen along these lines were also accompanied with a perspective on the business side that their role was to give solutions, which were invariably insufficient, led to the dev team just doing the minimum (since they had no idea what the core problem was and thus were unable to determine where 'sufficient' really was), and that compounded.
The problem is that product and dev are different people. Product managers don’t understand their product as well as the devs do. devs should manage the product definition and work with feature requesters at a very high level, where the details are the choices of the devs (or senior devs acting as product managers.)
Ditto. A colleague called into question an off the top of my head estimate. A 2 day estimate and tried squeezing me to cut it down more... For work I wouldn't even be doing. Nope. Not going to do it. I tried to call into question his estimate of how long it would take for one of his sales guys to sell the lead he received yesterday. Once he threw out a number I asked why so long? Needless to say the conversation didn't end on a good note. Live and learn. We'll talk it out tomorrow. A quick way to lose trust and respect is implying incompetency and squeezing/trying to manipulate people. Really kinda pissed at him right now.
Re #1, this is kinda the system 2 vs 1 thing in the article. I was lucky enough to have a PM drill me into this. More specifically: The bigger a task is, the likelier it's going to go over time. The bigger a task is, also the less likely each bit is well specified and more likely that there are significant portions glossed over that you'll discover during implementation, which will delay things and/or result in them being done poorly. If you're having a hard time breaking the task down, it's probably because the task isn't well defined enough, so go back to stakeholders and define better. In this model, planning takes a lot of time and effort. Also in this model, research-y tasks are different, and instead get a fixed timebox after which you have to re-decide how much more time to spend on it if at all.
Re #2, part of the problem here I think is that PMs aren't trusted / empowered with that ability or final say. "Stakeholders" get to make that decision, but with ignorance of what it's currently like in the trenches.
In the context of a fixed-time, fixed-price/budget project it will fail to meet these constraints unless the scope is very clearly delineated at the outset, and ruthless scope/change management is applied throughout the progress of the project. And if at some point someone realizes that there was an omission in the scope definition, the time and budget must change.
If the client or management want to change the scope, in any way, the discussion must immediately start with questions such as what other feature are we dropping, how much time are we adding to the delivery plan, what are the consequences on other features or the current design and code?
Even deciding to remove a feature can cause delays (e.g. having to re-structure a test scenario). Or changing a string in the user interface, which may require re-arranging the layout, re-translating the string, suddenly having to manage plurals, checking that a variation of the old string wasn't already used elsewhere, etc.
Well sure, if you doom the project before you start, with a fixed price and scope, it will fail.
The original primary point of the Agile Manifesto is to stop doing that. There's no excuse for it, unless you are required by law as a government contractor, and in thar case no one expects a successful result anyway.
Software is hard because we 'don't know the unknowns'.
It's not like building a building - if it were, we could just schedule it down to each detail. We would be still all be using 'waterfall'.
If we knew how to write the module, we'd just do it, test it, and voila.
But most of our time is spent 'figuring stuff out'. Oh, product has asked for a tiny change (support a specific kind of font), but our libraries don't support that, meaning major possible architectural changes.
The other major unspoken factor is that requirements change. In fact, this is the #1 thing, really. Requirements change because customer's needs change, because business changes because our understanding of the market changes, etc., and this is a reality of our world. So the nature of how we're going to adapt is difficult to predict.
This is why the #1 predictor of outcomes is the quality of developers on the team i.e. experience, intelligence, know-how of the domain.
If you're doing something where there are very few unknowns, well then it should be possible to schedule within reason.
It's not like building a building - if it were, we could just schedule it down to each detail. We would be still all be using 'waterfall'.
I hear that it doesn't work that well for buildings either, as you end up with people in the field detecting bugs in the plans and having to improvise fixes without input from the designers.
For actual buildings, they spend a fortune up front planning.
Soil and dirt samples at $100K - just to do a bid on a project, let alone the actual build.
Can't spend $500M on something only to find you have to tear it down.
Software is not magic, but it does require thoughtfulness and experience ... (hint to all those who think startups require mostly new/young blood - it's mostly wrong, but that's for another post)
> The way to do this well is to break down a feature until they’re at about half-day sizes tasks.
You're identifying what the outcome of good planning looks like, not how it's done. What prevents this is that teams read "agile" as "make it up as you go along," when in reality if you're spending more than 10 minutes discussing a feature, make planning that feature a sprint task.
> In my experience most deadlines are movable and a strong PM/Eng team can convince management to push a deadline back.
It's not generally the team in that room discussing those deadlines, though, it's software development manager and product manager. The worst managers are software engineers who get promoted to management, and when they get into those meetings they look at their shoes and are railroaded by upper management.
You really do need a manager who has the intestinal fortitude to go to bat for you.
I'd add #3: over-engineering. 90% of the time a simple quick solution will be as good as a super sophisticated and abstracted solution that will be delivered 1 years later, when the underlying problem it is trying to solve has changed (cf the usual tree swing cartoon [1]).
With the exception of video games, I maintain that it's always better to be 10% over estimate on every milestone than to all of a sudden be 50% off of the final deadline.
To work it takes being comfortable saying "that's my/our fault", and my experience is that a lot of people who would rather die than admit fault. It's part of a larger raft of skills around transparency. So they cover up problems hoping for some miracle that never comes (or more cynically, they hope to be employed somewhere else before the shit hits the fan).
If you have deadlines, you already know management is incompetent.
Deadlines presume a known fixed scope. But I’ve yet to meet a CEO or sales team or whomever gets to decide what features go in who will fix the scope if a project.
They are always willing to add features, but can never take them out.
As a long time engineer, and occasional pm. I steer teams to think about the worst case, and then triple that time estimate.
I know this sounds like setting up a team for failure, but I’ve seen it work again and again. It sets clear expectations for quality and delivery upwards and downwards which everyone can agree on.
Once this is done, the easier part is keeping everyone focused, and using all the leftover time well to raise quality.
> This is where System 2 comes in—if we performed a more thorough analysis, these factors would have been considered in our answer. Then it would be clear that it’s much more likely to take 20 or 30 minutes to run to the store instead of 10.
I have encountered an article (can't find the link now, sadly) that claimed that when developers gave estimations, breaking down tasks to sub-tasks actually had a reverse correlation to their accuracy. In other words, their first gut reactions were actually better than estimations given after going in-depth through all the details and sub-tasks.
Maybe because people over-rely on a fine-grained breakdown? When I try that, I miss big stuff. Kind fits in with that Fibonacci estimation from Agile, eh?
> I have encountered an article (can't find the link now, sadly) that claimed that when developers gave estimations, breaking down tasks to sub-tasks actually had a reverse correlation to their accuracy. In other words, their first gut reactions were actually better than estimations given after going in-depth through all the details and sub-tasks.
I've noticed this as well. The last few years my 'gut' estimates have tended to be more accurate (and larger) - but since I can't support them with anything other than "Trust me, I've been writing software for over a decade", they're never taken seriously.
My guess would be that breaking down a complex software system into sub-tasks (esp. when you are breaking things down within modules) that are too fine grained fails to capture the inter-dependencies between sections of the code-base. So, you complete task A and move onto to task B, then realize you have to revisit the code you wrote during task A because task B is dependent on it and some of your early assumptions were imperfect. And then you need to fix the unit tests you wrote for task A.
Bottom-up estimation is probably more accurate for the work to be done, but not for other project overhead. Using data from an analogous project to forecast the delivery date would include that overhead.
Having been involved professionally in software development for about 14 years now, I have to say I respectfully disagree. More planning does not result in better predictions. In fact, it often results in worse ones. That is just my empirical observation.
My best guess as to why, is that there are managers involved that attempt to negotiate the planned delivery time down. They do this, in part, because it's hard to get devs to work late or on the weekend when the project is on schedule, but easier to do when there is obvious risk of falling behind schedule. So, from their point of view, the best way to get the product delivered early is to get the schedule made too optimistically.
Not saying they SHOULD do this, or even that they are consciously thinking this way, but it's what the situation incentivizes them to do, and it's what normally happens. The gut level immediate answer is based on past experience, and the long drawn out meeting produced, System 2 answer, is based on management bargaining the developers down to a shorter timeline.
Do you really want to know why some development teams struggle to deliver on time, budget or at all? Because the majority of software projects out there are utter useless crap, commissioned by people who have no idea what the heck they are doing and focus on the smallest stupidest details, before even getting a decent amount of users or even wondering whether users would like those changes (before implementing them). People who, once the money runs out, will make the project crumble, frustrate developers who have to reimplement the same stupid piece of logic 20 times because "that button looks too big" or "this would be a really cool animation to have" while everything else goes to shit.
We get into software development because we expect it to be a creative, challenging and fun profession that creates value and yet most of us answer to clients or employers who expect us to spend 80-90% of our time working on boring, senseless stuff. You want us to do that? Great! But don't expect high quality and on time delivery.
The real reason behind delays is that we just don't give a crap about your "social network for cows" and we can't wait to save enough money to get the fuck out and either start a business, work for a decent company or start investing.
Apologies but it feels good to rant every once in a while.
Higher up manager here. I fully accepted the #noestimates movement and it is a complete blessing for all the teams and organizations I've implemented it in. Roast me.
Did you do that in an agency as well? I mean it's one thing if you are developing a product, but telling your clients "the project is finished when it's finished and we won't be able to tell you how much it will cost you until it's finished" doesn't really fly in my experience.
Seems like charging a specific fee might help alleviate some of the concerns about that. Though I suppose you still need to know how much money you should charge.
I really hate estimates and have always hated estimating projects. I do appreciate the need/desire for them by some in management. I've tried to figure out how other people do it and have yet to find anything satisfactory. It doesn't have to be an extremely simple process, but I don't really know how one gets better at estimating. The only way to get better is to just understand the domain more clearly but estimating doesn't really help with that.
Kanban does have a cool idea of just attempting to break down work into equal parts and measuring the throughput of these roughly equally sized components. The only problem is actually sizing stories to be equally sized. Some changes don't have any kind of real stopping point of functionality without large changes. It seems kind of arbitrary to chunk it out just because your project management system wants you to.
Me personally: no. But friends of mine owning an agency do this with all their clients. The gist of it is: don't think project, but iterations that deliver value every time you end one. The customer pays per iteration.
How do you prioritize which features to work on? If you have to pick between two features with similar expected value (or similar cost of delay) then you need at least rough relative estimates to make a rational decision.
I hate it when PMs ask me for an estimate of effort on a task I have never done before and I get this question all the time. I get asked to get a new process through a deployment system I've never worked with before and that's fine. How long will it take? That depends on how complex the deployment system is and I haven't worked with it yet.
When I first started working as a web developer, my boss would ask me how long it would take to finish a project. I would give him an estimate time, and he would always double it. As it turns out he was always right. But because of it, I learned to provide better time estimates. Overtime I became more accurate and started to provide more realistic time frames. Depending where you work or who you work with there's a lot more that goes into your day to day tasks than just coding.
Yes, plan, the more planning the better. But set the deadline first, and plan and design around meeting the deadline. Ask yourself, "I have two weeks to deliver this, but if I had to deliver something tomorrow afternoon, what would I do?" -- and do that first.
I agree with making fine-grained plans as a way to uncover issues. Just remember, no plan survives contact with the enemy. On the other hand, fortune favors the prepared. (I actually think those are both Eisenhower quotes, aren't they?)
The reason that planning software development is bullshit is that you simply cannot know all the little details because in software development you're always doing something you've never done before (because if you did you could just copy-paste your previous work).
To use the 'going to the supermarket for milk' example. I could make a fairly accurate estimate for that because I've gone to the supermarket hundreds of times. I know all the different things that can go wrong and account for them, because I've encountered them before. The elderly person who wants to pay cash and has a bag full of coins. The guy who finds a 10 cent discrepancy on his €92,30 receipt and has to argue for 5 minutes with the cashier (while the row behind him keeps growing). etc. etc.
Now imagine that today is your first time going to a supermarket. In fact, before today you had never heard of the concept of supermarkets, or milk for that matter. How good will your estimate be ?
The only time you can make a decent estimate is after you've finished. Or to put it differently: making an accurate estimate is possible, if you're going to accept that making the estimate is going to take a long time, but I can't tell you how long.
Development time estimation, and every methodology that attempts it (I'm looking at you, Scrum), are little more than desperate attempts by managers to feel in control and relevant.
> Development time estimation, and every methodology that attempts it (I'm looking at you, Scrum), are little more than desperate attempts by managers to feel in control and relevant.
More charitably, they're an opportunity to introduce system 2 slow thinking on the team.
There are certainly novel engineering problems that simply won't yield to estimation, but there are also lots of well-solved problems that will. Managing the latter isn't desperate, its just sound practice toward reproducible results (ideally deployed only where reproducible results create value).
> but there are also lots of well-solved problems that will.
Certainly when you're writing a double-linked list a third time you can estimate how long will it take you. The problem is that in real life you never solve sufficiently the same well-solved problem. Feature details, matters of integration with other pieces of the system intervene - and in practice those errors introduced by new circumstances are significant comparing to predicted time. So the practice may be sound, but it doesn't give the desired results, like predictability of required time.
So you integrate with a few systems, you learn where the gotchas tend to be, you call those out as things you'd like to know more about when making your initial estimates, etc. Today's software dev has probably integrated with more systems than they have written linked lists. We just have a habit of forgetting all the little details.
If you're looking for a formula that lets you estimate once and then move on to the "real work" now that that tedious BS is out of the way, there isn't one. But you can still do something. There's ground between "I can't estimate that in 30 minutes" and "I can't estimate it at all" and it's up to you and your stakeholders to come to terms on how to balance the risks in your estimates (in broad terms: too inaccurate estimates vs too much time wasted estimating requirements that have since changed).
(There's a parallel theme in some other comments: "if it's a well-understood problem why am I not using the existing solution?" Similarly, there's a difference between "well-understood" and "perfectly abstractable and automatable", otherwise almost none of us would be employed these days. :) )
Even if the solution exists you still have to integrate and test it. Those tasks can require a lot of engineering labor, but fortunately they're fairly amenable to estimation.
I completely agree. I'm lucky to work in an environment where management aren't too fussy about sticking to deadlines, as long as progress is pretty decent. It's a nice way of working, I use system 1 for estimates which generally works pretty well and saves time needlessly trying to pin down an exact eta. On the whole our team delivers a lot of features on time and generally doesn't go over 2 weeks late for anything, even though it often wouldn't matter if we did go over.
I appreciate that sometimes hard deadlines are necessary but if I can I'll always avoid them.
A lot of people have been praising this "estimates based on data and rigorous planning" for decades. Steve McConnell wrote a whole book about it (Software Estimation: Demystifying the Black Art).
It works only very special circumstances: when the task can be very well defined because it has been done a lot of times before and the people doing the task are very experienced on it.
But the caveat is that if you are doing such a task you're not really doing any innovation at all. Actually, if it is so well defined and trivial it should probably already have been automated.
True innovation must have a lot of unknowns, it isn't innovation without it.
For what it's worth, the examples in that book are usually contract development shop where you're doing the same thing for multiple clients over and over.
If you actually read Steve McConnell's book the focus is really on accurate estimates more than data and rigorous planning. He makes it clear that a high level of precision is impossible in some cases. (Many people fail to appreciate the difference between accuracy and precision.)
I haven't read the book and was going to ask about accurate vs precise but found this helpful article discussing it as it relates to software here if anyone else is curious about that distinction:
http://itsadeliverything.com/accuracy-vs-precision-in-estima...
This is being a little extremist, no? I'm consistently asked to give estimates, and I mostly give estimates that are accurate within sane expectations.
I think what's missing here is that I can predict how long something is going to take when I talk with the managers/feature-planners about what the feature is. If I feel I have a solid understanding of the requirements, which pieces of code it'll touch, etc - then I can give a reasonable estimate.
Now, the bigger the project the more difficult this becomes. Details I've missed is always a possibility, and that just stacks up the bigger the project is.
So I think being specific in this type of discussion is really important. A feature that's going to take 6 months to implement is insanely tough to estimate, and very likely to have a wrong estimate. A feature that takes a week can pretty reliably be accurate within a day or two, in my experience.
I'm not defending management in these sorts of things.. I think it's our job to not only give estimates, but include understanding about the expected validity of that estimate.
All estimates are worth while though, your post makes it sound like they're not. You tell me you need a feature that's going to take 12+ months? It's so hard to predict that it's laughably wrong.. but that's still insanely useful. Why? Because now the management knows it's highly unlikely to be a 1 week, 1 month, w/e feature. It's big.
> I think what's missing here is that I can predict how long something is going to take when I talk with the managers/feature-planners about what the feature is. If I feel I have a solid understanding of the requirements, which pieces of code it'll touch, etc - then I can give a reasonable estimate.
I'd argue you can't.
Right now I'm working on a feature that I estimated as 'a couple of hours'. I've been working on it all day by now. Not because the feature is that difficult, or because I didn't get a set of good requirements. The requirements are crystal clear and should be easy to implement.
The reason it's taking so long is that I'm fighting with a UI component that doesn't behave as expected (or documented). What should have been a trivial part of this task, a matter of seconds and not worth estimating at all, turns out to take all day because I didn't know this specific component is misbehaving and I couldn't have known until I started implementing it.
You cannot predict things like that without doing it.
> Right now I'm working on a feature that I estimated as 'a couple of hours'. I've been working on it all day by now. Not because the feature is that difficult, or because I didn't get a set of good requirements. The requirements are crystal clear and should be easy to implement.
Just because you're wrong sometimes doesn't mean you can't be right more often than not.
Another way to look at it is, as a company how can you budget anything if you can't make even the slightest of estimations?
What you're suggesting results in throwing away any ability to budget, roadmap, coordinate, etcetc. Absolute prediction is of course impossible, but if what you're saying is true than I should buy a lottery ticket because my estimations are insanely lucky.
And sure, I get estimations that run long, we all do - I'm not saying otherwise.
This is for me the dissonance between agile methodologies and reality. Agile methodologies assume having a flexible scope and time due to the project unknowns (technical, business, social, etc.). But usually budgeting in the real world is required to follow: I promise to deliver X scope that’s going to cost Y dollars in Z time.
I'm trying to get an app to the app store. Mysteriously, the archive button on XCode isn't working. It says it compiles successfully and yet there is no archive to submit when I'm done. I've googled all over and everyone else has a different solution to this problem. This little issue has delayed release a day so far.
I've done this 50 times before (10 times in this project!) and these kinds of issues still occasionally occur. How does anyone predict for this kind of thing?
The reason that planning software development is bullshit is that you simply cannot know all the little details because in software development you're always doing something you've never done before (because if you did you could just copy-paste your previous work).
If you can plan and execute putting a person on the moon, you can plan and execute software development.
I think the problem is the overwhelming majority of software development is so low stakes, easy to replace, assumed that bugs are par for the course, etc... that most orgs have stopped putting real effort into planning AND executing on that planning, instead just cargo culting the planning phase.
> If you can plan and execute putting a person on the moon, you can plan and execute software development.
But the budget for planning is limited. To realistically plan out most software projects would entail almost as much work as simply executing on them. That level of resources for planning vs. execution is acceptable for putting people on the moon. Probably not for building a line of business app.
You've missed the point. Nobody can (or did) accurately estimate how long the planning and development would take to get someone on the moon. Half the materials weren't even invented yet in 1962. Nobody knew that Velcro was flammable in pure oxygen.
Sure once all that expensive and unknowable planning and development is done, you can double-click that executable and put a man on the moon.
> If you can plan and execute putting a person on the moon, you can plan and execute software development.
Do we have the planning and estimates of projects that are similar to "put people on the moon"? I'd like to see how the plan/estimates of these projects look like and how they change.
Getting to the moon took many many man years of planning, experiments, material development and cost a few lives. It also involved several of the worlds leading scientists, and probably the best computers money could get at the time.
If you can accept those kinds of preparations and also agree to not change the specs much (i.e. no, we aren't going to Mars instead) after we've started developing it I'll say we can get you a pretty nice estimate.
Yet if I ask you how long it takes to buy icecream in Japan, based on your experience grabbing milk in the US, your guess will be "less than half a day". Which is good enough for me.
And if I ask you how long it takes to build a Wordpress blog for a music band, based on your experience building blogs for realtors. You guess will likely be good enough for me. :-)
But then you say go for it and get upset about all the time I spend getting a passport, booking flights, and figuring out how to transport the ice cream back.
I’m kidding, of course. It really depends on how well business and development understand each other.
> I’m kidding, of course. It really depends on how well business and development understand each other.
Yup. This is the crux of the whole issue, and it's being completely blown out of proportion. There's always going to be details the estimator (dev) won't know, but the more clear a picture you have in your head, the more confidence you can have in an estimation.
This is directly related to the size of the project too, of course. If it's a 12 month project, a single developer is massively unlikely to have a reasonable understanding of all the potential roadblocks that the project will take. However a 1 day task, with tools you know/etc, you can have a pretty good idea how long it'll take.
There's always some unknowns, but that's literally what an estimate is for - attempting to plan for the unknowns. I can write an add function and I'd bet a lot of money that it won't take more than a day.
Unreasonable management seems to be giving developers PTSD here. As I'm having trouble understanding why the idea of an estimate, by definition exactly what we're doing, somehow doesn't apply to software development.
If you're being pedantic you could say the same about a lot of things. How long to dig a 1foot hole in open ground? Well who knows, there might be a massive rock - but that doesn't mean you can't give an estimate. It's literally what an estimate is for, an approximation based on assumptions and expectations.
> Unreasonable management seems to be giving developers PTSD here.
It's inexperience all around. Also, most places have no clear long term track, so people bounce after 2 years rather than growing to understand each other.
Managers are generally free to cut estimates in half, and move to a new role or move to a new organization when things fall apart. They're not trying to be villains, they have a lot of pressure coming from a lot of directions and think that estimates are bogus anyway, thanks to star trek and the occasional miracle that happens. (sometimes things really are much easier than they seem)
Developers are optimistic, wildly underestimate, and feel bad when they miss them. The overreact and start massively overestimating. When they progress to the next stage they take the time to give a good thorough estimate to have it slashed by a manager, who knows they overestimate.
Nobody sticks around long enough to really understand the state the other is in, so you get this vaguely Kafkaesque world that makes everyone a little crazy.
Estimates in software are effectively experienced guesses. This is in contrast to other industries, such as construction, where planning is done for months to ensure the estimate of time and materials is accurate. If you attempt to treat one as the other, you run into huge problems.
Software isn't like building a bridge; it's like planning to build the bridge.
I'm pretty good at software estimating; I can roughly tell when something will take an hour, 2 hours, 4 hours, a day, a week, or month, 3 months, or 6 months -- in those intervals. It's more difficult and time consuming to get more specific than that.
I've also had a 1 day task blow out to entire week more than a few times. At that point, you have to start questioning the predictive value of estimates.
Heh. But even in construction projects, serious cost/time overruns are very common. I can't find a good source through quick googling, but what I do find is horrifying.
It is actually quite fascinating to see how our cousins get on. He's quite blunt about the dynamics that cause failure and that relatively few megaprojects ever live up to their business case.
I estimate it takes 20 mins to buy an ice cream in Japan. If it takes "half a day" that's (charitably) 12x (8-hour days), or (uncharitably) 24x (16-hour days).
If I estimate a month to build your feature and it takes me a year or two, you're not going to be very happy...
To be fair, Scrum doesn't try to capture development time estimates, only the effort needed to provide value. Work items inherently vary in size, or estimated effort. However, enough work should be planned during planning for the dev team to forecast what it believes it can do in the upcoming Sprint. It's all about transparency, not adherence.
That said, requirements emerge daily. Transparency is essential for honest dialogue.
I agree and disagree with the notion that an estimate can only be provided after the work is done. If the work is broken down properly, you can use empiricism to get a good understanding of what is needed based on previous, similar work completed. But to think an estimate can only be provided after the work itself is done is kind of ridiculous. If you section off variables with manageably-sized work you should have a good understanding of what's needed to pull it off.
Very well put. It reminds me of the “Halting Problem”. Although for a subset of programs, it can be proven that they will finish in a predictable amount of time. Maybe this also holds for projects.
This is the reason big design up front hasn't been the gold standard in planning. Most modern methodologies acknowledge that there'll be a lot of changing requirements and learning-while-building.
The point is, given this information, you could a) run around like crazy doing all manner of reckless and crazy cowboy programming and putting out fires b) carefully plan a relatively short period of time on a small, achievable, well defined goal (not "build the project"), then making a note of what worked and what didn't, and readjust before you plan the next short period of time
You need to have some fairly specific goal in mind to start doing anything. AKA are we building a PS4 game or a self driving car?
Methodologies like Scrum are all about transitioning a long term objective into short term progress. But, you really do need some fairly well thought out goal.
Further course corrections quickly get very expensive. Sort of like sailing from Virginia to Europe then deciding 1/4 of the way in on France then changing your mind to Spain at the midway point is probably fine. However, changing your mind to Australia is a much bigger deal.
Agreed! Nothing I disagree with here. But last minute itinerary changes from France to Australia back to France again point to a larger, more systematic problems IMHO. Perhaps it's a lack of understanding of the market and customers, or just bad management and lack of company vision - but no software process is going to fix that.
And then a poor maintenance programmer is faced will lack of architecture because the thing was built without any in mind... Just as a perpetual prototype.
Massive software companies deliver high-quality software, on-time all the time for one reason: resources. They got the money and the time and the developers and all the enterprise accounts up the wazoo.
Typically no one wants to pay more for less, but that's the best strategy to deliver high-quality software on time.
You reduce the scope drastically but do not adjust the timelines. You pay for more development resources(tooling, developers, servers, time, etc.).
If you can't deliver at that point, then there's a serious fundamental issue with either the leadership or the team or both.
Startups don't have this luxury. So they must execute on the least complex scope on a very tight timeline. High-complexity scope increases the fatality risk to the company.
So as the scope increases in complexity the risk increases and the likely the company itself will fail increases. Again, because startups do not have the cash to pay for the proper amount of resources.
”Massive software companies deliver high-quality software, on-time all the time for one reason: resources. They got the money and the time and the developers and all the enterprise accounts up the wazoo.”
I think that is a myth right up there with “If I was rich, I’d be happy.” Massively funded software projects fail all the time.
This claim of inescapable novelty has a kernel of truth, but it is being exaggerated beyond any sense of proportion. If everything you have done in your day-to-day work has been beyond-estimating original, then either you could out-Knuth Knuth (in which case, kudos to you, and I hope you will have time to write the books someday, but your experience is not generally applicable), or you have been goofing off some of the time.
That which is passed off as novelty is often ignorance (not necessarily on the part of the developer.) To continue the retail analogy, software projects are often like the guy who is building a deck on his house, but keeps going back to the hardware store because he made no attempt to think things through, or measure before cutting (note that, in this analogy, the would-be builder is not necessarily playing the role of the developer. In some cases, that might be the hardware store owner's role.)
But it's exactly this 'ignorance' that is the major part of every software project. There are unknown unknowns, and there is no way from a standing start to grasp either their number or their nature. I'm happy if people are saying that this exploratory work should be done (in part or in total) before estimating, but in my experience that's just not palatable to most organisations.
I envy _hugely_ anyone who builds software that is even 50% the same as what they were working on six months ago.
Well, then let's call it what it is, because there are things that can be done to mitigate ignorance to some extent, if there is the will to do so from all parties.
> there are things that can be done to mitigate ignorance to some extent
Sure, but all you're doing is moving the uncertainty from the development to the planning phase, so instead of a development phase of unknown length and cost, you end up with a planning phase of unknown length and cost.
Firstly you you are still assuming the same gross exaggeration of novelty, and secondly, you can do analysis and planning at a much higher level of abstraction than development.
Waterfall gave the impression that planning and analysis is costly by insisting it should be carried down to the lowest levels of abstraction, before doing anything else.
> Firstly you you are still assuming the same gross exaggeration of novelty
I disagree. Even when you're doing mundane everyday stuff you have done a thousand times before, one trivial variation may throw of your estimate completely. Maybe an edge-case in this specific project triggers a bug in a 3rd party's library, and you're losing weeks waiting for the 3rd party to roll out the fix (been there, done that). You just don't know in advance
> you can do analysis and planning at a much higher level of abstraction than development
Of course, but then you'll get an inaccurate estimate.
> Waterfall gave the impression that planning and analysis is costly by insisting it should be carried down to the lowest levels of abstraction
That's only necessary if you want an accurate estimate. I think accurate estimates are pipe dreams.
There is a false dichotomy here between knowing everything and knowing nothing, between perfection and utter chaos. You may only have experienced the latter, but there are places where people know pretty much what they are doing, most of the time.
> ...there are places where people know pretty much what they are doing, most of the time.
It's called spec programming, where you have programmers slinging code against precise specifications. Common in ERP and regulatory or business compliance-driven code.
On the other hand, you have development. That is where you're attacking the less defined problems. Sure, we know the tools in our toolset, how to apply them, and time it takes to apply each tool. Problem is, the time to figure out the series of tools needed and their precise applications burns enormous time, time that could have been spent towards a MVP.
> Problem is, the time to figure out the series of tools needed and their precise applications burns enormous time.
In the use of the adjectives 'precise' and 'enormous', you are just repeating the same false dichotomy in a slightly different form, by ignoring (or not understanding) my point about the use of abstraction in analytical thinking.
The people who are most successful in what you distinguish as development understand a good deal more about what they are working on than spec programmers need to, and they are not, as you imply, groping around in the dark for a solution - that's what spec programmers do when faced with a development problem.
Just to give an example of two projects I worked on at the same company within a few months of each other:
The first involved taking all sorts of situational data from various sources in an airport, e.g. radar systems with their own binary formats, and combining them with the work of a postgrad researcher who had come up with an algorithm to optimise when pilots should turn on an aircraft's engine in order to save fuel and cut emissions.
The second involved managing 3G dongles on fleets of laptops, tracking and sending alerts on their usage and billing info in realtime, back in the days when business users would accidentally run up £10,000 bills abroad.
If at any point in the first half of either of these projects I'd been asked to come up with an estimate for the total amount of work, it would have been wildly inaccurate. Each project had both low and high level surprises at every turn that would have just compounded to blow those estimates up. There is basically no significant way in which one informed the other, even though parts of the systems used similar event sourcing architectures on the exact same server-side stack, and were done for the exact same startup which broadly offered real-time systems to utility companies of various sorts.
It's entirely possible that there exists a breed of programmer who could have estimated these projects in their entirety, much earlier and more accurately than we did, but I've never met them, and I'd rather work in places that didn't compel people to.
I've known and worked at other places where basically any software methodology would (and did) work, but I certainly don't think it's rare (as you seem to suggest) to find yourself at the other end of the spectrum.
Aaargh20318 et al are making the claim that it is always inescapably like this and there's nothing you can do about it, because every day of programming is always fundamentally unlike anything you have ever done before. All I am suggesting is that this is an exaggeration.
Doing something about it takes some commitment, however. Nobody estimates well without having practiced it, and been honest with themselves about where it went wrong.
It's like building a deck except even the screw hasn't been invented yet. And you think: I'll just create a device that attaches the wood like a big paperclip but then realize after you make it that it's not strong enough and won't scale so you create a nail. But then you realize you may need to reuse the wood so you finally create a screw.that can be removed.
You aren’t wrong - for some things. I think it’s a spectrum.
For the things I work on, and frankly always have, I spend most of my time in the “R” part of R&D. Now granted, I’ve never worked on classical “business” or CRUD type problems.
It’s natural to think that once you have lots of experience, that estimating will be easier. In my experience, I found myself simply tackling harder or unique problems. Instead of getting comfortable, you find yourself mostly operating at the edge of your competence. This is great fun [1] but makes estimating pretty hard.
[1] I personally enjoy being on the “black diamond” part of the learning curve. (“Black diamond” = most difficult ski trail)
Agree 100%. IMHO, a lot of it has to do with the programmer mentality of wanting to be "challenged" to solve problems one step away from their competency. If you're fresh out of school, implementing your own allocator might be a challenge you take upon to solve a memory constraint, when you can just take what someone else (with more experience) has built. And then senior people insist on designing yet another over engineered platform technology because everything has to be 'future prof' (when in reality it never is and only ends up being performance-proof). How many JS frameworks have we had so far that do the same thing and just repeat mistakes of previous ones? Its like one every year.
There is also a lot you can do to decrease impact of novelty on the overall schedule and most people I work with do exactly nothing to reduce variability in the process and improve their estimation ability.
It is easy to claim every time you write code you are doing something new but, hey, every time they build a building they do something new. Yet buildings (except for extremely ambitious projects) typically can be delivered reliably by reputable developers.
Also consider that the management or project is looking for just some vague white liquid and at the store you only see cans (powdered milk? Is that white?)
To keep with the 'going to the shops to buy milk' theme you can break it down into bits you can estimate. Assuming you've never bought milk before:
- how long will research take? Where do you buy milk? Where is the nearest shop? 3 mins.
- how long to get to nearest shop and back (eg 10mins x 2) = 20 mins.
- find milk in shop (30 secs to 5 minutes), so say 4 mins.
- Buy milk. (1 to 5) say 3 mins.
Don't forget that these are estimates and always add a bit of headroom. The above comes to 31 mins, so say 35 and you know you're covered. If the manager isn't happy with it, give him the low down figures.
Programmers aren't special snowflakes and there's nothing special about programming. The exact same can be said about anything be it marketing, sales or design.
Having been through multiple month long planning "phases", that were eventually shown to be wildly inaccurate I disagree.
This is exactly what we used to do in the waterfall days. It didn't work.
The only way that's been compatible with me is to build something really small but valuable. So small that its hard to be disastrously wrong. Once you've released that value, build upon it.
Stakeholders tend be much happier as they at least have something they can use really early on.
Conway's Law: "organizations which design systems ... are constrained to produce designs which are copies of the communication structures of these organizations."
And most organization -- or collections of three or more people -- have dysfunctions. There are rarely process problems where technology is the main problem; it's the wet-ware between the keyboards and the chairs that make or break projects.
I find that in the case of the second solution: "thorough planning" you get a bunch of people fabricating estimates of estimates with padding or not padding or trying to estimate a bunch of unforeseen events that are too far ahead to get an accurate handle on.
Sure, the second is likely to be much more accurate just because the second is likely to be much, much longer but I have never given one of these estimates without it being followed up by immense disappointment by business with a desire to strong arm me into something shorter. This demonstrates a fundamental issue with the topic. Its not about the estimate at all.
I think the problem with estimates is their finality and assumption of correctness. I figure once you're in the developer-years category you might as well iterate the estimate a bit to get a better sense of it.
Error bars should be translated into risk for the business decision.
Too often I see people claim to be making "rational, facts based" decisions on estimates (beyond 1 year+) are complete codshit. This is not rational decision making. These decisions should be about risk management assuming failure as opposed to thinking you can slot year+ development estimates together.
I think very often the desire to lock down development estimates into "rational fact" are business decisions of risk masquerading as technical developer decisions of fact. I have yet to see a situation where we deliver an estimate that blows a business decision out of the water where the business just backs down. It just learns to ask a different question and gets the answer it wants out of that.
For me, not designing the product as a platform is no. 1 root cause of all the delays and confusions. People then to deliver for speed, make compromises. Over a period we end up in mud pile and slows down all future developments. This makes the speed slower & slower with age.
One statistic that has stuck with me is that 80% of software projects fail. It's the default that your software will not succeed, or do well.
One small reason for that is poor time estimation, but having good time estimation won't make your software succeed. I think having good planning, makes for a good working environment, but it doesn't mean the project as a whole will succeed. It might mean the developers will be happier to be with you and pivot to the next idea.
I would like to look at the percentage of successful projects and see what proportion of these had good estimation attributes.
That's a meaningless statistic. The vast majority of large software projects do eventually deliver something of value, even if it's late, missing features, and full of defects. Is that a failure? Depends on the circumstances.
Upon reading the first couple of paragraphs I decided to test it on myself. I had to buy X, which is available in convenience stores (comvini, Japan). I estimated it'd take 5-7 minutes, but I was very surprised when it actually took 4 minutes! To be fair, this has to do a lot more with Japan than with myself. The same test in my home country (Spain) I would have had to guess 15+-5 min during day time. Sidenote: I was very surprised about the concept of "driving for milk", which I'm assuming is a very American thing.
I'm fairly realistic about software estimation which only came after a LOT of retrospection. It normally takes what I estimate, both personally and professionally. The hardest factor I've learned to include is the level of detail. For instance with my personal website [1] I gave myself a full Saturday because I knew I wanted high detail level but I had in mind the overall design. It took the Saturday +1h of a couple of improvements/bugfixes (under 10% error). With my current job I'm also under 10% error.
In the past I have been bitten a LOT about my unrealistic estimations, so the only solution I had to move forward was to learn from those and so I did. So now, from the article, I know that my "quick thinking" is around 50% of the project. I force myself to think a bit more and the details trickle down.
Another thing I've learned is that projects tend to fill as much time as possible (Parkinson's law [2]). So if you are told a deadline, half it! Put the half as your internal deadline, then the project will be just on time.
Finally, complexities are exponential, so learn to say no to unnecessary cruft. "A small change" might seem like a 1% change for business and for you initially, but it will more likely than not grow into a 10-20% change in the end. FFS that is why the duck was added in the first place, to avoid wasting time [3].
The thought occurs to me that no matter how much planning you do or how good at estimating you are, the development is going to take however long it's going to take.
You can plan your trip to the store to buy milk and figure out exactly how long it will take, but the only thing that actually matters is that you arrive back home, with milk.
If that milk is absolutely necessary, whether it takes 30 minutes or ten minutes to get it is really a secondary concern. If you spend five minutes getting a better estimate you've delayed the milk by five minutes regardless of how long it takes or how right you were about the timing.
I think we spend too much time thinking about time estimation when the planning we should be doing is figuring out what is so important that the time it takes to build is worth it even if the time estimates are off.
This is literally what agile is for. Step 1 (and this is the hardest part) is to negotiate the need for either a flexible scope or deadline. Accept that you're estimate for a fixed scope will just never be accurate and just make room to adjust. Putting "buffer" into your estimate is also not the right approach because it's planning for failure. Flexible scope means you have work to fill up what would go into your buffer that you can launch without if you need to, but would be very nice to have.
Create your high-level backlog and do MoSCoW prioritization. Figure out your "musts", "shoulds", "coulds", "wonts". Now apply some estimates to your features and add 10% for unforeseen growth. Estimate velocity based on team size and now you've got a date when you could conceivably hit your musts, shoulds, coulds. Set your "deadline" if you must somewhere deep in the coulds. If your musts go over, you are still able to launch an MVP. If things go well, you can start delivering non-musts.
Adjust your plan every sprint based on actual velocity.
There are ways to deliver on time & budget but most developers are not going to like it neither are their "clients".
Use the frameworks exactly as they are intended, don't try to invent new solutions that aren't native to the framework.
Anything that takes you outside the beaten path in development is going to be a potentially infinite black box.
In other words, a lot of software engineering can't be put in timeboxes because it's actually R&D more than it's development and where each little step forward can add a potentially infinite amount of new tasks to be done or problems to be solved. Add to that the constant need to update, upgrade, improve, re-design and you know it's just not doable.
So the primary problem IMO is that we think about a lot of development as if it's something that can be put in boxes. Some can of course and the better and more solid the team becomes to better they are but the teams who struggle are mostly struggling because the expectations for what they are actually doing (inventing problem-solving) isn't matching up with what they are being paid to do (build)
238 comments
[ 2.3 ms ] story [ 270 ms ] threadGenerally that results in one of two things:
* Product delivered on-time with massive technical debt.
* Product delivered late with massive technical debt.
Frankly I don't know if adding front-line engineers to the deadline decisions is going to make the issue better or worse, but fundamentally having non-technical or formerly-technical people defining deadlines definitely doesn't work.
We need to set realistic goals so that we can work hard to meet them. The mindset you are describing is part of the reasons that we have these problems in the first place.
And then came the Assumptions.
And the Assumptions were without form.
And darkness was upon the face of the Workers.
And they spoke among themselves, saying, "It is a crock of shit, and it stinketh."
And the workers went unto their Supervisors and said, "It is a pail of dung, and none may abide the odour thereof."
And the Supervisors went unto their Managers, saying, "It is a container of excrement, and it is very strong, such that none may abide by it."
And the Managers went unto their Directors, saying, "It is a vessel of fertiliser, and none may abide its strength."
And the Directors spoke amongst themselves, saying one to another, "It contains that which aids plant growth, and it is very strong."
And the Directors then went onto the Vice Presidents, saying unto them, "It promotes growth and is very powerful."
And the Vice Presidents went unto the President, saying unto him, "This new plan will actively promote the growth and vigour of the company; with powerful effects."
And the President looked upon the Plan, and saw that it was good.
And the Plan became Policy.
This is How Shit Happens.
I've worked with Engineering VP's/CTO's that are pressured by deadlines and promise unrealistic features. The key element needed in this is a 'leader' that can push back and manage expectations.
Engineers hate releasing poor quality products, but the reality is that customers often will buy and use something laden with defects rather than nothing at all. And in any case, after the initial botched release, you can always fix stuff. See Apple.
This is the same type of argument that makes short-term profit take precedence over long-term success. Why in the world are we getting a deadline of 4 months? You, as an executive, had to know burn rate was blowing through cash a lot earlier than that.
Engineers hate releasing poor quality products because we know we'll eventually have to fix them and we will likely be given almost no leeway time-wise to fix the actual debt, as time will always favor ugly hacks that "work" over well-written code that prevents the problem in the future.
The tech titans aren't monoliths with a collective hive-mind. Each business unit and team is responsible for projecting revenue targets and then hitting them, same as any other kind of business. And, yes, if the business unit managers fail to hit their financial goals too many times, their project will be cut and the team reassigned or laid off, same as a startup running out of financial runway.
Here's the reality, and maybe you've forgotten: The engineers are the ones who are going to get dumped on, and you and the VP's are the ones who are going to reap the benefits. Never forget that.
Not that I'm claiming that all executive decisions are clueless, most people are just trying to do a good job with limited information, and mistakes happen. But you run into enough executives without respect that runs in the other direction, and you start to question all deadlines.
(Which isn't necessarily a bad thing. In a healthy environment, questioning the feasibility of a deadline can lead to appropriate descoping of overreaching features, reduction of risk, etc. But the situations that lead to this kind of suspicion are rarely healthy to begin with.)
This is everything that is wrong with software.
What is software really? It is translation of business process and need into code - I spent the first 10 years of my work life mastering code, I will spend the rest of it understanding whatever business I'm working for.
Assuming you have some experienced engineers (not mangers) on your staff then having the vp set a date (and defacto choose the project) is absolutely backwards.
Bring the problem to your engineers (money) and don't look for a moon shot look for 10 smaller projects that get you 1/2 way to the goal in 1/10th the time. Don't think of it as a "hack day" think of it as a hack week or hack month.
If you have a large project and it is to "save the company" (or push it forward) and your treating it like a must do rather than a foundation (and taking care) your creating debt and starting a cycle that you will never pay off.
Sometimes...
Other times its driven by the fact they are personally going to get a fat bonus
What worked for us is what's basically mentioned in the article: taking a more structured, slightly painstaking approach to estimation. It's more tedious than gut feel story point estimation, but a lot less painful towards the end of the project.
Most dev work right now is still just attempting to deliver correct software. let alone on time or on budget.
"Debuggers" are the monitoring, telemetry, sensing, and forensics equipment, data, and processes applied to each test, launch, and component.
We have debuggers because code is otherwise a black box. Debuggers are a visualisation technology.
House building has become somewhat more complicated over the years as building codes are made stricter and new techniques are developed, but for the most part it is just a switch to a new way of doing things vs. in the software world where we will continue to pile on more and more levels of abstraction, but can't reasonably expect to have at least some kind of understanding of the underlying layers.
I worked professionally in a couple of high-tech engineering jobs, as architect, surveyor, city planner, civil engineering, construction, stage design, movies, automotive engineering (F1), games and at internet service providers. (but no airplanes, nuclear power plants or rockets, sorry. Lots of hospitals still.) The SW development jobs compared to that were always the easiest, most creative, most rewarding, and best paid also.
The problem with SW planning is pure lack of experience and poor engineering practices. A good SW engineer can hold his estimates, just as an experienced architect can hold his date and costs. In practice it rarely happens, I only know a couple of successful projects with smaller companies, but the good and big companies mostly deliver on time and budget.
As you mentioned houses and bridges. With houses you don't change engineering paradigms every 5 years as it might fancy you. You never try out new materials which are not tested over 10 years at least. These things need to stay up for 50 years at least. With bridges you cannot really plan them properly. Well, you can. But a typical bridge engineer multiplies the safety factor with 10, leading up to 10x more material as needed, as you can hardly estimate the dynamic peaks and resonance frequencies. With highly dynamic forces such as e.g. in a F1 gearbox, a huge spring with max 4000 Nm forces on your shafts. There you cannot apply any 10x rule. This thing needs to be perfect. You plan to avoid the peaks at all, avoiding the dangerous frequencies. Comparable to wind forces on a bridge. But in practice you look at best practices. You build dynamic physical models and simulations. And measure. And apply safety margins. E.g. with the big steel bridge next to my home, the famous "Blaues Wunder"in Dresden, the general public and politicians never trusted their engineers with this fancy new steel design, so it had to made 10x bigger and heavier, and even on opening day they invited the whole town to test it live, with lots of heavy trucks, a tramway and pedestrians walking over the bridge and jumping on it all at once. It stills stands today, but could have been made much simpler and elegant by todays knowledge.
Now compare that to modern SW engineering. You got a fancy new framework and language, looking for engineers with at least 10 years experience in that, with the framework existing for 2 years. You got massively overhyped tools which rarely deliver on their hype, complete lack of security, memory safety, type safety or concurrency safety, completely outdated API's like blocking IO or POSIX or threads, and undesigned monsters like C or C++. Good luck with that. That's of course a death march.
Still with proper tools and proper management it's much easier than everything else. In automotive everything was properly planned, with traditional waterfall. Everything worked as planned. Even if it was hugely complex, much more complex than a simple OS. In my own SW I never miss my milestones or overdo my budget. In architecture we always did. With huge costs up into billions. Only once we went under costs by 200 millions with the biggest building construction project in Austria. This was a miracle then.
1. Poor engineering planning. The way to do this well is to break down a feature until they’re at about half-day sizes tasks. Identify any obvious risks or ambiguities with the technical approach and communicate them to Product and figure out how to de-risk them. Communicate costs to Product so PMs can make cost/benefit trade offs (eg “if we built it the way you specced it it’ll take two weeks. But if you make these trade offs it’ll take two days.” As a Product Manager I might not want the feature if it costs two weeks but do if it costs less than 5 days. Without costing the features in dev days I can’t makr this trade off).
2. Failure to make trade offs that drive towards shipping software. The way you ship is by declaring it’s done, even when it’s not really done. When code is being written unexpected issues always arise. It’s Product’s job to make hard trade offs that drive towards shipping. Eg “that’s an edge case bug, let’s punt it to v2” Or “let’s cut that nice to have feature and squash the showstopper bugs and ship”.
People bring up impossible deadlines set by management. In my experience most deadlines are movable and a strong PM/Eng team can convince management to push a deadline back. And if a deadline can’t be moved, it’s all the more important to know how much features will cost and make hard trade offs to get the product out the door.
But you are right that estimates are for managers. Managers should watch their reporter's progress and use that as evidence to build evidence. If the first task of 20 took 1 week, it's a 2year project. Update estimate as time and progress and work remaining increases.
I’m quite serious.
The thing you need to realize is that you’ll get it wrong, no matter the tool... until you invest time in reviewing past estimates and plans.
People think because they’re engineers they can just magically make estimates... but its a skill. If you don’t have a feedback loop to measure your progress and adjust your actions, you’ll literally never improve at it.
...that’s why even ‘Senior’ developers and ‘technical leads’ are often terrible at this; they’ve simply never bothered to seriously make an effort to learn how to do it.
Most of that will be doing spikes: http://agiledictionary.com/209/spike/
It's one the most underappreciated agile concepts. I wish people would shut up about standups once in a while and give more thought to doing spikes.
Spending 30 minutes researching four different libraries to see which one to would be a spike, for instance, but it doesn't necessarily involve building a proof of concept.
In my experience, they just pretend it doesn't exist and remove it from the "plan". Where I see this the most is in figuring out how things work. For some reason, non-programmers have it in their head that once you've finished college, you ought to know (even though they don't) how everything that goes on a computer works instantaneously. We're going to use Apache Kafka! Ok, need to spend some time figuring out how that works. Um, ok, that should take like an hour, right? Because there's no time in the budget to learn anything. If you push back, they say, "ok, never use anything new that you have to learn. Only use the things you already know". That doesn't work either, because the things that we have to integrate with don't work with the old things. Or, "you don't need to know how that works, Bob knows how that works, just ask him. But don't let it take any time, because there's no time in the plan for that" And Bob is sick of all your questions that you'd know the answer to if you'd just spent a two or three days reading the online documentation and installing a test system and testing it out.
I'll add this: a product team that ships earns 1) more credibility, and 2) that credibility is reciprocated towards the team in the form of greater flexibility and freedom to set their own targets.
I have never seen that work in practice.
In practice, being able to break things down meaningfully and accurately to such a minute level implies that a lot of time was spent in grooming, which means we're just back to waterfall again. (Or the tasks at hand really are just that simple and/or the people involved have a lot of expertise.)
1. Are you sure you can do that in half a day?
2. You forgot about this one part of the task that you thought would take half a day.
3. There was a bug in the plugin that you were going to use to do this task in half a day, and you spent three days fixing it.
4. This task will actually take longer, but management won't buy that. Looks like it should be half a day on paper, so that's what we'll say.
5. I've never even done this one thing before, but it sounds like it should just take a couple of hours.
I understand the concept - trying to say something will take a week is even more error prone. But I think most of us are still pretty bad at estimating a budget for tasks even at such a small scale.
Not to mention that that approach only works if you produce another version of something you already produced many times. There certainly are such development efforts, for example, I think anything data- and form based, like data entry + database + some reporting, what used to be paper cards a long time ago, and then punch cars, those type of apps can probably be made using that method.
So that's when they had to build the freight version of the Boeing 747. Then they thought "that was easy", and started the Dreamliner, something very different that they had not done before... (and it even still is an airplane, so not even that outrageously different as different pieces of software can be)
I think the "break it down" approach also faces the problem of the "coastline length measurement".
I'm also reminded of an anecdote from my early days with computers: I had to write an assembler routine (as part of an 8 bit assembler program) that seemed too hard, so I outsourced it to someone else. He didn't know what to do either, so I explained it to him. When I was done I realized I had just implemented that difficult function - I wrote it down in half an hour and that was that. Planning well, finding out what to do, is harder than the actual programming. Again and again, in all projects, I find that the >90% part is finding out what to do. There are stories from projects that lost all their code, years of work, and they thought they were done and finished (business kaputt) - but when they rewrote the software they fond it took them only weeks or instead of years. When you already know what you want and how the end result is supposed to look like (and feel and behave) then writing it in code is the easy part.
So you've advocating an approach where you... don't plan? I'm confused.
Perhaps you're getting too hung up on the half day part? That measure is a useful forcing function that ensures the dev team has thought through the approach in enough detail to identify where the risks are and what smart trade-offs can be made.
We practice a lean-laced Scrum with 1-week sprints. And yes, it works. And yes, we go fast. So yes, it can work (thought we have several years of refining our methodology and have a big culture of pragmatism and problem-solving).
Indeed, for a lot of us in smaller companies, getting our clients to actually consider taking an agile approach at all is a stretch.
Story generation is done by the PO, that's most of his job (though we split stories during backlog refinement if needed).
I don't know if it sounds like a lot (I feel like it could), but turns out it's actually efficient, every minute of each meeting is productive because we know exactly what we're going in for, how it will go and what you want to get out of it!
IMHO most disasters I’ve seen occur when ‘Product’ tries to strictly control the ‘Developers’. It just does not work for certain kinds of projects. It also diminishes the developers’ involvment and leads to a culture of mediocrity and CYAs.
Also, a person that does not have to do the work will always be waaaay more optimistic about how much it takes to do the work.
How do you fix it? Trust and ownership. Product and Dev need to have an interface - a boundry if you will - and the discussions and talk are about defining this interface and setting expectations. Product trusts Dev on implementation. Dev owns the delivery and makes it happen.
Almost every issue I've had with the business/developer dichotomy (and I use that instead of product because different orgs place product differently) has been due to the business trying to pitch solutions to dev that invariably end up not actually addressing the problem (even simple things like "that's the wrong color" or similar, are because they believe they're responsible for the solution, and they have a very clear idea of what the solution should be, without actually being able to articulate or recognize the problem. Does the color actually matter? If you have a problem that implies it does, yes, if you just came up with a solution on your own and in your head that solution was yellow, no).
The real world is a messy place where even with a good team, a good plan and sensible expectations you can fail spectacularly.
If there is one thing that I really like is aggressive validation of the assumptions that are being made + keeping the product as small and lean as humanly possible (and the latter should apply across the board to everything you do).
Complexity and Entropy are preparing the coffin for your work even before you have the idea. You can give ‘em a run for their money, but in most cases you’re going to end up 6 feet under. As soon as you accept this and are genuinely open to failure you will be ready for the opposite.
Currently having issues with this one right now, and it is the most frustrating thing I've experienced at a job.
Re #1, this is kinda the system 2 vs 1 thing in the article. I was lucky enough to have a PM drill me into this. More specifically: The bigger a task is, the likelier it's going to go over time. The bigger a task is, also the less likely each bit is well specified and more likely that there are significant portions glossed over that you'll discover during implementation, which will delay things and/or result in them being done poorly. If you're having a hard time breaking the task down, it's probably because the task isn't well defined enough, so go back to stakeholders and define better. In this model, planning takes a lot of time and effort. Also in this model, research-y tasks are different, and instead get a fixed timebox after which you have to re-decide how much more time to spend on it if at all.
Re #2, part of the problem here I think is that PMs aren't trusted / empowered with that ability or final say. "Stakeholders" get to make that decision, but with ignorance of what it's currently like in the trenches.
What does work is waterfall. Whatever is in the repo at the end gets shipped.
If the client or management want to change the scope, in any way, the discussion must immediately start with questions such as what other feature are we dropping, how much time are we adding to the delivery plan, what are the consequences on other features or the current design and code?
Even deciding to remove a feature can cause delays (e.g. having to re-structure a test scenario). Or changing a string in the user interface, which may require re-arranging the layout, re-translating the string, suddenly having to manage plurals, checking that a variation of the old string wasn't already used elsewhere, etc.
The original primary point of the Agile Manifesto is to stop doing that. There's no excuse for it, unless you are required by law as a government contractor, and in thar case no one expects a successful result anyway.
Software is hard because we 'don't know the unknowns'.
It's not like building a building - if it were, we could just schedule it down to each detail. We would be still all be using 'waterfall'.
If we knew how to write the module, we'd just do it, test it, and voila.
But most of our time is spent 'figuring stuff out'. Oh, product has asked for a tiny change (support a specific kind of font), but our libraries don't support that, meaning major possible architectural changes.
The other major unspoken factor is that requirements change. In fact, this is the #1 thing, really. Requirements change because customer's needs change, because business changes because our understanding of the market changes, etc., and this is a reality of our world. So the nature of how we're going to adapt is difficult to predict.
This is why the #1 predictor of outcomes is the quality of developers on the team i.e. experience, intelligence, know-how of the domain.
If you're doing something where there are very few unknowns, well then it should be possible to schedule within reason.
I hear that it doesn't work that well for buildings either, as you end up with people in the field detecting bugs in the plans and having to improvise fixes without input from the designers.
Soil and dirt samples at $100K - just to do a bid on a project, let alone the actual build.
Can't spend $500M on something only to find you have to tear it down.
Software is not magic, but it does require thoughtfulness and experience ... (hint to all those who think startups require mostly new/young blood - it's mostly wrong, but that's for another post)
Walking on water and developing software from a specification are easy - if both are frozen - Edward Berard
Comparably it'd be as if building developer built from prefab that changes the design every week.
You're identifying what the outcome of good planning looks like, not how it's done. What prevents this is that teams read "agile" as "make it up as you go along," when in reality if you're spending more than 10 minutes discussing a feature, make planning that feature a sprint task.
> In my experience most deadlines are movable and a strong PM/Eng team can convince management to push a deadline back.
It's not generally the team in that room discussing those deadlines, though, it's software development manager and product manager. The worst managers are software engineers who get promoted to management, and when they get into those meetings they look at their shoes and are railroaded by upper management.
You really do need a manager who has the intestinal fortitude to go to bat for you.
[1] https://www.tamingdata.com/wp-content/uploads/2010/07/tree-s...
And then some poor soul gets to rework all this...
Basic abstractions help a lot. Specifically, having things weakly coupled.
To work it takes being comfortable saying "that's my/our fault", and my experience is that a lot of people who would rather die than admit fault. It's part of a larger raft of skills around transparency. So they cover up problems hoping for some miracle that never comes (or more cynically, they hope to be employed somewhere else before the shit hits the fan).
Deadlines presume a known fixed scope. But I’ve yet to meet a CEO or sales team or whomever gets to decide what features go in who will fix the scope if a project.
They are always willing to add features, but can never take them out.
I know this sounds like setting up a team for failure, but I’ve seen it work again and again. It sets clear expectations for quality and delivery upwards and downwards which everyone can agree on.
Once this is done, the easier part is keeping everyone focused, and using all the leftover time well to raise quality.
I have encountered an article (can't find the link now, sadly) that claimed that when developers gave estimations, breaking down tasks to sub-tasks actually had a reverse correlation to their accuracy. In other words, their first gut reactions were actually better than estimations given after going in-depth through all the details and sub-tasks.
I've noticed this as well. The last few years my 'gut' estimates have tended to be more accurate (and larger) - but since I can't support them with anything other than "Trust me, I've been writing software for over a decade", they're never taken seriously.
My guess would be that breaking down a complex software system into sub-tasks (esp. when you are breaking things down within modules) that are too fine grained fails to capture the inter-dependencies between sections of the code-base. So, you complete task A and move onto to task B, then realize you have to revisit the code you wrote during task A because task B is dependent on it and some of your early assumptions were imperfect. And then you need to fix the unit tests you wrote for task A.
And so on, and so forth.
My best guess as to why, is that there are managers involved that attempt to negotiate the planned delivery time down. They do this, in part, because it's hard to get devs to work late or on the weekend when the project is on schedule, but easier to do when there is obvious risk of falling behind schedule. So, from their point of view, the best way to get the product delivered early is to get the schedule made too optimistically.
Not saying they SHOULD do this, or even that they are consciously thinking this way, but it's what the situation incentivizes them to do, and it's what normally happens. The gut level immediate answer is based on past experience, and the long drawn out meeting produced, System 2 answer, is based on management bargaining the developers down to a shorter timeline.
We get into software development because we expect it to be a creative, challenging and fun profession that creates value and yet most of us answer to clients or employers who expect us to spend 80-90% of our time working on boring, senseless stuff. You want us to do that? Great! But don't expect high quality and on time delivery.
The real reason behind delays is that we just don't give a crap about your "social network for cows" and we can't wait to save enough money to get the fuck out and either start a business, work for a decent company or start investing.
Apologies but it feels good to rant every once in a while.
I really hate estimates and have always hated estimating projects. I do appreciate the need/desire for them by some in management. I've tried to figure out how other people do it and have yet to find anything satisfactory. It doesn't have to be an extremely simple process, but I don't really know how one gets better at estimating. The only way to get better is to just understand the domain more clearly but estimating doesn't really help with that.
Kanban does have a cool idea of just attempting to break down work into equal parts and measuring the throughput of these roughly equally sized components. The only problem is actually sizing stories to be equally sized. Some changes don't have any kind of real stopping point of functionality without large changes. It seems kind of arbitrary to chunk it out just because your project management system wants you to.
https://www.scaledagileframework.com/wsjf/
this can work really well if you pare down the list of features and build the simplest version of these features
I agree with making fine-grained plans as a way to uncover issues. Just remember, no plan survives contact with the enemy. On the other hand, fortune favors the prepared. (I actually think those are both Eisenhower quotes, aren't they?)
To use the 'going to the supermarket for milk' example. I could make a fairly accurate estimate for that because I've gone to the supermarket hundreds of times. I know all the different things that can go wrong and account for them, because I've encountered them before. The elderly person who wants to pay cash and has a bag full of coins. The guy who finds a 10 cent discrepancy on his €92,30 receipt and has to argue for 5 minutes with the cashier (while the row behind him keeps growing). etc. etc.
Now imagine that today is your first time going to a supermarket. In fact, before today you had never heard of the concept of supermarkets, or milk for that matter. How good will your estimate be ?
The only time you can make a decent estimate is after you've finished. Or to put it differently: making an accurate estimate is possible, if you're going to accept that making the estimate is going to take a long time, but I can't tell you how long.
Development time estimation, and every methodology that attempts it (I'm looking at you, Scrum), are little more than desperate attempts by managers to feel in control and relevant.
https://news.ycombinator.com/item?id=17154355
More charitably, they're an opportunity to introduce system 2 slow thinking on the team.
There are certainly novel engineering problems that simply won't yield to estimation, but there are also lots of well-solved problems that will. Managing the latter isn't desperate, its just sound practice toward reproducible results (ideally deployed only where reproducible results create value).
Certainly when you're writing a double-linked list a third time you can estimate how long will it take you. The problem is that in real life you never solve sufficiently the same well-solved problem. Feature details, matters of integration with other pieces of the system intervene - and in practice those errors introduced by new circumstances are significant comparing to predicted time. So the practice may be sound, but it doesn't give the desired results, like predictability of required time.
If you're looking for a formula that lets you estimate once and then move on to the "real work" now that that tedious BS is out of the way, there isn't one. But you can still do something. There's ground between "I can't estimate that in 30 minutes" and "I can't estimate it at all" and it's up to you and your stakeholders to come to terms on how to balance the risks in your estimates (in broad terms: too inaccurate estimates vs too much time wasted estimating requirements that have since changed).
(There's a parallel theme in some other comments: "if it's a well-understood problem why am I not using the existing solution?" Similarly, there's a difference between "well-understood" and "perfectly abstractable and automatable", otherwise almost none of us would be employed these days. :) )
But if it's a solved problem, why not use the existing solution ?
I appreciate that sometimes hard deadlines are necessary but if I can I'll always avoid them.
It works only very special circumstances: when the task can be very well defined because it has been done a lot of times before and the people doing the task are very experienced on it.
But the caveat is that if you are doing such a task you're not really doing any innovation at all. Actually, if it is so well defined and trivial it should probably already have been automated.
True innovation must have a lot of unknowns, it isn't innovation without it.
I think what's missing here is that I can predict how long something is going to take when I talk with the managers/feature-planners about what the feature is. If I feel I have a solid understanding of the requirements, which pieces of code it'll touch, etc - then I can give a reasonable estimate.
Now, the bigger the project the more difficult this becomes. Details I've missed is always a possibility, and that just stacks up the bigger the project is.
So I think being specific in this type of discussion is really important. A feature that's going to take 6 months to implement is insanely tough to estimate, and very likely to have a wrong estimate. A feature that takes a week can pretty reliably be accurate within a day or two, in my experience.
I'm not defending management in these sorts of things.. I think it's our job to not only give estimates, but include understanding about the expected validity of that estimate.
All estimates are worth while though, your post makes it sound like they're not. You tell me you need a feature that's going to take 12+ months? It's so hard to predict that it's laughably wrong.. but that's still insanely useful. Why? Because now the management knows it's highly unlikely to be a 1 week, 1 month, w/e feature. It's big.
I'd argue you can't.
Right now I'm working on a feature that I estimated as 'a couple of hours'. I've been working on it all day by now. Not because the feature is that difficult, or because I didn't get a set of good requirements. The requirements are crystal clear and should be easy to implement.
The reason it's taking so long is that I'm fighting with a UI component that doesn't behave as expected (or documented). What should have been a trivial part of this task, a matter of seconds and not worth estimating at all, turns out to take all day because I didn't know this specific component is misbehaving and I couldn't have known until I started implementing it.
You cannot predict things like that without doing it.
But to clarify: You can't predict things well that you haven't done before.
The point being, when trying to predict an unknown it's best to give a range and not a fixed estimate. If possible.
Just because you're wrong sometimes doesn't mean you can't be right more often than not.
Another way to look at it is, as a company how can you budget anything if you can't make even the slightest of estimations?
What you're suggesting results in throwing away any ability to budget, roadmap, coordinate, etcetc. Absolute prediction is of course impossible, but if what you're saying is true than I should buy a lottery ticket because my estimations are insanely lucky.
And sure, I get estimations that run long, we all do - I'm not saying otherwise.
I've done this 50 times before (10 times in this project!) and these kinds of issues still occasionally occur. How does anyone predict for this kind of thing?
If you can plan and execute putting a person on the moon, you can plan and execute software development.
I think the problem is the overwhelming majority of software development is so low stakes, easy to replace, assumed that bugs are par for the course, etc... that most orgs have stopped putting real effort into planning AND executing on that planning, instead just cargo culting the planning phase.
But the budget for planning is limited. To realistically plan out most software projects would entail almost as much work as simply executing on them. That level of resources for planning vs. execution is acceptable for putting people on the moon. Probably not for building a line of business app.
Sure once all that expensive and unknowable planning and development is done, you can double-click that executable and put a man on the moon.
Okay, but I think you've missed the point. Planning is hard, and you can't possibly account for all the variables.
Everything after that statement I can agree with
I think you vastly overestimate the quality of the software and systems that took men to the moon. Try reading https://www.google.com/amp/s/amp.space.com/26593-apollo-11-m... for example - you might conclude "they got lucky"
Do we have the planning and estimates of projects that are similar to "put people on the moon"? I'd like to see how the plan/estimates of these projects look like and how they change.
If you can accept those kinds of preparations and also agree to not change the specs much (i.e. no, we aren't going to Mars instead) after we've started developing it I'll say we can get you a pretty nice estimate.
I’m kidding, of course. It really depends on how well business and development understand each other.
Yup. This is the crux of the whole issue, and it's being completely blown out of proportion. There's always going to be details the estimator (dev) won't know, but the more clear a picture you have in your head, the more confidence you can have in an estimation.
This is directly related to the size of the project too, of course. If it's a 12 month project, a single developer is massively unlikely to have a reasonable understanding of all the potential roadblocks that the project will take. However a 1 day task, with tools you know/etc, you can have a pretty good idea how long it'll take.
There's always some unknowns, but that's literally what an estimate is for - attempting to plan for the unknowns. I can write an add function and I'd bet a lot of money that it won't take more than a day.
Unreasonable management seems to be giving developers PTSD here. As I'm having trouble understanding why the idea of an estimate, by definition exactly what we're doing, somehow doesn't apply to software development.
If you're being pedantic you could say the same about a lot of things. How long to dig a 1foot hole in open ground? Well who knows, there might be a massive rock - but that doesn't mean you can't give an estimate. It's literally what an estimate is for, an approximation based on assumptions and expectations.
It's inexperience all around. Also, most places have no clear long term track, so people bounce after 2 years rather than growing to understand each other.
Managers are generally free to cut estimates in half, and move to a new role or move to a new organization when things fall apart. They're not trying to be villains, they have a lot of pressure coming from a lot of directions and think that estimates are bogus anyway, thanks to star trek and the occasional miracle that happens. (sometimes things really are much easier than they seem)
Developers are optimistic, wildly underestimate, and feel bad when they miss them. The overreact and start massively overestimating. When they progress to the next stage they take the time to give a good thorough estimate to have it slashed by a manager, who knows they overestimate.
Nobody sticks around long enough to really understand the state the other is in, so you get this vaguely Kafkaesque world that makes everyone a little crazy.
Software isn't like building a bridge; it's like planning to build the bridge.
I'm pretty good at software estimating; I can roughly tell when something will take an hour, 2 hours, 4 hours, a day, a week, or month, 3 months, or 6 months -- in those intervals. It's more difficult and time consuming to get more specific than that.
I've also had a 1 day task blow out to entire week more than a few times. At that point, you have to start questioning the predictive value of estimates.
If I estimate a month to build your feature and it takes me a year or two, you're not going to be very happy...
That said, requirements emerge daily. Transparency is essential for honest dialogue.
I agree and disagree with the notion that an estimate can only be provided after the work is done. If the work is broken down properly, you can use empiricism to get a good understanding of what is needed based on previous, similar work completed. But to think an estimate can only be provided after the work itself is done is kind of ridiculous. If you section off variables with manageably-sized work you should have a good understanding of what's needed to pull it off.
This is the reason big design up front hasn't been the gold standard in planning. Most modern methodologies acknowledge that there'll be a lot of changing requirements and learning-while-building.
The point is, given this information, you could a) run around like crazy doing all manner of reckless and crazy cowboy programming and putting out fires b) carefully plan a relatively short period of time on a small, achievable, well defined goal (not "build the project"), then making a note of what worked and what didn't, and readjust before you plan the next short period of time
You need to have some fairly specific goal in mind to start doing anything. AKA are we building a PS4 game or a self driving car?
Methodologies like Scrum are all about transitioning a long term objective into short term progress. But, you really do need some fairly well thought out goal.
Further course corrections quickly get very expensive. Sort of like sailing from Virginia to Europe then deciding 1/4 of the way in on France then changing your mind to Spain at the midway point is probably fine. However, changing your mind to Australia is a much bigger deal.
Typically no one wants to pay more for less, but that's the best strategy to deliver high-quality software on time.
You reduce the scope drastically but do not adjust the timelines. You pay for more development resources(tooling, developers, servers, time, etc.).
If you can't deliver at that point, then there's a serious fundamental issue with either the leadership or the team or both.
Startups don't have this luxury. So they must execute on the least complex scope on a very tight timeline. High-complexity scope increases the fatality risk to the company.
So as the scope increases in complexity the risk increases and the likely the company itself will fail increases. Again, because startups do not have the cash to pay for the proper amount of resources.
I think that is a myth right up there with “If I was rich, I’d be happy.” Massively funded software projects fail all the time.
That which is passed off as novelty is often ignorance (not necessarily on the part of the developer.) To continue the retail analogy, software projects are often like the guy who is building a deck on his house, but keeps going back to the hardware store because he made no attempt to think things through, or measure before cutting (note that, in this analogy, the would-be builder is not necessarily playing the role of the developer. In some cases, that might be the hardware store owner's role.)
I envy _hugely_ anyone who builds software that is even 50% the same as what they were working on six months ago.
Sure, but all you're doing is moving the uncertainty from the development to the planning phase, so instead of a development phase of unknown length and cost, you end up with a planning phase of unknown length and cost.
Waterfall gave the impression that planning and analysis is costly by insisting it should be carried down to the lowest levels of abstraction, before doing anything else.
I disagree. Even when you're doing mundane everyday stuff you have done a thousand times before, one trivial variation may throw of your estimate completely. Maybe an edge-case in this specific project triggers a bug in a 3rd party's library, and you're losing weeks waiting for the 3rd party to roll out the fix (been there, done that). You just don't know in advance
> you can do analysis and planning at a much higher level of abstraction than development
Of course, but then you'll get an inaccurate estimate.
> Waterfall gave the impression that planning and analysis is costly by insisting it should be carried down to the lowest levels of abstraction
That's only necessary if you want an accurate estimate. I think accurate estimates are pipe dreams.
It's called spec programming, where you have programmers slinging code against precise specifications. Common in ERP and regulatory or business compliance-driven code.
On the other hand, you have development. That is where you're attacking the less defined problems. Sure, we know the tools in our toolset, how to apply them, and time it takes to apply each tool. Problem is, the time to figure out the series of tools needed and their precise applications burns enormous time, time that could have been spent towards a MVP.
In the use of the adjectives 'precise' and 'enormous', you are just repeating the same false dichotomy in a slightly different form, by ignoring (or not understanding) my point about the use of abstraction in analytical thinking.
The people who are most successful in what you distinguish as development understand a good deal more about what they are working on than spec programmers need to, and they are not, as you imply, groping around in the dark for a solution - that's what spec programmers do when faced with a development problem.
The first involved taking all sorts of situational data from various sources in an airport, e.g. radar systems with their own binary formats, and combining them with the work of a postgrad researcher who had come up with an algorithm to optimise when pilots should turn on an aircraft's engine in order to save fuel and cut emissions.
The second involved managing 3G dongles on fleets of laptops, tracking and sending alerts on their usage and billing info in realtime, back in the days when business users would accidentally run up £10,000 bills abroad.
If at any point in the first half of either of these projects I'd been asked to come up with an estimate for the total amount of work, it would have been wildly inaccurate. Each project had both low and high level surprises at every turn that would have just compounded to blow those estimates up. There is basically no significant way in which one informed the other, even though parts of the systems used similar event sourcing architectures on the exact same server-side stack, and were done for the exact same startup which broadly offered real-time systems to utility companies of various sorts.
It's entirely possible that there exists a breed of programmer who could have estimated these projects in their entirety, much earlier and more accurately than we did, but I've never met them, and I'd rather work in places that didn't compel people to.
I've known and worked at other places where basically any software methodology would (and did) work, but I certainly don't think it's rare (as you seem to suggest) to find yourself at the other end of the spectrum.
Doing something about it takes some commitment, however. Nobody estimates well without having practiced it, and been honest with themselves about where it went wrong.
For the things I work on, and frankly always have, I spend most of my time in the “R” part of R&D. Now granted, I’ve never worked on classical “business” or CRUD type problems.
It’s natural to think that once you have lots of experience, that estimating will be easier. In my experience, I found myself simply tackling harder or unique problems. Instead of getting comfortable, you find yourself mostly operating at the edge of your competence. This is great fun [1] but makes estimating pretty hard.
[1] I personally enjoy being on the “black diamond” part of the learning curve. (“Black diamond” = most difficult ski trail)
There is also a lot you can do to decrease impact of novelty on the overall schedule and most people I work with do exactly nothing to reduce variability in the process and improve their estimation ability.
It is easy to claim every time you write code you are doing something new but, hey, every time they build a building they do something new. Yet buildings (except for extremely ambitious projects) typically can be delivered reliably by reputable developers.
If you're really working on new problems all the time, consider yourself extremely lucky.
- how long will research take? Where do you buy milk? Where is the nearest shop? 3 mins.
- how long to get to nearest shop and back (eg 10mins x 2) = 20 mins.
- find milk in shop (30 secs to 5 minutes), so say 4 mins.
- Buy milk. (1 to 5) say 3 mins.
Don't forget that these are estimates and always add a bit of headroom. The above comes to 31 mins, so say 35 and you know you're covered. If the manager isn't happy with it, give him the low down figures.
This is exactly what we used to do in the waterfall days. It didn't work.
The only way that's been compatible with me is to build something really small but valuable. So small that its hard to be disastrously wrong. Once you've released that value, build upon it.
Stakeholders tend be much happier as they at least have something they can use really early on.
And most organization -- or collections of three or more people -- have dysfunctions. There are rarely process problems where technology is the main problem; it's the wet-ware between the keyboards and the chairs that make or break projects.
I find that in the case of the second solution: "thorough planning" you get a bunch of people fabricating estimates of estimates with padding or not padding or trying to estimate a bunch of unforeseen events that are too far ahead to get an accurate handle on. Sure, the second is likely to be much more accurate just because the second is likely to be much, much longer but I have never given one of these estimates without it being followed up by immense disappointment by business with a desire to strong arm me into something shorter. This demonstrates a fundamental issue with the topic. Its not about the estimate at all.
I think the problem with estimates is their finality and assumption of correctness. I figure once you're in the developer-years category you might as well iterate the estimate a bit to get a better sense of it. Error bars should be translated into risk for the business decision.
Too often I see people claim to be making "rational, facts based" decisions on estimates (beyond 1 year+) are complete codshit. This is not rational decision making. These decisions should be about risk management assuming failure as opposed to thinking you can slot year+ development estimates together.
I think very often the desire to lock down development estimates into "rational fact" are business decisions of risk masquerading as technical developer decisions of fact. I have yet to see a situation where we deliver an estimate that blows a business decision out of the water where the business just backs down. It just learns to ask a different question and gets the answer it wants out of that.
One small reason for that is poor time estimation, but having good time estimation won't make your software succeed. I think having good planning, makes for a good working environment, but it doesn't mean the project as a whole will succeed. It might mean the developers will be happier to be with you and pivot to the next idea.
I would like to look at the percentage of successful projects and see what proportion of these had good estimation attributes.
I'm fairly realistic about software estimation which only came after a LOT of retrospection. It normally takes what I estimate, both personally and professionally. The hardest factor I've learned to include is the level of detail. For instance with my personal website [1] I gave myself a full Saturday because I knew I wanted high detail level but I had in mind the overall design. It took the Saturday +1h of a couple of improvements/bugfixes (under 10% error). With my current job I'm also under 10% error.
In the past I have been bitten a LOT about my unrealistic estimations, so the only solution I had to move forward was to learn from those and so I did. So now, from the article, I know that my "quick thinking" is around 50% of the project. I force myself to think a bit more and the details trickle down.
Another thing I've learned is that projects tend to fill as much time as possible (Parkinson's law [2]). So if you are told a deadline, half it! Put the half as your internal deadline, then the project will be just on time.
Finally, complexities are exponential, so learn to say no to unnecessary cruft. "A small change" might seem like a 1% change for business and for you initially, but it will more likely than not grow into a 10-20% change in the end. FFS that is why the duck was added in the first place, to avoid wasting time [3].
[1] https://francisco.io
[2] https://en.wikipedia.org/wiki/Parkinson%27s_law
[3] https://rachelbythebay.com/w/2013/06/05/duck/
You can plan your trip to the store to buy milk and figure out exactly how long it will take, but the only thing that actually matters is that you arrive back home, with milk.
If that milk is absolutely necessary, whether it takes 30 minutes or ten minutes to get it is really a secondary concern. If you spend five minutes getting a better estimate you've delayed the milk by five minutes regardless of how long it takes or how right you were about the timing.
I think we spend too much time thinking about time estimation when the planning we should be doing is figuring out what is so important that the time it takes to build is worth it even if the time estimates are off.
Hofstatder's law: It always takes longer than you expect, even after accounting for Hofstatder's law.
Create your high-level backlog and do MoSCoW prioritization. Figure out your "musts", "shoulds", "coulds", "wonts". Now apply some estimates to your features and add 10% for unforeseen growth. Estimate velocity based on team size and now you've got a date when you could conceivably hit your musts, shoulds, coulds. Set your "deadline" if you must somewhere deep in the coulds. If your musts go over, you are still able to launch an MVP. If things go well, you can start delivering non-musts.
Adjust your plan every sprint based on actual velocity.
Use the frameworks exactly as they are intended, don't try to invent new solutions that aren't native to the framework.
Anything that takes you outside the beaten path in development is going to be a potentially infinite black box.
In other words, a lot of software engineering can't be put in timeboxes because it's actually R&D more than it's development and where each little step forward can add a potentially infinite amount of new tasks to be done or problems to be solved. Add to that the constant need to update, upgrade, improve, re-design and you know it's just not doable.
So the primary problem IMO is that we think about a lot of development as if it's something that can be put in boxes. Some can of course and the better and more solid the team becomes to better they are but the teams who struggle are mostly struggling because the expectations for what they are actually doing (inventing problem-solving) isn't matching up with what they are being paid to do (build)