"Bro, I'll just get some other nerd to do it. Instagram was built by 12 guys and WhatsApp was acquired for $19B with only 50 engineers. Can't ship if you whine; so shape up and stop wasting time!"
A few years ago I was brought in lead a team in a division of a public company. Our mandate was to replace a major component of an enterprise system that you couldn't quite call legacy because they'd never actually gotten around to replacing the real legacy system with it.
My initial proposal was to build it incrementally: tackle the obviously critical functionality to get something working and ship it, then define and implement additional features on an ongoing basis. I had hoped this would fly, since the company talked a big game about being agile.
But no dice. We had to have a complete plan for all the features currently supported (some of them of dubious value, many of them incomplete and inconsistent in specification), and it had to come with a specific timetable. Under protest, the team and I worked hard to produce high-level estimates, and ended up basically guessing that the thing would take six months.
No dice, it had to be delivered in three. I made a suggestion along the lines of the strategy suggested in the article -- we could identify a subset of features that the team would be comfortable could be delivered within the deadline. It would probably be stronger, I argued, since it wasn't clear that all the added weight added much value.
No dice: everything was priority one. I pointed out something like, "you can't fight the laws of physics". I apparently gave the impression that we'd get it done anyway, though my memory of the conversation was that I was sternly disagreeable.
Our team goes ahead and starts implementing items from a value-prioritized backlog. Fast forward three months, and we have a working system that supports the most important use-cases. We considered it past ready to ship, knowing we'd need to keep iterating. The response from management, predictably, was frustration at the missing features, despite the advance warnings.
Management goes into "high-pressure" mode, and for the next three months I do my best to keep the team insulated. After more or less six months of total development time, we finally replace the prior component. All the users agree that it has far fewer bugs. I and many of my team members grumble that it has far too many, on account of the fact that we weren't given the chance to ship the minimum viable product when it was ready.
I'm not really sure what the moral of the story is.
Incredibly hard to do when you have the full weight of management on your shoulders. Especially if they back you into a corner in a meeting where you are the only technologist facing a whole hierarchy of management types.
I completely agree on how hard this is. However, I've built the best relationships with managers where I kept my foot down once in a while. I guess they like the struggle and the honesty in the end.
I've run a business longer than a decade so I know how hard it is and I've burned out several times over. But I just don't give up and a no has certainly become a NO.
I feel like this is why lead technologists turn over so often. Because they say the harsh reality that no one wants to hear, and no one cares that they are right, they just care that they are not delivering to their expectations. Maybe the issue is the expectation.
Did it result in burnouts?
I was part of such a mess and two out of four team members got burnout, myself one of them.
It did teach me to get the fuck out early the next time I feel the same emotions of inadequacy and pressure on me. And in the very next project after a very sudden ramp up of backlog and increased pressure I looked in the mirror and got out, cleanly and without burnout or bad feelings.
Helps to be on one death march project in your life to know what it looks like. Don't do more than one. It can damage you, give you long-term health problems, lead to substance abuse, etc. When I see a thing like that 10 miles out, I go "Nope, nah-ah - I am out of here".
The moral of the story is that hard-to-please idiot customers/employers don't want to pay more than realistic, normal-to-please customers/employers. So there is absolutely no reason why anyone would work for the first when the second is available. And believe me, the second is out there.
I've seen plenty of (experienced) businesses and individuals that refuse to work for toxic clients, because there is plenty of other work that pays exactly the same but without all the shit.
P.S.: According to your story, you sound like a good manager :).
Thanks for sharing; to me it sounds like a successful project which management/customer made feel like an unsuccessful one due to unreasonable demands. Way to ruin morale!
Well done for at least attempting to introduce some flexibility. One lesson for the future might be: how to more forcefully stand up for what you believe is the right way to go - but of course this is easier said than done!
A lot of managers think it's their job to decide what the quantity of things is. Like in an assembly line, we need 30 turbo encabulators at X price and it's your job to make it.
Software has an infinite quantity of numbers to use as tools to build an end product. A programmer's job is to decide what general ideas end up as what specific quantities. A manager cannot decide this, lest he do the programming job himself.
Money is a great tool for measuring quantities of material goods, or the value of material properties. But electricity provides infinite quantities, exchanging money for programmer time is closer to exchanging two forms of currency (binary numbers and dollars) for one another. Code is a constantly transforming river of numbers that we make a draw-down into a bucket that we call an end product. That is not a quantity of material good like iron ore, at all.
The factory model simply doesn't work and producing code should be treated closer to the stock market than the factory model. Early access/kickstarter/pay-over-time business models in video games represent an example of the constant transformation model working successfully. Business apps should be built by crazy people who will build them anyway and money holders should bet & invest in them like stocks that raise and lower value over time, rather than as promised end goods.
Honestly, I've seen highly technical managers make this mistake worse than the non-technical ones. Non-technical people usually ask sincere questions before shoving an unrealistic deadline in your face. Technical managers may have already told someone in the C-suite "ohh yeah, that's easy" before talking to the guy who has to do the work.
I've had ex-programmer managers say "I just want you to add this graph to the app. I could do it in 20 lines of python." In reality, the request is more like "Add in this plot which is the result of a long running calculation. The calculation has to run in a background thread so the app is still responsive, even though the whole program has been architected with a single-threaded design. It needs some mechanism in the UI to indicate it is making progress. We'll also want a way to cancel the task. Half the datasets are using a different sign convention, so you'll need to automatically handle that. Actually, the data is polluted with garbage, you'll need to spend an unknown amount of time debugging a legacy system to figure out where the bad inputs are coming from in order to understand what can be done to filter that out...so we can have this plot by end of day, right?"
Apologies for ranting about my "internal software" days.
From my experience (3 technical co-foundings, 1 additional top-tier startup, other companies) it’s rare for management to incorporate our estimates. I’ve been coached by a VP Eng on how to blow off actual estimating, in that top-tier startup. Sometimes, it may be quasi-rational in that the apparent business constraints just don’t care about the estimates, in which case, yeah, just cut to the deadline and budget and we’ll see if we can do anything that isn’t embarrassing. Management usually dictates rather than engages, that’s how you get to be CEO in the first place.
“We” are usually not at all clueless. “They” tend to be.
On HN, moderators edit titles that are misleading or linkbait, as the site guidelines ask: "Please use the original title, unless it is misleading or linkbait" (https://news.ycombinator.com/newsguidelines.html).
It's a linkbait trope to use "you" in a title, because it grabs attention whether the topic has anything to do with the reader or not. That's why "you" is headline writers' favorite pronoun. Combining it with a pejorative ("you have no idea") makes it even more sensational. That's definitely the sort of title we edit.
When we edit a title, we look for a representative phrase in the article itself that expresses its point in a more neutral way. That's what a moderator did in this case. The language comes from the article's own summary of itself: "After all of these years, I finally came to one simple conclusion. With all due respect: we are completely clueless about how long things should take." Reading the article text to find how it states its own conclusion, removing any residual linkbait (such as the superlative "completely"), and making that the title instead is the best way we've found to correct titles that break the site guidelines.
> It is our belief that over the 30 plus years of commercial computing has developed a series of sophisticated political games that have become a replacement for estimation as a formal process.
Unrelated: can we not do this thing with the whole left side of the screen being one static image? It's really distracting.
Timelines kill me. When I'm half way through a project I can tell you exactly when it's going to be done. Or I can plan the shit out of it upfront and give you an accurate timeline but that doesn't fly because you need the timeline upfront. I like working - can't I just work till it's properly done?
This sort of thing is why engineers are paid way too much, and way too little, and why so many see the profession as a young man's game. You're just not the master of your own fate. Sure, the computers are predictable (for some value of "predictable"...) but management is not---even if they try to be.
I've been mentally moving away from being paid to write software. I can see writing it for my own use, even professional use---I just don't want to be on the hook for ever-changing requirements, decided by people who are often kind, but not, in the end, competent. The best managers understand this and will give you leeway, but this is not a sustainable, repeatable thing. It lives and dies on one relationship.
I wonder sometimes: what if we just all stopped writing software one day, and started just using it? Writing software is a bad deal in a lot of ways---hard, socially isolating, etc---while using it is amazing---the computer does the work for you!
Don't get me wrong, I spent an hour at work today presenting on Lisp macros and loved every minute of it. But a dev career, for many, means a capped income and a razor's edge of apparent competence.
> I wonder sometimes: what if we just all stopped writing software one day, and started just using it? Writing software is a bad deal in a lot of ways---hard, socially isolating, etc---while using it is amazing---the computer does the work for you!
What's the difference, though? It's insane how much of my work the computer does when I write code these days.
> he best managers understand this and will give you leeway, but this is not a sustainable, repeatable thing. It lives and dies on one relationship.
This is one of the reasons you often hears about an entire team moving to a new company. They have a dynamic that works, and they do not want to risk it.
What jobs don't have drawbacks though? I have been thinking a bit about this:
1. Doctor: In many cases, throw away your life and work 60+ hours and also you need to specialize and study long and hard.
2. Laywer: I don't know enough about the profession. The job doesn't transfer well to other countries.
3. Consultant: 60+ hour days are the norm, interviews tend to be based on quick thinking in the high school arena. To pass the interview, you need to have excellent and super quick high school level knowledge of: math, logic, social and political skills. This sound denigrating but I found it tough to do this quick.
4. Investment banker: 100 hours
5. Construction worker: your body will thank you later (/sarcasm).
Programmers have a certain set of advantages and disadvantages (I agree with your disadvantages), but how is it worse than other white collar jobs?
You're not wrong. The labor market is relatively efficient.
Go on a reasoning chain with me:
- what solves the problem nicely is to sell software. Selling good software can be one of the easiest and most lucrative jobs in the world. In practice no particular employee gets that---the company pays enough to motivate, but takes the rest. The solution, then, is to be the software company.
So...start a startup? "What a novel idea Dropit, on HN of all places!" I have actually "done this" (or thought I was doing it) multiple times (failed every time), but looking back I can see a lot of trivial mistakes I made. But at least I can find some---with many code-for-hire fiascos, the mistake was taking the job in the first place.
So my conclusion: accept the job you have, for now, while saving money and trying to have a good, normal life, and put some effort into seeking out new opportunities. FU money is a thing, as is FU market position.
A strategy I thought out today was as follows (note: I'm based in The Netherlands):
1. Get a job for 32 hours per week
2. Work 10 hours on week days and 8 hours on one weekend day (take the other weekend day of), so that you clock 58 hours per week.
3. Don't take up your vacation, safe it.
4. Take the other 6 months off.
5. Oh, and pay less taxes. You're handing in 20% gross, but net you're only handing in 15%.
I think a schedule like this works for people like me, because I like to work hard and earn my freedom and then relax and doodle around for quite a while (i.e. 2 to 4 months) and then do a small side project and then work hard again.
Since I'm at the beginning of my career, it sounds like an interesting experiment.
Not for my situation, but for others: geo-arbitrage becomes interesting as you can literally fly to Thailand for 6 months and come back (a cheap retour ticket is found for around 400 euro's).
"this is not a sustainable, repeatable thing. It lives and dies on one relationship."
That can be quite sustainable; that one (or few) relationships can be long-lasting. If good managers leave and are replaced by bad managers, often quite soon team members will leave to join that manager in their new company - it's a well known fact that the direct manager matters more for job satisfaction than the particular company.
I know people who over the years have worked at 3-4 companies for the same boss (not continuously), and if I was unhappy at my current position, I remember a few previous managers whom I'd call to ask 'are you hiring?' and in the tech field the answer pretty much always is positive.
You might consider whether you are ready to go into management. We know the trope of the engineer forced into management, who did not want that job. But I think the opposite is just as common -- great engineers, who would also be great managers, that never (want to) make the transition. Understandable -- a good management day seems less enjoyable than a good code writing day. But I bet most companies, careers, and products would be the better for it.
I have wanted to make that transition for years now, but the opportunity never presented itself, I can't even find a way to force my way into the opportunity. It seems like all management in every company Ive worked at were coworkers of the founders at a previous company.
I'm not sure how to avoid answering these loaded questions though. In my experience trying to 'negotiate' with people like that: they will just ask you the same question in 5 different ways and it will eventually get so uncomfortable you'll tell them anything to move on.
Another option might be to never do business with cash-starved companies. Founders and execs will be constantly worried about running out of money, acquiring customers, and pulling off miracles, and all that stress will trickle down on you probably for no real benefits.
A key question to ask an employer then is how much 'run way' / money they have left? Or whether they have a revenue source. It seems fairly probing, but realistically, not everyone wants to invest heavily in a company that might not even be around in 6 months.
On the flipside, necessity is the mother of invention. How many behemoth companies do we see now that sit on their hands and don't bother innovating at all? There's surely a balance to strike here between desperate and panicking and fat and lazy.
I think what we're looking for here are healthy buffers.
Otherwise the engineer is relying on (possibly blind) trust that management won't try to "motivate" him/her into performing miracles (ie. breaking the project management triangle).
> I'm not sure how to avoid answering these loaded questions though. In my experience trying to 'negotiate' with people like that: they will just ask you the same question in 5 different ways and it will eventually get so uncomfortable you'll tell them anything to move on.
Once you realize what's going on, you can turn that into a game.
Do you know the game where somebody asks you a bunch of questions, and they you allowed to answer with anything except "yes" or "no"? It's very hard to do, because it's such an ingrained habit.
If you really want to not give an estimate, make it a game to not give one.
But, give them something else instead. Work out a bunch of questions about uses cases / data volume / whatever, and say "if those were answered, we could build a prototype in a few days that would let us make a more reliable estimate" or something like that.
Another comment: coming up with good estimates is work. The other day somebody asked me if I could come up with a rough estimate for a (poorly specified, IMHO) project, and my answer was: no, I don't have time. If you need it anyway, formulate it as a task in Jira, so that it gets prioritized along with all my other work.
(Fun fact: we estimate our tasks in story points, so then we estimated how much time it would take us to come up with an estimate... :D )
"and they you allowed to answer with anything except "yes" or "no"?"
None of the Celtic languages have an equivalent of yes and no. Just make a habit of speaking, say, Irish or Welsh at work. (To an English speaker, that sounds weird at first, but you get used to it quickly.)
At certain points, this article touched on the commoditization of engineers and engineering skills. Beyond extremely simple things, it's never going to happen. Especially as you build a company and end up with a few monoliths through various acquisitions, all while concurrently hundreds of microservices. Things get too intricate and too hairy between systems. There's no amount of handwaving that will convince me that machines will replace Software Engineers anytime soon. Solving valuable, enterprise-scale problems will never be as simple as a Wix drag-n-drop solution.
I'm not so sure. I think this is something that's happening right under our noses -- it's just easy to miss it if you're not looking at the right thing.
Software engineers aren't being commoditized by being replaced by machines that write software. They're being commoditized by their own frameworks, libraries and tools.
Take the game industry as an example. Twenty years ago, your game company needed a big team of software engineers employed to write a game engine with advanced graphics capabilities (let's assume you want advanced graphics). Today, a single developer can just download Unity or Unreal Engine and have at least the technology available to them immediately (art is different but in many ways similar; automation and process improvements are coming for those jobs too, I'm sure).
So you don't need the same number of engineers for the same result. Sure, you have a big team of engineers employed at Unity Technologies or Epic Games, but that's now a shared resource. That employment is no longer duplicated at the companies that decide to use those engines.
Another example is the push for 'DevOps' and 'Cloud'. Think of all those system administrator jobs and IT departments being made smaller because now you can just spin up a server on AWS or have your CI infrastructure managed by BitBucket.
> Another example is the push for 'DevOps' and 'Cloud'
It's been my experience that delivering business value is taking longer because of this, not less. It's hubris to believe that a single person can be competent enough in all these domains to replace multiple people who focus on specific domains.
By distilling DBAs, Configuration Management Engineers, System Administrators and Software Developers into single people business is getting shittier products less frequently which incur more operational overhead.
Death by a thousand cuts sounds more believable. I concur with your position: instead of it being black-and-white, it could simply lead to less demand for engineers as the "building blocks" -- really common open source technologies with nearly omniscient presence -- are essentially commoditized.
On the other hand, I could also see a reality where, since many low-level problems are solved for you, management expects more out of you, so the number of engineers stays about the same, but you get a higher level of productivity.
Agree with your second paragraph and that's actually been my real-world experience. I also think there's another effect where commoditizing a technology leads to the creation of jobs that specialize in that technology. With easy-to-obtain game engines, suddenly more companies are interested in using them, which itself leads to an increase in demand for engineers, just with different skillsets.
I wouldn't put money on this balance lasting forever though. To me, that's too close to dismissively saying "it's different this time", with regards to our profession.
One of the most painful types of people to work with is the CEO / founder who got lucky with a product early and came to believe they're a "product guy".
As in: My early product hit whatever trend / wave / need therefore I must be a product guy (and not just lucky).
The product guy has a preternatural ability to understand what the masses want. Watching them work -- witnessing their process -- is something to behold. They will steer products in a direction regardless of cost, complexity of likely outcome.
The outcome, quite often, is to tank their company. Since they don't understand why they were successful in the first place, it's very likely that their success won't last.
But if you're along for the ride, wow, expect the following:
1. You're the greatest (available) engineer we've ever encountered, building super-complicated XYZ is going to take this company to the next level!
2. This is taking much longer than expected and isn't matching up with our expectations but I'm 100% sure of my vision because I'm a product guy.
3. We're running out of money (because the market conditions that gave us early success have changed) and super-complicated XYZ isn't going to rescue us -- because you're a worthless piece of shit of an engineer!
See what happened there?
They're sometimes hard to distinguish from a vanilla bullshit artist. The bullshit artist will tell you how well capitalized he is, tell you he only wants the best (meaning he thinks you're expensive) and then try to slowly whittle your sense of self worth down until you get "the offer":
The offer is game-changing, life-altering for you: Instead of continuing to pay you with money, they're going to start paying you with magic pixie dust. The magic pixie dust will make you rich "when everything comes together."
When you tell bullshit artist that you don't work for magic pixie dust, that's when you learn that, in fact, you're a worthless piece of shit of an engineer.
I actually respect the bullshit artist more: They're bullshitting other people but they know they're full of shit. Product guys, depressingly, bullshit themselves.
Makes me glad I'm not a product guy. As an engineer you only risk your own hide or the work you do. As a product manager, you risk the entire enterprise.
It baffles me someone can be a product guy without understanding the market. Surely where, how & why the product sits where it does in the marketplace is intrinsic to the value of the product to the customer to begin with.
The author illustrates how easy it is to be clueless about how long things should take with his parenthetical about how trivial it is to make a Wix website. Yes, you can get the “used sailboat” class of website in Wix in an hour, but that’s probably not what you need or had in mind. The fact that the tool seems to do so much of the work for you will arguably just make the expectations even more unreasonable.
99% of Wix users need a website that's just a big phone number and street address that gets indexed in Google. A "used sailboat" is exactly what they need.
That's only true if you assume that 99% of Wix users are Wix users because they correctly picked a simple tool for a simple need. But that's a tautology. The problem I describe arises when someone looks at Wix, gets the impression that it is a tool that magically makes it trivial to create a great website, and then assumes they can get the yacht they have in mind for the price of a used sailboat. Most of the work involved with creating a good website isn't technology, but rather strategy, writing, design, and other things where Wix won't help you or even gets in the way.
‘we just need a rough estimate, we won’t hold you to it’.
‘Ok based on the 2 minute conversation we just had I think about 3 months’
‘What! That seems way too long’
Other times I’ve been asked for estimates on features even though there is a hard deadline due to some external factor. I really fail to see the point of estimating anything when there is already a decided upon end date. Anyway I usually point out that they will need to just put the features in order of importance and I’ll work down the list. And they should start thinking about what can be cut. This usually leads to protests of ‘we have already cut everything we can’. But I have to laugh as we get closer to the deadline that suddenly not every feature is as important as was originally thought and magically get cut.
Then when the deadline does whoosh by - nothing changes and the sun rises just as always. It turns out very many deadlines are not in fact hard, they were just thrown on the table in some meeting and everybody starts to act like it is the end of the world if that date is passed.
Very true. Plenty of projects I've worked on have had immovable dates that suddenly become movable once you get close enough to them.
On the flipside though I do think a deadline is necessary for everybody involved, developers and managers alike. It really helps to limit feature creep. And it forces people to think about what they really want or need.
I always talk them into making an ordering, and it's not difficult. If my managers fail they escalate to it to me. It usually goes something like this:
Me: You need to prioritize the items.
PO: I cannot, they're all important.
Me: If you do not, we will make them by the order we want,
possibly coin flip, but probably in order from easiest to hardest.
PO: Fair enough, you get them all done anyway.
Me: That is not a given and you know that, but I will send you an
email for confirmation that any of them can be dropped to
meet the deadline, okay?
PO: Hold on, can I at least pick a subset that you know will be done?
Me: Sure, and don't stop until you have roughly 3 equally sized,
by estimate, categories: Must-have, Ought-to-have, Nice-to-have.
By the time we're heading into the "Ought-to-have" I tell them to do it again. I fear that I some day might be in a position where I do not have the weight to do this, but as it stands right now, not a single developer produce a line of code if someone waltzes in and tries to decide both scope and deadline.
You know, I've seen a number of Uncle Bob talks where he emphasizes professionalism and every time he does I envision interactions like you've described. But every time I hear this I think to myself, well he's in a unique position because of his brand and companies seek him out for is skill set; he has the luxury of being professional.
You've added the qualifier he never seems to add, that someday you might not be in the position to act professional. This is a sad but accurate commentary on the current state of things in our "profession".
I guess also some people find out by trying. It's scary when you don't know if the response will be "fine, ok then" or "hm, well talk about this later", and you find yourself demoted or encouraged to leave (Europe) or fired (more a US thing I guess).
If you're punished for professionalism, you know it's time to find a different place to work. If you don't it's a one way ticket to stress and depression.
most managers is drug&alcohol infused money starving 10 year olds with no fucking idea what they are doing and knowing only how to abuse people like their fucking abusing power figures called parents
The Content-Type header of the article is set to the empty string. It still renders, but I'm guessing the author hand-coded a website and screwed it up somehow. He probably should have used one of those Wix-style solutions he mentions in the article.
Great article. This is very true for game development as well.
But serious question - for live products especially, you do need to have some sort of schedule where you are launching new features every X weeks. So it's important to know how long your features will take, so you can have a constant cadence of updates. Plus a lot of times you will have marketing initiatives or other things that need to be coordinated with your releases.
So my point is that you cannot just remove estimates. There is a need for knowing when the current sprint / feature will be completed, and being somewhat accurate about it.
I do really like the point about re-framing the conversation to start by asking the manager how long they want the engineering team to spend on the new feature. That will definitely change the dynamic and hopefully should encourage a conversation about what is realistic to do in the timeframe that the manager has in mind, and how the feature needs to change in order to achieve it.
But after that, the engineer still needs to go through and create estimates to make sure what they just agreed on is actually possible, and then those same estimates are necessary to plan out the development to make sure you are on track. So yeah, you can never really remove estimates.
> you do need to have some sort of schedule where you are launching new features every X weeks. So it's important to know how long your features will take, so you can have a constant cadence of updates.
The only way you can have new features every X weeks, is if those features take no more than X weeks to develop. You can estimate a new feature to take 2X to complete, but that would still mean it can't be released "in time".
Of course, you might still want to have a regular cadence of feature announcements, or at least be able to plan them in advance. But I feel like the best way to do that is to decouple finishing a feature from releasing it.
Estimates are primarily useful in deciding what tasks to pick up first. Luckily, that usually needn't be that exact - you don't need to know to the hour how long some development is going to take, just how much faster one thing will roughly be compared to the other. A manager can then decide whether it's worth it to risk picking up a larger task that might provide substantially more benefit than the smaller task.
> The only way you can have new features every X weeks, is if those features take no more than X weeks to develop.
You can still have releases every 2 weeks, where each feature takes 1-2 months, if you have multiple small teams each working on a different feature at the same time. That is how we do it.
But of course, it's typical that each feature takes an extra 1-2 weeks of development time and many times other devs are pulled off their own projects to help out, so then those other projects are even more delayed.
Its honestly so so nice when you have a project manager that understands being Agile(tm) is about the schedules and deliverable reacting to shifting realities as well as the engineering team. I guess too many pms don't understand that they shouldn't stick their neck out with what they don't have yet and paint themselves into a corner with promised deadlines.
My understanding is that it is harder for engineers to estimate how long a job takes, than to do the job. That is to say, that the complexity of doing a time estimation task is higher than the complexity of task you are estimating. I read this in the book "Making Software: What really works and Why we believe it" [0]. The assumption in the article is that an engineer can produce reliable estimates for complex work, and appropriately guide their managers, and don't think that's true.
I recently estimated a task would take 30 minutes.
It took 40 minutes to do that estimate.
But, TBF, several important decisions were made during the process of estimating, such as what should be excluded from the task, basic organization, and some research.
BTW the task being estimated was doing time estimates for a project (which came out to be 2-3 weeks).
This is often overlooked and undervalued — upfront time spent on understanding the problem and designing before getting down to the actual work ends-up saving a lot of actual time. But most people don't have patience for that and think it is wasted time.
Yeah I've only been coding professionally for a year now and I do SO MUCH MORE reading/researching, planning, annotating, and pseudo coding than I ever did when I started.
The outcomes are so much more predictable / better quality.
> I recently estimated a task would take 30 minutes.
My boss at my last job told me that he had observed that if he asked somebody “can you get this done by the end of they day”, he would get an accurate answer (either yes or no). Any further out, there was no correlation between what they said and what they actually delivered.
In that case it seems like it would be better for PM's to take the role of task estimation. Of course this only works if they are also the ones accountable for inaccurate estimates.
I don’t think the article assumes an engineer can produce reliable estimates for complex work; in fact it claims the opposite:
> But the reality is that if you can make a probabalistically accurate estimate, then its likely that the task should have been automated by some other means already. In other words, its easy to estimate a task that essentially amounts to copy and pasting some well known CRUD API end point patterns, but any even remotely creative or novel work is almost guaranteed to be totally unknown.
> I don’t think the article assumes an engineer can produce reliable estimates
Well, he sort of goes back and forth, but he includes this:
> The engineer comes back with this simplified description and says he can get a first version produced, but it will take a month instead of 2 weeks.
Which I see some variant of every time I see somebody rail against the unrealistic expectations of software estimation (which, by the way, I’ve been seeing people rail against since the late 80’s to no avail). The implication here is that if the manager had just listened to the developer and accepted his initial estimate of one month, the software would have been done in one month: the developer could estimate with precision, but the manager bumbled along and screwed everything up by trying to negotiate it down.
This is a dangerous position to take unless you’re absolutely sure about your one-month estimate: if you say one month, he says two weeks and you look him in the eye like the alpha wolf say, “no, one month, and no sooner”… you had damned well better be able to deliver in exactly one month. The reality is there’s probably no way to tell, _especially_ if other people are involved, so you’re better off shrugging your shoulders and saying, “yeah, sure, two weeks”, doing as much as you can, and preparing your story ahead of time.
Estimates are hard because you are literally making something that has never been made before. Yes, you have experience with sub parts and similar patterns but not the situation of this timeline, this team, this technical debt, etc.
Too often the request is along the lines of "How long does it take to build a house?" not "How long will it take to build the house in these blueprints with these systems on top of a mountain that is also impervious to mudslides?". People can generally estimate the former but the latter is all about the unknown details.
My usual way of doing this, is by spitballing an estimate with the team, multiplying it by 2.5 or 3, inventing some meaningless milestones and gantt-charts and presenting it to an outraged management. Then we haggle down to something realistic + a thin buffer. The management has a sense of control, we have realistic deadlines.
Caveat: I work in the financial industry, where the complexity of writing a compiler and whipping up a Tableau dashboard are perceived to be equal.
I only hire people who can reliably estimate. How can you call yourself a pythonista if you cant reliably estimate. Incompetent estimators have no place in my organisation.
These estimators don't exist. Such a reliable estimator has more incentive to use his ability to estimate tech stock prices and become rich rather then work for you.
In fact hearing this line, no one will work for you. Frame it and make it your motto, see what happens.
They sort of do though. As hyperpallium alluded to above, the trick is to vastly overestimate the time required, then let Parkinson's law take up the slack.
Estimating knowledge work is always more unwieldy, and in this context you are often working towards outcomes which ultimately have no precedent (although they may be made up of known components).
It is also more difficult to initially assess the skills fit of candidates for knowledge based work, especially those that require creative problem solving, and unlike other engineering disciplines past outputs are opaque and hard to rely on as simple markers of past performance.
For a project, in order to produce a good estimate you need to understand scope, then align it with precedent, adjust for your resources and productivity profile and then view all of that through a risk lens to set probable outcome ranges.
For a programme, in order to produce a good estimate you need to understand and manage the risks, constraints and dependencies across all your projects and ensure that the projected benefits (both hard and soft) are still net positive, meaning the investment makes sense.
From my observations at least it doesn't look like the idea of development as "investment" in a product or service is very common. I'm assuming because time to market is often the ultimate driver rather than cost, in which case, congratulations, you should increase your costs on more numerous and productive resources whilst aligning your strategy and risks to iterate on smaller scopes faster so that dead ends can be quickly parked.
The problem isn't so much the estimation process as it is more generally poor portfolio/programme governance and management practices and more specifically a lack of risk management and understanding of contingency at those levels. I find IT, and Software Development more particularly, to be some of the worst offenders, but that is because the risk profiles of such projects are vastly different to the risk profiles of other types of work. The productivity of your resources is difficult to discern and a lack of precedent for similar-enough projects and knowing what their variables were, all meaning you really can't produce a reliable estimate with incomplete information.
I've always had strangely accurate time estimates. The basic idea is to divide up the work, and make a fair estimate each part, with special attention to unknowns (which get much more time).
Then double everything.
The individual task estimates are often out, but the overall estimate is close... as if, with a population of tasks, there's regression to an accurately estimated mean.
(An alternative explanation is that I over-estimate, and Parkinsonianly, work expands to fill time.)
Or you've baked in the appropriate level of risk for the work at hand.
You might want to look into Reference Class Forecasting. Essentially an "outside view" of past performance which is added to provide an uplift adjustment to some measure of your estimate (time, cost, benefit).
Great article. When I worked at $OldJob, the leadership wanted an unsolved, research-grade problem solved in a few months. They demanded estimates, then refused to accept estimates beyond their timeline, then tried to hold people "accountable" for missing the estimates. Of course it was a mess, and of course we failed to hit our goals, and of course many many people burned out.
But I noticed that some people handled the situation fine. They stayed on management's good side, even though they were failing to deliver along with the rest of us. I will try to distill what I observed them do:
1) They did not fight on the estimates. If a manager forced them into a certain timeline, they registered their disagreement and just accepted the new timeline. I think they realized that fighting would just make the manager judge them less capable. (Pick your battles, eh?).
2) When the schedule slipped, they would communicate it in a way that made them seem more competent instead of less. The explanation usually had three parts: unforeseen events kept us from hitting the deadline, we accomplished some great things in the meantime, and here's why we're in great shape to hit the next deadline! For example: "When we made this schedule, we did not realize that AWS nodes were so unreliable! Despite this, our team has made incredible progress on implementing a fast method of storage and solid compression! We have reworked the schedule to reflect this new information, and we are already on track for the first milestone!"
It's possible that this trick just worked for the specific situation at $OldJob, but I really enjoyed learning it. They seemed to understand that certain explicit rules were not important, but that other unspoken rules needed to be followed. Are accurate estimates important? It depends on the situation! Sometimes wrong estimates can be more valuable than correct ones. Construction companies give wrong estimates all the time in order to win projects. Is staying on good terms with your boss important? Yes! Even if they are total shitbags, being adversarial won't help, only leaving will. These people demonstrated that there are often ways to fix a bad situation by breaking some explicit rules and carefully following implicit ones, and I wish I had the acuity to see these possibilities on my own.
Stakeholder management is an important but often forgotten art. Keep them up to date on what they can expect. Don't fight them, just inform them. If a deadline is unrealistic, don't fight it, just let them know you cannot commit to that deadline with any certainty. Once it's clear you're not going to meet that deadline, let them know in advance.
> there are often ways to fix a bad situation by breaking some explicit rules and carefully following implicit ones, and I wish I had the acuity to see these possibilities on my own.
Now extrapolate that to working in BigCo, which has tens of thousands of employees worldwide, with each country having its own unique unspoken rules and hidden undercurrents. The greatest lesson I learnt is the importance of giving people the benefit of the doubt unless they've really proven they are a bad actor, because over and over again problems turn out to be cultural at base.
I think your two points are really important soft skills or work politics skills that apply transferably to any job.
Your points illustrate:
- Communicating your concerns clearly and in a timely manner,
- while also committing to do the work as it has been agreed to the best of your ability,
- and also communicating your progress clearly and regularly.
> 2) When the schedule slipped, they would communicate it in a way that made them seem more competent instead of less.
In other words: Enable the bad estimates, then externalize accountability when they can't be achieved. In the past, I called these people "blameless go-getters" because they were always first to volunteer to take on work, but somehow it became everyone else's fault when they failed. If management is asleep at the wheel, it's a win-win situation for them.
You're exactly right that this method works in many companies. Like you said, it's all about understanding what the company actually values. When unrealistic schedules are forced on teams, the exact date might not be important. Instead, it might be leadership's dysfunctional way of emphasizing focus, urgency, and quick iteration. The savvy engineers and managers know how to put on a show that hits these key points while steadily delivering progress in the background.
Still, there's no escaping the fact that this is dysfunctional. More importantly, it doesn't have to be like this. It's eye-opening to move from a dysfunctional company like you described toward a company that values honest communication and understands engineering project management at the executive leadership level. When hiring someone out of a dysfunctional company like you described, it can take some time to break them of bad habits around schedule misdirection and estimation dishonesty.
If you look at the examples, they are clearly on the format "get the unavoidable problem -> understand the root cause of the problem -> fix the root cause -> tell your manager what you did".
Those people are basically doing their managers' jobs, and telling them that. If the managers aren't technical, they would be completely unable to do it, and are probably very afraid of somebody finding out, having somebody doing it will bring a lot of confidence.
> ... there's no escaping the fact that this is dysfunctional.
It may be "dysfunctional" but it is how things get done in many places.
Even in places where people pretend to follow a strict "Agile" methodology, there's often a level of management that is brokering deadlines and promises for the "completion" of the whole damn thing.
This works because no-one is demanding estimates because they want to know the answer, they are doing it to apply pressure to get it done. The project manager is being subject to the same thing, and if her minions are savvy enough to provide a story that she can take to her management, and if she's smart enough to realize that's the most she can realistically hope for, a sort of equilibrium is achieved.
- CYA above all. The chances of a significant project succeeding in such an environment are pretty slim. Prepare your umbrella from the get go for when the inevitably will hit the fan. Some heads will have to roll, and you will not be easy pickings as you have been prepping for this since day 1.
- Greenshift like there is no tomorrow. Your manager is going to anyways until 80% of the budget is spent and the last thing she wants is someone pooping on her parade. Always be positive, but make sure not to get caught on factual lies. Remember, you don't want to be the one easily thrown under the bus when she needs to find a scapegoat.
This strikes me as an understandable but pathological response to managerialism. Managerialism being our current dominant business philosophy.
To see it in contrast, think for a moment about a well-run hospital. At a hospital, the people who make the key decisions about cases are medical professionals, not managers. If a surgery was supposed to take 4 hours but takes 12, well, that's how long the surgery took. Everybody recognizes that the 4-hour number was an estimate, and estimates are not commitments. The most important thing is patient health, not manager feelings. Managers help organize the work, but they do not control the work.
I would love to see software development become a true profession, where stroking manager egos by making them always feel correct and in control is not the most important thing.
Same in the arts, until you get a Heaven's Gate or other big artist-driven films that flopped at the box office and ate up UA's budget and then company. The studio model is savagely more effective at survival.
Yeah, this. At my current company, which I would consider large (20k+ employees) and bureaucratic, we've actually gotten this part right. Managers in no way control what the team is working on or deadlines for that work. They behave more like career coaches and therapists. They DO ultimately have control over your career, and I feel like that's a part here where having a good manager matters. But it's very rare for them to step in and shape the teams sprints etc. This is all in my own experience, so it probably depends to some extent on the department you're in and your manager in particular, but I can say this is all true for every team in my department.
The managers are held accountable by their bosses, but usually what this boils down to is teams not doing what they said they would. Which of course happens from time to time because as the article mentions we're all pretty terrible at estimating and shit happens sometimes. It makes me nervous to think about changing jobs because it sounds like this is NOT the way it is in most other places...
> Managers in no way control what the team is working on or deadlines for that work. They behave more like career coaches and therapists. They DO ultimately have control over your career,
I work in a place like this. It's a horrible idea. It's politics 100% of the time for managers, because there's no other way to climb for them. Welcome to half-brained initiatives and goalpost technologies being championed rather than ROI exploration and derivation, because the managers that do get stuck on projects that cost money don't want to talk about that.
When I was a young programmer in the early nineties, an old IBM programmer told me 'Don't accept the invitation to fail.' Good advice but what if the goal posts are moved. Best then to keep positive and communicate early and often about how the changes will affect the schedule. Work together as a team and get your manager to help you and that'll go a long way rather than keeping 'obvious' things to yourself and catching management off guard when work slips.
Good article. I thought about this problem quite a bit. I've been on both sides of the question.
I thought I was brilliant when I came up with the solution of fixing the time frame and estimating the work that can be done in that time frame. Turns out I came up with sprints, 60 years after they were invented.
The fun solution for this would be to give 'hit dice' estimates for tasks. Assign type of dice and number of them to each eastimated.
How long will this take?
About 1d20 days.
Nobody will be happy with this, but it is the most realistic one. Cause tasks do have that variability to them.
The wises thing said here is:
"the reality is that if you can make a probabilistically accurate estimate, then it's likely that the task should have been automated by some other means already. "
Is there an answer to this problem? Maybe abandon long term estimates entirely. Having really short term estimates, with frequent updates.
252 comments
[ 4.3 ms ] story [ 267 ms ] thread/s
My initial proposal was to build it incrementally: tackle the obviously critical functionality to get something working and ship it, then define and implement additional features on an ongoing basis. I had hoped this would fly, since the company talked a big game about being agile.
But no dice. We had to have a complete plan for all the features currently supported (some of them of dubious value, many of them incomplete and inconsistent in specification), and it had to come with a specific timetable. Under protest, the team and I worked hard to produce high-level estimates, and ended up basically guessing that the thing would take six months.
No dice, it had to be delivered in three. I made a suggestion along the lines of the strategy suggested in the article -- we could identify a subset of features that the team would be comfortable could be delivered within the deadline. It would probably be stronger, I argued, since it wasn't clear that all the added weight added much value.
No dice: everything was priority one. I pointed out something like, "you can't fight the laws of physics". I apparently gave the impression that we'd get it done anyway, though my memory of the conversation was that I was sternly disagreeable.
Our team goes ahead and starts implementing items from a value-prioritized backlog. Fast forward three months, and we have a working system that supports the most important use-cases. We considered it past ready to ship, knowing we'd need to keep iterating. The response from management, predictably, was frustration at the missing features, despite the advance warnings.
Management goes into "high-pressure" mode, and for the next three months I do my best to keep the team insulated. After more or less six months of total development time, we finally replace the prior component. All the users agree that it has far fewer bugs. I and many of my team members grumble that it has far too many, on account of the fact that we weren't given the chance to ship the minimum viable product when it was ready.
I'm not really sure what the moral of the story is.
To me, it's - never waiver from your experience/gut estimates and a no is a no.
I've seen plenty of (experienced) businesses and individuals that refuse to work for toxic clients, because there is plenty of other work that pays exactly the same but without all the shit.
P.S.: According to your story, you sound like a good manager :).
Well done for at least attempting to introduce some flexibility. One lesson for the future might be: how to more forcefully stand up for what you believe is the right way to go - but of course this is easier said than done!
Software has an infinite quantity of numbers to use as tools to build an end product. A programmer's job is to decide what general ideas end up as what specific quantities. A manager cannot decide this, lest he do the programming job himself.
Money is a great tool for measuring quantities of material goods, or the value of material properties. But electricity provides infinite quantities, exchanging money for programmer time is closer to exchanging two forms of currency (binary numbers and dollars) for one another. Code is a constantly transforming river of numbers that we make a draw-down into a bucket that we call an end product. That is not a quantity of material good like iron ore, at all.
The factory model simply doesn't work and producing code should be treated closer to the stock market than the factory model. Early access/kickstarter/pay-over-time business models in video games represent an example of the constant transformation model working successfully. Business apps should be built by crazy people who will build them anyway and money holders should bet & invest in them like stocks that raise and lower value over time, rather than as promised end goods.
I've had ex-programmer managers say "I just want you to add this graph to the app. I could do it in 20 lines of python." In reality, the request is more like "Add in this plot which is the result of a long running calculation. The calculation has to run in a background thread so the app is still responsive, even though the whole program has been architected with a single-threaded design. It needs some mechanism in the UI to indicate it is making progress. We'll also want a way to cancel the task. Half the datasets are using a different sign convention, so you'll need to automatically handle that. Actually, the data is polluted with garbage, you'll need to spend an unknown amount of time debugging a legacy system to figure out where the bad inputs are coming from in order to understand what can be done to filter that out...so we can have this plot by end of day, right?"
Apologies for ranting about my "internal software" days.
“We” are usually not at all clueless. “They” tend to be.
It's a linkbait trope to use "you" in a title, because it grabs attention whether the topic has anything to do with the reader or not. That's why "you" is headline writers' favorite pronoun. Combining it with a pejorative ("you have no idea") makes it even more sensational. That's definitely the sort of title we edit.
When we edit a title, we look for a representative phrase in the article itself that expresses its point in a more neutral way. That's what a moderator did in this case. The language comes from the article's own summary of itself: "After all of these years, I finally came to one simple conclusion. With all due respect: we are completely clueless about how long things should take." Reading the article text to find how it states its own conclusion, removing any residual linkbait (such as the superlative "completely"), and making that the title instead is the best way we've found to correct titles that break the site guidelines.
> It is our belief that over the 30 plus years of commercial computing has developed a series of sophisticated political games that have become a replacement for estimation as a formal process.
Unrelated: can we not do this thing with the whole left side of the screen being one static image? It's really distracting.
Agreed, ive been meaning to rework this
I've been mentally moving away from being paid to write software. I can see writing it for my own use, even professional use---I just don't want to be on the hook for ever-changing requirements, decided by people who are often kind, but not, in the end, competent. The best managers understand this and will give you leeway, but this is not a sustainable, repeatable thing. It lives and dies on one relationship.
I wonder sometimes: what if we just all stopped writing software one day, and started just using it? Writing software is a bad deal in a lot of ways---hard, socially isolating, etc---while using it is amazing---the computer does the work for you!
Don't get me wrong, I spent an hour at work today presenting on Lisp macros and loved every minute of it. But a dev career, for many, means a capped income and a razor's edge of apparent competence.
What's the difference, though? It's insane how much of my work the computer does when I write code these days.
This is one of the reasons you often hears about an entire team moving to a new company. They have a dynamic that works, and they do not want to risk it.
1. Doctor: In many cases, throw away your life and work 60+ hours and also you need to specialize and study long and hard.
2. Laywer: I don't know enough about the profession. The job doesn't transfer well to other countries.
3. Consultant: 60+ hour days are the norm, interviews tend to be based on quick thinking in the high school arena. To pass the interview, you need to have excellent and super quick high school level knowledge of: math, logic, social and political skills. This sound denigrating but I found it tough to do this quick.
4. Investment banker: 100 hours
5. Construction worker: your body will thank you later (/sarcasm).
Programmers have a certain set of advantages and disadvantages (I agree with your disadvantages), but how is it worse than other white collar jobs?
Go on a reasoning chain with me:
- what solves the problem nicely is to sell software. Selling good software can be one of the easiest and most lucrative jobs in the world. In practice no particular employee gets that---the company pays enough to motivate, but takes the rest. The solution, then, is to be the software company.
So...start a startup? "What a novel idea Dropit, on HN of all places!" I have actually "done this" (or thought I was doing it) multiple times (failed every time), but looking back I can see a lot of trivial mistakes I made. But at least I can find some---with many code-for-hire fiascos, the mistake was taking the job in the first place.
So my conclusion: accept the job you have, for now, while saving money and trying to have a good, normal life, and put some effort into seeking out new opportunities. FU money is a thing, as is FU market position.
1. Get a job for 32 hours per week 2. Work 10 hours on week days and 8 hours on one weekend day (take the other weekend day of), so that you clock 58 hours per week. 3. Don't take up your vacation, safe it. 4. Take the other 6 months off. 5. Oh, and pay less taxes. You're handing in 20% gross, but net you're only handing in 15%.
I think a schedule like this works for people like me, because I like to work hard and earn my freedom and then relax and doodle around for quite a while (i.e. 2 to 4 months) and then do a small side project and then work hard again.
Since I'm at the beginning of my career, it sounds like an interesting experiment.
Not for my situation, but for others: geo-arbitrage becomes interesting as you can literally fly to Thailand for 6 months and come back (a cheap retour ticket is found for around 400 euro's).
Dang, I thought 12 hour days were bad.
That can be quite sustainable; that one (or few) relationships can be long-lasting. If good managers leave and are replaced by bad managers, often quite soon team members will leave to join that manager in their new company - it's a well known fact that the direct manager matters more for job satisfaction than the particular company.
I know people who over the years have worked at 3-4 companies for the same boss (not continuously), and if I was unhappy at my current position, I remember a few previous managers whom I'd call to ask 'are you hiring?' and in the tech field the answer pretty much always is positive.
Another option might be to never do business with cash-starved companies. Founders and execs will be constantly worried about running out of money, acquiring customers, and pulling off miracles, and all that stress will trickle down on you probably for no real benefits.
A key question to ask an employer then is how much 'run way' / money they have left? Or whether they have a revenue source. It seems fairly probing, but realistically, not everyone wants to invest heavily in a company that might not even be around in 6 months.
Otherwise the engineer is relying on (possibly blind) trust that management won't try to "motivate" him/her into performing miracles (ie. breaking the project management triangle).
https://en.wikipedia.org/wiki/Project_management_triangle
Once you realize what's going on, you can turn that into a game.
Do you know the game where somebody asks you a bunch of questions, and they you allowed to answer with anything except "yes" or "no"? It's very hard to do, because it's such an ingrained habit.
If you really want to not give an estimate, make it a game to not give one.
But, give them something else instead. Work out a bunch of questions about uses cases / data volume / whatever, and say "if those were answered, we could build a prototype in a few days that would let us make a more reliable estimate" or something like that.
Another comment: coming up with good estimates is work. The other day somebody asked me if I could come up with a rough estimate for a (poorly specified, IMHO) project, and my answer was: no, I don't have time. If you need it anyway, formulate it as a task in Jira, so that it gets prioritized along with all my other work.
(Fun fact: we estimate our tasks in story points, so then we estimated how much time it would take us to come up with an estimate... :D )
None of the Celtic languages have an equivalent of yes and no. Just make a habit of speaking, say, Irish or Welsh at work. (To an English speaker, that sounds weird at first, but you get used to it quickly.)
Software engineers aren't being commoditized by being replaced by machines that write software. They're being commoditized by their own frameworks, libraries and tools.
Take the game industry as an example. Twenty years ago, your game company needed a big team of software engineers employed to write a game engine with advanced graphics capabilities (let's assume you want advanced graphics). Today, a single developer can just download Unity or Unreal Engine and have at least the technology available to them immediately (art is different but in many ways similar; automation and process improvements are coming for those jobs too, I'm sure).
So you don't need the same number of engineers for the same result. Sure, you have a big team of engineers employed at Unity Technologies or Epic Games, but that's now a shared resource. That employment is no longer duplicated at the companies that decide to use those engines.
Another example is the push for 'DevOps' and 'Cloud'. Think of all those system administrator jobs and IT departments being made smaller because now you can just spin up a server on AWS or have your CI infrastructure managed by BitBucket.
It's been my experience that delivering business value is taking longer because of this, not less. It's hubris to believe that a single person can be competent enough in all these domains to replace multiple people who focus on specific domains.
By distilling DBAs, Configuration Management Engineers, System Administrators and Software Developers into single people business is getting shittier products less frequently which incur more operational overhead.
On the other hand, I could also see a reality where, since many low-level problems are solved for you, management expects more out of you, so the number of engineers stays about the same, but you get a higher level of productivity.
I wouldn't put money on this balance lasting forever though. To me, that's too close to dismissively saying "it's different this time", with regards to our profession.
As in: My early product hit whatever trend / wave / need therefore I must be a product guy (and not just lucky).
The product guy has a preternatural ability to understand what the masses want. Watching them work -- witnessing their process -- is something to behold. They will steer products in a direction regardless of cost, complexity of likely outcome.
The outcome, quite often, is to tank their company. Since they don't understand why they were successful in the first place, it's very likely that their success won't last.
But if you're along for the ride, wow, expect the following:
1. You're the greatest (available) engineer we've ever encountered, building super-complicated XYZ is going to take this company to the next level!
2. This is taking much longer than expected and isn't matching up with our expectations but I'm 100% sure of my vision because I'm a product guy.
3. We're running out of money (because the market conditions that gave us early success have changed) and super-complicated XYZ isn't going to rescue us -- because you're a worthless piece of shit of an engineer!
See what happened there?
They're sometimes hard to distinguish from a vanilla bullshit artist. The bullshit artist will tell you how well capitalized he is, tell you he only wants the best (meaning he thinks you're expensive) and then try to slowly whittle your sense of self worth down until you get "the offer":
The offer is game-changing, life-altering for you: Instead of continuing to pay you with money, they're going to start paying you with magic pixie dust. The magic pixie dust will make you rich "when everything comes together."
When you tell bullshit artist that you don't work for magic pixie dust, that's when you learn that, in fact, you're a worthless piece of shit of an engineer.
I actually respect the bullshit artist more: They're bullshitting other people but they know they're full of shit. Product guys, depressingly, bullshit themselves.
99% of Wix users need a website that's just a big phone number and street address that gets indexed in Google. A "used sailboat" is exactly what they need.
If I tell sombody that a full custom site costs idk 20 times more than a Wix site they might not want to pay for that.
‘we just need a rough estimate, we won’t hold you to it’.
‘Ok based on the 2 minute conversation we just had I think about 3 months’
‘What! That seems way too long’
Other times I’ve been asked for estimates on features even though there is a hard deadline due to some external factor. I really fail to see the point of estimating anything when there is already a decided upon end date. Anyway I usually point out that they will need to just put the features in order of importance and I’ll work down the list. And they should start thinking about what can be cut. This usually leads to protests of ‘we have already cut everything we can’. But I have to laugh as we get closer to the deadline that suddenly not every feature is as important as was originally thought and magically get cut.
On the flipside though I do think a deadline is necessary for everybody involved, developers and managers alike. It really helps to limit feature creep. And it forces people to think about what they really want or need.
You've added the qualifier he never seems to add, that someday you might not be in the position to act professional. This is a sad but accurate commentary on the current state of things in our "profession".
Manager: ‘we just need a rough estimate, we won’t hold you to it’.
Me: ‘Ok based on the 2 minute conversation we just had I think about 3 months’
Manager: ‘Ok, I guess then we need to outsource it.’
Me: ‘Ok, then I need 3 months for guiding them through it, and 3 months for fixing their bugs.’
But serious question - for live products especially, you do need to have some sort of schedule where you are launching new features every X weeks. So it's important to know how long your features will take, so you can have a constant cadence of updates. Plus a lot of times you will have marketing initiatives or other things that need to be coordinated with your releases.
So my point is that you cannot just remove estimates. There is a need for knowing when the current sprint / feature will be completed, and being somewhat accurate about it.
I do really like the point about re-framing the conversation to start by asking the manager how long they want the engineering team to spend on the new feature. That will definitely change the dynamic and hopefully should encourage a conversation about what is realistic to do in the timeframe that the manager has in mind, and how the feature needs to change in order to achieve it.
But after that, the engineer still needs to go through and create estimates to make sure what they just agreed on is actually possible, and then those same estimates are necessary to plan out the development to make sure you are on track. So yeah, you can never really remove estimates.
Am I wrong?
The only way you can have new features every X weeks, is if those features take no more than X weeks to develop. You can estimate a new feature to take 2X to complete, but that would still mean it can't be released "in time".
Of course, you might still want to have a regular cadence of feature announcements, or at least be able to plan them in advance. But I feel like the best way to do that is to decouple finishing a feature from releasing it.
Estimates are primarily useful in deciding what tasks to pick up first. Luckily, that usually needn't be that exact - you don't need to know to the hour how long some development is going to take, just how much faster one thing will roughly be compared to the other. A manager can then decide whether it's worth it to risk picking up a larger task that might provide substantially more benefit than the smaller task.
You can still have releases every 2 weeks, where each feature takes 1-2 months, if you have multiple small teams each working on a different feature at the same time. That is how we do it.
But of course, it's typical that each feature takes an extra 1-2 weeks of development time and many times other devs are pulled off their own projects to help out, so then those other projects are even more delayed.
[0] https://www.amazon.co.uk/Making-Software-Really-Works-Believ...
It took 40 minutes to do that estimate.
But, TBF, several important decisions were made during the process of estimating, such as what should be excluded from the task, basic organization, and some research.
BTW the task being estimated was doing time estimates for a project (which came out to be 2-3 weeks).
The outcomes are so much more predictable / better quality.
My boss at my last job told me that he had observed that if he asked somebody “can you get this done by the end of they day”, he would get an accurate answer (either yes or no). Any further out, there was no correlation between what they said and what they actually delivered.
> But the reality is that if you can make a probabalistically accurate estimate, then its likely that the task should have been automated by some other means already. In other words, its easy to estimate a task that essentially amounts to copy and pasting some well known CRUD API end point patterns, but any even remotely creative or novel work is almost guaranteed to be totally unknown.
Well, he sort of goes back and forth, but he includes this:
> The engineer comes back with this simplified description and says he can get a first version produced, but it will take a month instead of 2 weeks.
Which I see some variant of every time I see somebody rail against the unrealistic expectations of software estimation (which, by the way, I’ve been seeing people rail against since the late 80’s to no avail). The implication here is that if the manager had just listened to the developer and accepted his initial estimate of one month, the software would have been done in one month: the developer could estimate with precision, but the manager bumbled along and screwed everything up by trying to negotiate it down.
This is a dangerous position to take unless you’re absolutely sure about your one-month estimate: if you say one month, he says two weeks and you look him in the eye like the alpha wolf say, “no, one month, and no sooner”… you had damned well better be able to deliver in exactly one month. The reality is there’s probably no way to tell, _especially_ if other people are involved, so you’re better off shrugging your shoulders and saying, “yeah, sure, two weeks”, doing as much as you can, and preparing your story ahead of time.
Too often the request is along the lines of "How long does it take to build a house?" not "How long will it take to build the house in these blueprints with these systems on top of a mountain that is also impervious to mudslides?". People can generally estimate the former but the latter is all about the unknown details.
Caveat: I work in the financial industry, where the complexity of writing a compiler and whipping up a Tableau dashboard are perceived to be equal.
In fact hearing this line, no one will work for you. Frame it and make it your motto, see what happens.
It is also more difficult to initially assess the skills fit of candidates for knowledge based work, especially those that require creative problem solving, and unlike other engineering disciplines past outputs are opaque and hard to rely on as simple markers of past performance.
For a project, in order to produce a good estimate you need to understand scope, then align it with precedent, adjust for your resources and productivity profile and then view all of that through a risk lens to set probable outcome ranges.
For a programme, in order to produce a good estimate you need to understand and manage the risks, constraints and dependencies across all your projects and ensure that the projected benefits (both hard and soft) are still net positive, meaning the investment makes sense.
From my observations at least it doesn't look like the idea of development as "investment" in a product or service is very common. I'm assuming because time to market is often the ultimate driver rather than cost, in which case, congratulations, you should increase your costs on more numerous and productive resources whilst aligning your strategy and risks to iterate on smaller scopes faster so that dead ends can be quickly parked.
The problem isn't so much the estimation process as it is more generally poor portfolio/programme governance and management practices and more specifically a lack of risk management and understanding of contingency at those levels. I find IT, and Software Development more particularly, to be some of the worst offenders, but that is because the risk profiles of such projects are vastly different to the risk profiles of other types of work. The productivity of your resources is difficult to discern and a lack of precedent for similar-enough projects and knowing what their variables were, all meaning you really can't produce a reliable estimate with incomplete information.
Then double everything.
The individual task estimates are often out, but the overall estimate is close... as if, with a population of tasks, there's regression to an accurately estimated mean.
(An alternative explanation is that I over-estimate, and Parkinsonianly, work expands to fill time.)
But I noticed that some people handled the situation fine. They stayed on management's good side, even though they were failing to deliver along with the rest of us. I will try to distill what I observed them do:
1) They did not fight on the estimates. If a manager forced them into a certain timeline, they registered their disagreement and just accepted the new timeline. I think they realized that fighting would just make the manager judge them less capable. (Pick your battles, eh?).
2) When the schedule slipped, they would communicate it in a way that made them seem more competent instead of less. The explanation usually had three parts: unforeseen events kept us from hitting the deadline, we accomplished some great things in the meantime, and here's why we're in great shape to hit the next deadline! For example: "When we made this schedule, we did not realize that AWS nodes were so unreliable! Despite this, our team has made incredible progress on implementing a fast method of storage and solid compression! We have reworked the schedule to reflect this new information, and we are already on track for the first milestone!"
It's possible that this trick just worked for the specific situation at $OldJob, but I really enjoyed learning it. They seemed to understand that certain explicit rules were not important, but that other unspoken rules needed to be followed. Are accurate estimates important? It depends on the situation! Sometimes wrong estimates can be more valuable than correct ones. Construction companies give wrong estimates all the time in order to win projects. Is staying on good terms with your boss important? Yes! Even if they are total shitbags, being adversarial won't help, only leaving will. These people demonstrated that there are often ways to fix a bad situation by breaking some explicit rules and carefully following implicit ones, and I wish I had the acuity to see these possibilities on my own.
Now extrapolate that to working in BigCo, which has tens of thousands of employees worldwide, with each country having its own unique unspoken rules and hidden undercurrents. The greatest lesson I learnt is the importance of giving people the benefit of the doubt unless they've really proven they are a bad actor, because over and over again problems turn out to be cultural at base.
Your points illustrate: - Communicating your concerns clearly and in a timely manner, - while also committing to do the work as it has been agreed to the best of your ability, - and also communicating your progress clearly and regularly.
This ought to be valued by any employer.
> 2) When the schedule slipped, they would communicate it in a way that made them seem more competent instead of less.
In other words: Enable the bad estimates, then externalize accountability when they can't be achieved. In the past, I called these people "blameless go-getters" because they were always first to volunteer to take on work, but somehow it became everyone else's fault when they failed. If management is asleep at the wheel, it's a win-win situation for them.
You're exactly right that this method works in many companies. Like you said, it's all about understanding what the company actually values. When unrealistic schedules are forced on teams, the exact date might not be important. Instead, it might be leadership's dysfunctional way of emphasizing focus, urgency, and quick iteration. The savvy engineers and managers know how to put on a show that hits these key points while steadily delivering progress in the background.
Still, there's no escaping the fact that this is dysfunctional. More importantly, it doesn't have to be like this. It's eye-opening to move from a dysfunctional company like you described toward a company that values honest communication and understands engineering project management at the executive leadership level. When hiring someone out of a dysfunctional company like you described, it can take some time to break them of bad habits around schedule misdirection and estimation dishonesty.
Those people are basically doing their managers' jobs, and telling them that. If the managers aren't technical, they would be completely unable to do it, and are probably very afraid of somebody finding out, having somebody doing it will bring a lot of confidence.
Even in places where people pretend to follow a strict "Agile" methodology, there's often a level of management that is brokering deadlines and promises for the "completion" of the whole damn thing.
- CYA above all. The chances of a significant project succeeding in such an environment are pretty slim. Prepare your umbrella from the get go for when the inevitably will hit the fan. Some heads will have to roll, and you will not be easy pickings as you have been prepping for this since day 1.
- Greenshift like there is no tomorrow. Your manager is going to anyways until 80% of the budget is spent and the last thing she wants is someone pooping on her parade. Always be positive, but make sure not to get caught on factual lies. Remember, you don't want to be the one easily thrown under the bus when she needs to find a scapegoat.
I chose not to stay in such environments.
To see it in contrast, think for a moment about a well-run hospital. At a hospital, the people who make the key decisions about cases are medical professionals, not managers. If a surgery was supposed to take 4 hours but takes 12, well, that's how long the surgery took. Everybody recognizes that the 4-hour number was an estimate, and estimates are not commitments. The most important thing is patient health, not manager feelings. Managers help organize the work, but they do not control the work.
I would love to see software development become a true profession, where stroking manager egos by making them always feel correct and in control is not the most important thing.
The managers are held accountable by their bosses, but usually what this boils down to is teams not doing what they said they would. Which of course happens from time to time because as the article mentions we're all pretty terrible at estimating and shit happens sometimes. It makes me nervous to think about changing jobs because it sounds like this is NOT the way it is in most other places...
I work in a place like this. It's a horrible idea. It's politics 100% of the time for managers, because there's no other way to climb for them. Welcome to half-brained initiatives and goalpost technologies being championed rather than ROI exploration and derivation, because the managers that do get stuck on projects that cost money don't want to talk about that.
Task X will take 2 weeks.
Task X will take 2 weeks of development time and 6 weeks to test, validate, productionize and roll out.
Task X would take 2 weeks if we drop everything else we're doing, but because we can't, it will take about 12-18 weeks to finish.
I thought I was brilliant when I came up with the solution of fixing the time frame and estimating the work that can be done in that time frame. Turns out I came up with sprints, 60 years after they were invented.
The fun solution for this would be to give 'hit dice' estimates for tasks. Assign type of dice and number of them to each eastimated.
How long will this take?
About 1d20 days.
Nobody will be happy with this, but it is the most realistic one. Cause tasks do have that variability to them.
The wises thing said here is: "the reality is that if you can make a probabilistically accurate estimate, then it's likely that the task should have been automated by some other means already. "
Is there an answer to this problem? Maybe abandon long term estimates entirely. Having really short term estimates, with frequent updates.