I've often thought that what software projects need is some equivalent of a balance sheet and P&L report - "assets" and "liabilities" together with "income" and "expenditure" equivalents.
Looking at technical debt in isolation and always regarding it as a bad thing always seemed simplistic to me.
Agreed. I've seen teams use no metrics. And I've seen teams use one or two metrics. I haven't seen management deal with complexity like a professional analyst for a pro sports club would: with a range of different metrics to inform human judgements.
That's because it's just a way for tech people to sell work, that shows no immediate benefit to the business, to accountants, who sign the checks, that don't like debt.
How do you measure complexity with an objective number? I have a few ideas for what could go into it for a given task or process (number of branching paths, number of methods called, LOC, time to develop, etc.) but how do you assign weights to these?
To start with, you don't. Baseball managers and scouts look at a lot of metrics: runs, hits, walks, OBP, VAR, number of pitches in a career, convicted felonies, etc. But at the end of the day, there are people making judgement calls. They're just, ideally, exceptionally well-informed people.
So what I'd like to see, at least initially, is a healthier universe of metrics we can automatically grab from code bases. Deciders can set up dashboards or reports for those as needed. There would still be actual conversations and decisions involving actual humans, but they'd be looking at this data as they're having qualitative discussions about their options.
Sure, and credit cards can be a good thing, too. The problem is that most non-technical and formerly-technical project managers just charge everything to their credit cards and don't even know they own debt and are making interest payments on it.
When that's the level of awareness the average development team is dealing with, there's limited practical purpose in exploring the nuances of healthy debt management. It's more useful to talk about technical debt like nutritionists talk about empty calories and transfats.
I like your comparison to nutrition. To stretch the analogy slightly, sometimes in life you only have time for that microwave meal rather than a freshly prepared salad. It's about ingestion of calories to keep going, not the quality of those.
The same _can be_ true with code. There can be a better way to do something but if you cannot afford the time to develop and iterate this idea then it can be advantageous to develop the simplest solution that meets your needs today and make progress.
I see accusations of this a lot, and honestly I just really wonder what the team structures are like at a lot of companies- or what the experience level is of the average HN reader. Because you're right, PMs don't have the insight to understand the nuances of technical debt and the benefits and costs associated with varied approaches, and really it's not their job to. As technical folks, it's our job to learn how to say "no", and not just in an abrasive way that shuts down the conversation, but to work with project managers and decision makers, understand the reasons for their decisions, and push back firmly but using language that they can relate to and speaking to their problems.
Regardless of how much PMs might want things or fiddle with the schedule, as professionals it's our job- not theirs- to say when things can't be done, shouldn't be attempted, or might cost more than they're worth.
> As technical folks, it's our job to learn how to say "no", and not just in an abrasive way that shuts down the conversation, but to work with project managers and decision makers, understand the reasons for their decisions, and push back firmly but using language that they can relate to and speaking to their problems.
That gets back to company culture (which others have identified higher up). Not every organisation responds appropriately when developers push back (either due to imposed deadlines or office politics).
> Not every organisation responds appropriately when developers push back...
In my experience, organizations generally do the right thing if all the developers push back. The problem is that you can always find developers that will push a hack or deploy an overwrought solution for a simple problem.
And then after saying no, you can watch the executives hire or promote up someone who says what they want to hear regardless of its feasibility, usually while using the latest buzzwords. Lacking domain knowledge or the interest to acquire it is a good sign that they will lack the awareness to avoid facile answers.
Saying "No" or shutting that door down is a bad thing that I don't like either. Rephrase it by using "sure, we could build that, however it will require X person-days with a team of Y senior level engineers."
Don't do that. All they're going to hear is "Sure, we could do that." They're not going to listen to the rest of it, and then it's going to be your fault when stuff doesn't work.
That's the point. Almost anything is possible with the right team. Once it's communicated in terms of cost then the assessment is made whether it's worth it or not.
This. As a PM, this is great for communicating tradeoffs to clients and senior management. As a dev, this is great for communicating tradeoffs to PMs. And conservative estimates of development time (e.g., the dev doubles the time estimate) result in solid, well-designed, thoroughly tested products.
And just like there are plenty of crappy software developers, there are plenty of crappy PMs who choose not to listen to developers, and ignore all of that. Recall the article about the toxic culture over at Take Two from yesterday. No matter how much push back happened, they would be met with "Tough. I guess you live here for the next 6 months."
> I just really wonder what the team structures are like...
Maybe you're lucky or maybe I'm benighted, but it's rare that an organization consciously decides that this is the quarter to finish off the Java 5 migrations, even if that means delaying the Jabberwocky Project and even if there's a non-zero chance of customer impact.
Partly because it's a really hard problem to attach an accurate and objective number to the value of something like that.
> ...technical debt and the benefits and costs associated with varied approaches...
That's not entirely accurate. Engineers are experts at technical benefits of different approaches. And they should be experts on the entire cost of the systems they are responsible for. But they're not necessarily experts on the benefits. They aren't necessarily qualified to anticipate what different features and different release schedules will do to the revenue for the product. In B2C companies with millions of users, you can run statistics and A/B testing to make this stuff more objective, but a lot of companies live and die by big sales to a smaller number of customers. So determining the benefits of shipping a month early depends a lot on knowing particular people well enough to guess how they'll react to good or bad news.
YES! As a developer I can FEEL the pain of our code debt but I don't know how to convey the importance to management. There's definitely a balance to it and I'm certainly biased in favor of letting those in the code make the judgement call.
One solution I thought of was to enact a 20% code debt policy, similar to google's 20% time but for code debt. Hey developer, is there something causing you pain? Fix it!
Ya I usually "sneak in" cleanups associated with stories. The problem with that is it doesn't allow for bigger more impactful refactors. So usually the debt that hurts the most never gets dealt with.
This approach tends to work well. If they don't want to worry about it, they've effectively delegated it to you.
The only problem is that this setup is subject to the tragedy of the commons. If it takes 25% more work to make a change cleanly, the developers throwing hacks around look 20% more productive if we ignore technical debt. And, worse, they're often the only people in the world that understand the hacks, so they get to be the hero-experts that come in and save the day when everyone else is trying to figure out what the hell is going on.
So even if we'll do our best to bake the cost of cleanup into our day-to-day tasks, that only works as long as a critical mass of the developers in the organization do the same thing. Otherwise, you'll have a full time job plus some just cleaning up messes.
You have to trust that doing good work makes things better and that you are actually faster over time than others who create a mess. If you don't get any benefits from clean code then it's not worth it.
This works only if you own your part of the code for a some time so your investment can pay off.
In the scenario you describe the only thing you can do is pad your estimates. Don't do this explicitly but if there is some objection to cost make the assertion that the usecase though straightforward has to contend with certain legacy issues.
There are two outcomes (1) they trust you, and grudgingly accept your estimates. You address what debt you can and ensure padding doesn't increase the next time thus trust is maintained (2) they force a timeline on you, in which case when it's late or broken in production you can at least say you needed more time.
There is a third more egregious possibility, that they get an alternative estimate from elsewhere or one of your colleagues and in my experience this is the principle weakness to standing your ground "we'll take our business elsewhere" and I am truly sorry for your trouble if that is the case. Hopefully you have secure employment and the whole mess then just becomes the problem of the person that undercut you.
Of course, in all these situations you have to make a call based on your personal/organisational circumstances but I just want to say that ultimately being upfront, diligent and continuously demonstrating trustworthiness may not always pay off in each particular instance, but over the longer term as reputation accrues they do.
In the end it's up to the devs to make it clear dealing with debt is just part of being a professional dev. In my team I never tell management about refactoring. We just do it as part of the daily job. You won't get major rewrites done that way but over time you can clean up a lot of problems if you are a little strategic.
You're stranded in a foreign place and have no cash in the local currency. There's no ATM around, but there are businesses that take credit and you can pay the bill for it at the end of the month.
Something unexpected pops up like a family member dies and you need to arrange a funeral, but don't have the cash on hand because some other emergency last month used up your liquid savings. You know that you can pay it off over the long run, but don't have any other way to deal with it right now. So you go into a small amount of debt that you can likely pay off without paying too much over the original cost.
There are a million ways that they can be good. There are probably more ways they can be bad, but this question is very simplistic.
The title of this article matches the tone, but doesn't really match the content. The author makes several good points that would be better characterized as, "given that technical debt is inevitable, try to make simple, easy to manipulate tools."
Even going back to my teen years I've always tried to avoid ending up in a bubble of agreement. It's become even more important of a skill in the age of the internet.
Whenever I'm told that I'm not allowed to disagree for some reason or another, I immediately stop and reevaluate my views. Usually they don't change, but it's worth it for the times that they do.
And often times when I've changed my views like that, there's a cultural shift a little while later. A lot of what is held as sacred truth is due to fear of the "opposing idea" being right.
Think about programming in the late 90s and early 2000s, but not believing OO to be a panacea. I use OO when it's appropriate, but it's often not. And speaking up about that in those times meant you were viewed as ignorant.
If you program in Java in the year 2000, you have a vested interest in OO adoption. You've invested your time in a language that was explicitly designed to not allow for anything but OO. And if OO is not the end-all be-all of programming, then maybe you've wasted your time. So anyone claiming otherwise is indirectly threatening your livelihood. You're terrified that they might be right, so you insist that they never could be.
It is clickbait. Just enough for you to click, read and comment. If the article actually relayed the content - How to manage technical debt better you might not even open the link.
This article has a clickbait title with a more reasonable premise in the body. No, technical debt isn't good, just like getting jabbed with a sharp object in the throat isn't good for you. The reasonable point the article is making is that there are some things worse than accruing technical debt, like spending too much time making an ivory tower system that doesn't actually have product fit. Similarly, getting jabbed with a sharp object in the throat in the case of a tracheotomy can be necessary, but that doesn't mean that you should run around jabbing yourself in the throat just for the sake of it. Technical debt is similar; sometimes necessary, but never choose it for its own sake.
More likely that contrarian headlines garner more activity than their obvious or expected counterparts, pushing them to higher rankings more frequently. Here on hn, it’s possible that if a user reads a headline they already agree with, they will say to themselves “oh ya, I know that already, don’t need to read that.” Then when they see a contrarian headlibe they say to themselves “hmm, what’s this.” I would posit something more along those lines. Although I now realize, your comment may have been rhetorical in the sense of suggesting that this post (and others like it) take advantage of this trend but lack in substance.
It is a strange phenomenon, that the brain tends to ignore things that are expected, and is often jarred by the unexpected. It’s maybe one of the downfalls of expecting the unexpected. Because it makes even the unexpected seem ordinary. Or maybe the reverse is true, and expecting the unexpected makes the ordinary seem extraordinary.
> Although I now realize, your comment may have been rhetorical in the sense of suggesting that this post (and others like it) take advantage of this trend but lack in substance.
Yeah this is what I was getting at. Ive been noticing that people have been dissatisfied with the quality of content recently citing marketing type activities like this. Kind of wish it could just be a forum without the competitive drive that we are bombarded with everywhere else.
I feel it’s easy to become a compulsive contrarian on Hackernews because of how many people are often willing to support or defend your contrarian view.
From my recent experience in a new team, this article strikes a chord.
Joining a team that is scared of technical debt and runs circles round themselves in a bid to avoid it end up producing more debt than they would otherwise. For example, making code overly DRY and wrong abstractions coupling unrelated parts of the system unnecessarily.
We must embrace the reality of technical debt. It is unavoidable and not something to be scared of. It can be managed by keeping things simple, clearly tested and keeping a cool head.
> making code overly DRY and wrong abstractions coupling unrelated parts of the system unnecessarily
I would say that's just bad engineering due to lack of insight or experience. DRY is the most simple and most dangerous principle (or "best-practice") of them all, because when it goes wrong it's the one that hurts the most.
Yes I agree. Bad engineering led them to not accepting the reality that technical debt is unavoidable. In a misguided attempt to avoid it, they overengineered ironically leading to more debt than might otherwise have accrued.
One mentor of mine taught me the "rule of three". The first time you have to do something, you write it in-place. The second time, same thing. The third time is when you refactor into a common module all three components can use.
When you're writing it the second time, you're accruing technical debt. You pay it off when you refactor for the third component.
Managing technical debt is one of them most important parts of engineering. When you're resource constrained, which is almost a given in the real world, you need to think carefully about what kinds of debts are worth accruing, and what kinds are absolutely not.
I think this is both an art and a science, and the right balance comes with not just engineering experience, but also domain experience, team maturity/talent, market/competitive pressures, leadership, and culture.
There are so many factors involved that it's not surprising so many projects accrue horrible debt.
> the right balance comes with not just engineering experience, but also domain experience, team maturity/talent, market/competitive pressures, leadership, and culture.
I couldn't have stated it better myself. Most of the time it start from culture, leadership and then experience etc.
I have worked on multiple projects with varied amount of technical debt. But the project with the most amount of debt happened because of poor leadership.
The guy who was managing the project was more interested in currying favor. The end result - everyone's salary was being shared across the company in un-encrypted plain text format.
A manager I worked with previously was like that--no interest in making sure the project could be maintained or actually worked, only promising things to people who didn't even use our platform (me and a coworker checked our logs before I left the company) and getting "wins" so he could talk about them in meetings.
That's organisational debt, which is a prime cause of technical debt.
It reliably fascinates me that in spite of more than a century of history, management culture lacks both words and concepts for many common and predictable organisational failure modes.
> It reliably fascinates me that in spite of more than a century of history, management culture lacks both words and concepts for many common and predictable organisational failure modes.
Does it? The manager in GP, for instance, seems to be a textbook example of a thermocline of truth.
the crux of Technical Debt is captured in these 2 points
> the right balance comes with not just engineering experience, but also domain experience, team maturity/talent, market/competitive pressures, leadership, and culture.
>the project with the most amount of debt happened because of poor leadership.
To add. It is hard to get funding for technical debt. Senior management are typically not aware (don't care) that a system is only working because some poor soul wakes up at night to clear caches. Sometimes I feel you might be better off letting the system have a spectacular crash than keeping system on life support. The trouble with letting system crash is may be a career limiting. Technical debt is complicated. Take companies that pay overtime to employees that get called at night to fix the system. They may not exactly be keen to have the calls stop.
Also, some debts are not even debts, but normal limitations of the code
If your system started out with tens of requests per second, and it was designed for that and it works fine under that load, having it break if the load increases 10x is not technical debt
It can depend; if it's because of sloppy programming (such as using string concatenation a lot instead of using a string builder, for example) then it's tech debt. If it's designed like that, then it's an architectural problem. I do think any naively implemented program should be able to scale up one order of magnitude, but two orders of magnitude require a new architecture.
Came to say something similar; I see it with library choices a lot as well; poor library choices ~= sloppy programming. I've had a lot of issues in the the past with 'conventional wisdom' lib choices that result in a lot of re-work as predictable issues with those choices arise.
Sometimes, choosing to use a library in the first place may also implicitly increase technical debt (in having to maintain this dependency, and even more so if this library shares dependencies with other dependencies that you have).
Sort of. You could have used better techniques/frameworks/whatever and gotten it right from the get-go. It does help to have the experience to intuit/know what you'll have to make scale and to what degree so that you can spend cycles on avoiding likely pitfalls without wasting too much effort.
I would be a very rich person if I got a dollar for every time the "right framework/solution/technique" broke unexpectedly when pushing it in certain ways.
Or it might not even have existed (or was unstable) when you started work, but it got better later.
I was thinking of things like: thread-per-client vs C10K. Just go C10K all the way. That means write services to be async, async, async. Just do it. It will not be the case that thread-per-client is better than C10K except when you know you'll only have a handful of clients. Write code to be thread-safe / reentrant. Write code to be separable into libraries.
I'd also urge you to use higher-level languages from the get-go. If you can use Rust instead of C, do it. If Haskell is appropriate, use it. These are all choices that are fantastically easier to make early on than later.
I think that's the point. You choose the best tech at the moment given the requirements with some headroom. Why waste engineering time to make sure it is hardened to support 10x load? Sometimes it is easy (use a different library) but most of the time it is hard.
Some changes can be extraordinarily difficult, so if you can foresee that you'll need, say, to make some code into a library, or thread-safe, or async, then it's best to make it that way from the get-go.
Obviously there's a bit of YAGNI in tension here.
But if you have the experience to write it correctly from the get-go at low additional cost, then it's worth doing that instead of risking a massive rewrite later.
I have never had any problem with putting some code into a library after the software is done. It is almost always a 1-4 hours exercise, and when it's not, it's a 2 days task at most.
Async code is also something that I have never seen it being a huge problem. Mostly because only a small part of your code needs to be async, most of it can stay synchronous, scheduled by the small async part.
Thread safety is a real problem. One can indeed trade some of it for some development speed, and this can become a huge bottleneck later/soon.
"10x" is large or small depending on the plausible variance you can expect. A free casual game? 10x spike is quite expected. a $1000/month B2B Saas? 10x comes with a lot of predictable lead time.
Software is significantly different from buildings in that it can be substantially upgraded or redesigned after going into production. That flexibility means that the first release need not be the last release. The obvious charitable interpretation is that the parent was talking about software.
> Software is significantly different from buildings in that it can be substantially upgraded or redesigned after going into production.
I used to think this was true, but after talking to architecture/civil engineering friends a lot about this, I've learned that this is less true than I believed. There are often changing requirements for buildings due to different use cases, changing regulations or building codes, or a change in energy prices (or the social cost of using a lot of energy). Sometimes new technologies are used without a lot of experience with them, leading to high operating expenses for the lifetime of the building.
I asked my friend about your comment and he gave me a few interesting examples: 1) "flying form construction" that leads to exposed concrete slabs on buildings that bleed heat in the winter. This was really cheap to build but turned out to be really awful for energy efficiency (once we started caring). 2) a building in the 50s-60s that was built with automatic window blinds. The maintenance cost of fixing the blinds when they inevitably broke led to them not getting fixed, and the building becoming basically useless.
If you're interested in reading more about this, Stewart Brand's "How Buildings Learn" is a really good read for laypeople. I was really surprised by how much of the book could be applied to software engineering.
(This is not a field I'm familiar with so any errors are mine, not my friend's.)
Buildings can definitely accumulate technical debt when they're put through a series of changes of use over time. It manifests as odd cable routing, inconsistent floor levels, repeatedly moved doorways, obstructed windows, etc.
I don't agree with this. As was pointed out here too, software changes in production all the time. This happens with Buildings too, but on a different time scale. Buildings that were made with forward layouts for HVAC and electrical systems will be easier to upgrade in the future. If you look at old buildings that had networking and such retrofitted in a visible and ugly fashion, I think its fair to say that you're looking at technical debt.
In the sense that the alternative is to demolish and build a new building, your example is technical debt. But your example is really an example of legacy systems.
Sometimes, supporting the legacy system can be technical debt. But sometimes when people try to replace the legacy system, they spend a lot of time and money and end up costing way more than simply maintaining the legacy system. This is part of why some nuclear plants will continue to operate on PDP-11's until the year 2050.
Technical debt isn't really debt if it's actually just a part of the cost of doing business. You design your system for its intended purpose, and if it's accomplishing that purpose well, you don't need a redesign. Don't break what doesn't need fixing.
What do you call the work that needed to be done in order to remove the existing electrical cables in my house and replace them with a wiring layout that was able to add a second outlet in my bedroom? (Or, far better - the unofficial extra wires that the contractor found which had been loosely spliced into the existing system by the previous owner in order to power the new bathroom lights he added, which just stopped working when we removed the old cable because nobody knew that connection existed).
Technical debt is kicking the can down the road. What you're talking about is mostly just a feature overhaul. The modifications the previous tenant made were technical debt.
That's a decent argument for bottom-up programming. Though I'd also say what's really at issue here is short term vs. long term technical debt. Writing software bottom-up front-loads some short term technical debt but leaves you much better able to avoid long-term technical debt because you have a much more flexible product.
There's a fine line between iterative development ("first do it, then do it right, then do it fast") and actual technical dept caused by bad practices (e.g. copy-and-edit instead of re-use). The former method is well-aware of shortcomings and heading towards eliminating them eventually, so I'm not sure if it should even be called technical debt. Actual technical debt (like real life debt) has the problem of "interest" attached to it, so it grows exponentially. So if you're not constantly working to reduce it, it may grow out of hand quickly.
Fantastic article. As an engineering manager working for a lot of young startups, I’ve found technical debt to be inevitable. The author seems to have come to the same conclusion I have: embrace technical debt by leveraging it conciously and tracking it carefully.
Technical debt is a debt that someone else will have to pay off.
My greatest accomplishment at my current job has been deleting 45K-60K lines of code.
Not every bit of code will be perfect for many reasons, but please be thoughtful at least, and not just hammer out code. Someone else is going to have to clean up your mess.
On the flip side, be thoughtful that your job exists because that code was hammered out.
I've created loads of technical debt over my career, in the process literally creating whole product lines that later became the career home for teams of people.
Funny thing is that technical debt is not like money debt. There is a lot of possibilities that this debt will just be deleted, obsoleted, project discontinued.
So it is more like survivorship bias, you only get to fix debt that is still alive. Debt that went away and no one is ever paying it is nowhere to be seen.
Technical debt can benefit from being managed in it's own category - it's inevitable as bugs and features. Unmanaged technical debt is sometimes the 'bad' technical debt we think and read about in hindsight.
When I'm a little more aware of technical debt being created - I track and manage it just like a feature or a bug in the project management system. Separate type, label or category of ticket. It surprising several tools don't have this out of the box for some reason.
There's a few benefits to technical Debt being categorized separately:
Tracking technical debt helps provide a sense of the technical debt tickets that are being opened, and floating around both in numbers, size and unknowns.
Keeping technical debt items tracked along side features, and bugs in it's own category is an invaluable form of cross-training to new team members who are learning the why the current code base came to be how it is, and not just what they may see or think is best.
We see how good we might be as a team and individually at identifying technical debt, and if we are improving at identifying it, and help develop a ratio for future accuracy.
Regular technical debt review helps create a sense of what technical debt is occurring, and what can be fixed under the hood on another project, if desired, or possible.
Keeping an eye on technical debt is important too, if something initially small is at risk of painting the project into a corner, or becoming exponentially difficult to re-factor.
Would love to hear strategies and tactics you might use in your TD practice.
even if its not timely to implement - design the replacement right away, and update it if necessary. this helps scope the work, understand what other components are being effected, and allows related systems to be written with an eye towards the fixed version instead of the damaged one.
We make the technical debt ticket anyways describing the problem, taking a few more minutes to describe "how it should be" so a design process can begin... it will be much easier to pick it up later on... maybe even tie it to an ongoing Technical Debt Roadmap that can be tied into the main product roadmap.
i forgot the most important part, this helps makes the decision of when to actually commit to making the change. instead of setting aside bandwidth to address debt just on principle, you can make a reasoned argument that says 'new feature y would fall out naturally if we reworked x, and is going to be a disaster if we don't, so maybe we probably should'
A lesson I learned early in my career was to avoid spinning out too much on why a particular piece of code someone else wrote was so “bad”.
The metaphor in this article fits perfectly- the developer who wrote this borrowed from the future, and without being there at the time, I can’t really judge their calculations.
Also, ugly code that solves a problem for your customers is worth more (ie. is more 'good') than pretty code which doesn't. Customers don't care what the back-end looks like, they just want their problem solved, and pretty code never increased revenue.
I'd say that technical debt is often the best kind of debt you can take on as a startup. Difficult to maintain code is not going to be the thing that puts you out of business. Usually that's making the wrong product or not being able to reach your market.
Compare that to the other kinds of debt (convertible, for instance). They cost a lot of time to obtain and, potentially, a lot of money in the long run if you do well.
Many of the most successful startups had (have?) shitty code. Of course there's great counterexamples (i.e. WhatsApp having a 5 person backend team when they were acquired), but as a programmer and founder I find that my urge to make shitty code and ship fast is waay smaller than my urge to make fantastic beautiful code.
Of course shipping fast does not imply writing crap code in general. At all. But sometimes it does, and when it does, I've learned that hard way that maybe it's better to choose the crap code.
How often being unable to change your code is the cause of bad product-market fit?
How does needing 40 people instead of 5 to run your infrastructure compares with the carry cost of monetary debt?
If you were only arguing that the people around here probably needs to be more open to technical debt because we here are mostly perfectionists I would probably agree. But I don't think I can agree with most of what is your post.
By the time we can afford to pay 5(!), let alone 40 people to run our infrastructure, I don't think you can call us a startup anymore.
This is my point exactly. Until you've found product market fit, the size of the infrastructure team "once you're big" should be the least of your concerns.
Personally I don't understand how any tech company with a 5+ person infrastructure team can still be called a startup (unless the infrastructure is the product ofc, eg if you're a PaaS or something)
I'd like to quibble a bit with the definition of technical debt being used. A decision not to refactor code when you don't know if you will be improving it is not technical debt in my book. There are many times when I look at a piece of code and say, "This is a smell here, but until we know better what we are building I can't remove the smell". It actually would be technical debt if I covered up the smell with something pretty that wasn't related to what I actually needed. Later on I'll have to remove that pretty code (or work around it). Hopefully making me feel better about apparent lack of smells is worth it ;-)
The idea of technical debt is that you are trading some kind of completeness/correctness in the code for something else that is valuable. Usually that valuable thing is time, but it can be something else. For example, perhaps you know that your requirements are wrong. However, the client is unhappy with answering questions. You could pester them some more, or you could implement the obviously wrong requirements and wait for them to find the problem themselves. In this way you defer the frustration of your client to a later time when they might be more approachable.
It should go without saying that you should almost never do the above: I'm only using it as an example of tech debt that isn't to do with saving time.
There are lots of times where technical debt is beneficial. The key is to understand the cost/benefit. This is where we usually go wrong. The cost/benefit analysis is usually done by someone who only understands the benefit. It's just making the analysis of borrowing money and making the assumption that you don't have to pay it back. Hmm... how much money should I borrow in that case? Obviously, as much as I can get!
I think this is just another way of saying "avoid gold-plating." Building a new service to stand up to millions of users and nation-state attackers is probably overkill until you know whether anyone actually wants it.
That said, there are definitely early technical decisions that will be very painful to change later on, like how you authenticate users and protect PII.
There are different kinds od technical debt. One should aim to keep abstractions simple, and to be able to refactor as quickly as possible. But that also means that the code should be manageable. Unreadable code is never a good thing, and will slow down development considerably.
"Technical Debt is most of the time used as a negative connotation, but it doesn't need to be like that all the time. It's like taking a loan. It will allow you to deliver software faster in adverse circumstances without having to invest enough time to find the best design for a problem."
Lol yeah, taking out a loan for all your future hires to pay back later.
"If there's a chance that the code you're writing will never be touched again, why bother to invest a lot of time in it?"
In that case, leave some Technical Debt at the end and allow the next requirement to drive a better design."
"If there's a chance that the code you're writing will never be touched again, why bother to invest a lot of time in it?
In that case, leave some Technical Debt at the end and allow the next requirement to drive a better design."
I found the article excellent and I like the financial debt metaphor a lot. However, the quote above is somewhat confusing. I usually get rid of technical debt (post task) so that a developer can read and understand the code better. They need to do this in order to make some future unknown change. Since a "better design" would at least involve understanding the current design then a strongly disagree with leaving technical debt in code you do not think will be touched again.
While it's excellent to make the code readable for the next guy, I read this as talking about the architecture level, where reading the code will be largely unnecessary because it is all junked for Rev.2
I did this for one SAAS company and it worked out great. Although I had been very much a design-everything, pre-optimize excessively type of coder/designer, for this project at a company I co-founded I went completely the opposite direction.
With only a slight nod to the architecture I expected we'd need later, I just coded up the first version in the easiest high-level tools we had, separating only one module. This allowed us to get a much better handle on the actual requirements, especially since the marketing guys of course went out and sold some of the prototype. Fortunately the sales were small enough that we didn't get swamped, but it gave me real data, and strongly reinforced my plan to build a highly modular set of interacting services such that each could run on separate hardware, which could also be scaled in hours. This very scaleable architecture later allowed us to take business from competitors who were (AFAICT) were stuck in a monolithic architecture.
But after the first version had served it's purpose, no one has ever read any of the code again in the 15 yrs since. We just started on the main version and never looked back.
So as a person who formerly would have recommended the opposite, I can strongly recommend making a really brief Plan A prototype / first version that you plan to throw away. Just make sure that you do actually throw it away, and soon (code that sort-of works does have a way of sticking around too long).
Good point. I guess it boils down to team size. If the team size is large then the code is more likely to be refactored in stages than rewritten. And more people on a team usually means more code reading than writing. Also, you have a higher turnover of people so onboarding cost is important.
Small team size means you don't have a lot of these problems.
not necessarily, 50-100 person team in random bank for example can’t even do all of the tickets they should in whatever timeframe they have less alone refactor something properly, and not cos they are busy, but for variety of other reasons, one of them being turnover.
this has a lot to do with how your code looks to begin with, bad code is generally hard to refactor, and refactoring is not a cure for bad code.
I'd think it may not apply so much with huge teams, as if you probably need to have a very extensive definition of the problem to justify a large team at the outset, so doing the long-term architecture & design up-front (vs. a throw-away version) would likely be best.
I'd hope for a situation where the initial small team built the throw-away, ran it hard, then use that knowledge to build the big version and the big team.
I know most here are focused on the dev side of things, but coming from the sysadmin side where I've had to deal with the technical debt of devs + the entire rest of the business IT technical debt, it is anything but good.
I've seen the insides of hundred of companies, and the number one cause of technical debt is lack of proper manpower to work ratios. Usually due to short term thinking of IT as a janitorial cost sink instead of as an investment.
any company that can break this mold is going to be light years ahead in the long run, because I have a secret to tell you all...
The infrastructure holding most businesses up is set to crumble at the slightest puff o wind, whether that wind be legal, like a lawsuit or an audit, or technical like an internal data breach...
I’ve always found the metaphors around Technical Debt really interesting. They simplify it to a point where everyone “gets it”; however I have often worried whether there’s a cult of “anti-technical-debt”, much like there are the cults of “absolutely no credit card debt / usage ever”. Sure, it works for some, but I’ve also seen product progress completely halt over the argument of “that’s adding to our technical debt”, surely it would be phrased better as “this is adding to our customers experience, at the cost of our chase for purity and technical perfection”?
As has been said a few times in this thread: there’s nuance to engineering, and a careful trade off. I think the metaphors are weaponized to a point where that nuance is lost in harsh memories of student-era credit card debt.
147 comments
[ 3.7 ms ] story [ 194 ms ] threadLooking at technical debt in isolation and always regarding it as a bad thing always seemed simplistic to me.
To start with, you don't. Baseball managers and scouts look at a lot of metrics: runs, hits, walks, OBP, VAR, number of pitches in a career, convicted felonies, etc. But at the end of the day, there are people making judgement calls. They're just, ideally, exceptionally well-informed people.
So what I'd like to see, at least initially, is a healthier universe of metrics we can automatically grab from code bases. Deciders can set up dashboards or reports for those as needed. There would still be actual conversations and decisions involving actual humans, but they'd be looking at this data as they're having qualitative discussions about their options.
(relevant, not a complete answer by any stretch)
When that's the level of awareness the average development team is dealing with, there's limited practical purpose in exploring the nuances of healthy debt management. It's more useful to talk about technical debt like nutritionists talk about empty calories and transfats.
The same _can be_ true with code. There can be a better way to do something but if you cannot afford the time to develop and iterate this idea then it can be advantageous to develop the simplest solution that meets your needs today and make progress.
Regardless of how much PMs might want things or fiddle with the schedule, as professionals it's our job- not theirs- to say when things can't be done, shouldn't be attempted, or might cost more than they're worth.
[1] - https://www.youtube.com/watch?v=BKorP55Aqvg
That gets back to company culture (which others have identified higher up). Not every organisation responds appropriately when developers push back (either due to imposed deadlines or office politics).
In my experience, organizations generally do the right thing if all the developers push back. The problem is that you can always find developers that will push a hack or deploy an overwrought solution for a simple problem.
Maybe you're lucky or maybe I'm benighted, but it's rare that an organization consciously decides that this is the quarter to finish off the Java 5 migrations, even if that means delaying the Jabberwocky Project and even if there's a non-zero chance of customer impact.
Partly because it's a really hard problem to attach an accurate and objective number to the value of something like that.
> ...technical debt and the benefits and costs associated with varied approaches...
That's not entirely accurate. Engineers are experts at technical benefits of different approaches. And they should be experts on the entire cost of the systems they are responsible for. But they're not necessarily experts on the benefits. They aren't necessarily qualified to anticipate what different features and different release schedules will do to the revenue for the product. In B2C companies with millions of users, you can run statistics and A/B testing to make this stuff more objective, but a lot of companies live and die by big sales to a smaller number of customers. So determining the benefits of shipping a month early depends a lot on knowing particular people well enough to guess how they'll react to good or bad news.
One solution I thought of was to enact a 20% code debt policy, similar to google's 20% time but for code debt. Hey developer, is there something causing you pain? Fix it!
That's sort of how I do it. The key is to never tell management explicitly. It's just normal part of the job.
The only problem is that this setup is subject to the tragedy of the commons. If it takes 25% more work to make a change cleanly, the developers throwing hacks around look 20% more productive if we ignore technical debt. And, worse, they're often the only people in the world that understand the hacks, so they get to be the hero-experts that come in and save the day when everyone else is trying to figure out what the hell is going on.
So even if we'll do our best to bake the cost of cleanup into our day-to-day tasks, that only works as long as a critical mass of the developers in the organization do the same thing. Otherwise, you'll have a full time job plus some just cleaning up messes.
This works only if you own your part of the code for a some time so your investment can pay off.
There are two outcomes (1) they trust you, and grudgingly accept your estimates. You address what debt you can and ensure padding doesn't increase the next time thus trust is maintained (2) they force a timeline on you, in which case when it's late or broken in production you can at least say you needed more time.
There is a third more egregious possibility, that they get an alternative estimate from elsewhere or one of your colleagues and in my experience this is the principle weakness to standing your ground "we'll take our business elsewhere" and I am truly sorry for your trouble if that is the case. Hopefully you have secure employment and the whole mess then just becomes the problem of the person that undercut you.
Of course, in all these situations you have to make a call based on your personal/organisational circumstances but I just want to say that ultimately being upfront, diligent and continuously demonstrating trustworthiness may not always pay off in each particular instance, but over the longer term as reputation accrues they do.
Can you give an example?
Something unexpected pops up like a family member dies and you need to arrange a funeral, but don't have the cash on hand because some other emergency last month used up your liquid savings. You know that you can pay it off over the long run, but don't have any other way to deal with it right now. So you go into a small amount of debt that you can likely pay off without paying too much over the original cost.
There are a million ways that they can be good. There are probably more ways they can be bad, but this question is very simplistic.
Why bad for you thing is actually good for you. or Why thing everybody likes is not so great after all.
Is everyone on hacker news just a compulsive contrarian?
No we're not.
Whenever I'm told that I'm not allowed to disagree for some reason or another, I immediately stop and reevaluate my views. Usually they don't change, but it's worth it for the times that they do.
And often times when I've changed my views like that, there's a cultural shift a little while later. A lot of what is held as sacred truth is due to fear of the "opposing idea" being right.
Think about programming in the late 90s and early 2000s, but not believing OO to be a panacea. I use OO when it's appropriate, but it's often not. And speaking up about that in those times meant you were viewed as ignorant.
If you program in Java in the year 2000, you have a vested interest in OO adoption. You've invested your time in a language that was explicitly designed to not allow for anything but OO. And if OO is not the end-all be-all of programming, then maybe you've wasted your time. So anyone claiming otherwise is indirectly threatening your livelihood. You're terrified that they might be right, so you insist that they never could be.
You say that but at the same time there's people that voluntarily had holes drilled in their skull.
We agree!
I woke up immediately because that isn’t possible.
It is a strange phenomenon, that the brain tends to ignore things that are expected, and is often jarred by the unexpected. It’s maybe one of the downfalls of expecting the unexpected. Because it makes even the unexpected seem ordinary. Or maybe the reverse is true, and expecting the unexpected makes the ordinary seem extraordinary.
I’m gonna go and have some breakfast.
Yeah this is what I was getting at. Ive been noticing that people have been dissatisfied with the quality of content recently citing marketing type activities like this. Kind of wish it could just be a forum without the competitive drive that we are bombarded with everywhere else.
Joining a team that is scared of technical debt and runs circles round themselves in a bid to avoid it end up producing more debt than they would otherwise. For example, making code overly DRY and wrong abstractions coupling unrelated parts of the system unnecessarily.
We must embrace the reality of technical debt. It is unavoidable and not something to be scared of. It can be managed by keeping things simple, clearly tested and keeping a cool head.
I would say that's just bad engineering due to lack of insight or experience. DRY is the most simple and most dangerous principle (or "best-practice") of them all, because when it goes wrong it's the one that hurts the most.
That's just another form of technical debt. Over-engineering is also a well known cause of debt accumulation.
When you're writing it the second time, you're accruing technical debt. You pay it off when you refactor for the third component.
I think this is both an art and a science, and the right balance comes with not just engineering experience, but also domain experience, team maturity/talent, market/competitive pressures, leadership, and culture.
There are so many factors involved that it's not surprising so many projects accrue horrible debt.
I couldn't have stated it better myself. Most of the time it start from culture, leadership and then experience etc.
I have worked on multiple projects with varied amount of technical debt. But the project with the most amount of debt happened because of poor leadership.
The guy who was managing the project was more interested in currying favor. The end result - everyone's salary was being shared across the company in un-encrypted plain text format.
It reliably fascinates me that in spite of more than a century of history, management culture lacks both words and concepts for many common and predictable organisational failure modes.
Does it? The manager in GP, for instance, seems to be a textbook example of a thermocline of truth.
> the right balance comes with not just engineering experience, but also domain experience, team maturity/talent, market/competitive pressures, leadership, and culture.
>the project with the most amount of debt happened because of poor leadership.
If your system started out with tens of requests per second, and it was designed for that and it works fine under that load, having it break if the load increases 10x is not technical debt
Or it might not even have existed (or was unstable) when you started work, but it got better later.
I'd also urge you to use higher-level languages from the get-go. If you can use Rust instead of C, do it. If Haskell is appropriate, use it. These are all choices that are fantastically easier to make early on than later.
Its the same readon why I almost always immediately use vue or react. Because it doesnt cost me much (~0) but if that thing has to scale i’ll be ready
Obviously there's a bit of YAGNI in tension here.
But if you have the experience to write it correctly from the get-go at low additional cost, then it's worth doing that instead of risking a massive rewrite later.
I have never had any problem with putting some code into a library after the software is done. It is almost always a 1-4 hours exercise, and when it's not, it's a 2 days task at most.
Async code is also something that I have never seen it being a huge problem. Mostly because only a small part of your code needs to be async, most of it can stay synchronous, scheduled by the small async part.
Thread safety is a real problem. One can indeed trade some of it for some development speed, and this can become a huge bottleneck later/soon.
I used to think this was true, but after talking to architecture/civil engineering friends a lot about this, I've learned that this is less true than I believed. There are often changing requirements for buildings due to different use cases, changing regulations or building codes, or a change in energy prices (or the social cost of using a lot of energy). Sometimes new technologies are used without a lot of experience with them, leading to high operating expenses for the lifetime of the building.
I asked my friend about your comment and he gave me a few interesting examples: 1) "flying form construction" that leads to exposed concrete slabs on buildings that bleed heat in the winter. This was really cheap to build but turned out to be really awful for energy efficiency (once we started caring). 2) a building in the 50s-60s that was built with automatic window blinds. The maintenance cost of fixing the blinds when they inevitably broke led to them not getting fixed, and the building becoming basically useless.
If you're interested in reading more about this, Stewart Brand's "How Buildings Learn" is a really good read for laypeople. I was really surprised by how much of the book could be applied to software engineering.
(This is not a field I'm familiar with so any errors are mine, not my friend's.)
They have now finally decided to move the MPs out so they can do the refactoring they've needed to do for years...
Sometimes, supporting the legacy system can be technical debt. But sometimes when people try to replace the legacy system, they spend a lot of time and money and end up costing way more than simply maintaining the legacy system. This is part of why some nuclear plants will continue to operate on PDP-11's until the year 2050.
Technical debt isn't really debt if it's actually just a part of the cost of doing business. You design your system for its intended purpose, and if it's accomplishing that purpose well, you don't need a redesign. Don't break what doesn't need fixing.
My greatest accomplishment at my current job has been deleting 45K-60K lines of code.
Not every bit of code will be perfect for many reasons, but please be thoughtful at least, and not just hammer out code. Someone else is going to have to clean up your mess.
I've created loads of technical debt over my career, in the process literally creating whole product lines that later became the career home for teams of people.
So it is more like survivorship bias, you only get to fix debt that is still alive. Debt that went away and no one is ever paying it is nowhere to be seen.
So it's like survivorship bias, where the most important technical debt (the one that kills its project) is often not noticed.
When I'm a little more aware of technical debt being created - I track and manage it just like a feature or a bug in the project management system. Separate type, label or category of ticket. It surprising several tools don't have this out of the box for some reason.
There's a few benefits to technical Debt being categorized separately:
Tracking technical debt helps provide a sense of the technical debt tickets that are being opened, and floating around both in numbers, size and unknowns.
Keeping technical debt items tracked along side features, and bugs in it's own category is an invaluable form of cross-training to new team members who are learning the why the current code base came to be how it is, and not just what they may see or think is best.
We see how good we might be as a team and individually at identifying technical debt, and if we are improving at identifying it, and help develop a ratio for future accuracy.
Regular technical debt review helps create a sense of what technical debt is occurring, and what can be fixed under the hood on another project, if desired, or possible.
Keeping an eye on technical debt is important too, if something initially small is at risk of painting the project into a corner, or becoming exponentially difficult to re-factor.
Would love to hear strategies and tactics you might use in your TD practice.
We make the technical debt ticket anyways describing the problem, taking a few more minutes to describe "how it should be" so a design process can begin... it will be much easier to pick it up later on... maybe even tie it to an ongoing Technical Debt Roadmap that can be tied into the main product roadmap.
The metaphor in this article fits perfectly- the developer who wrote this borrowed from the future, and without being there at the time, I can’t really judge their calculations.
Compare that to the other kinds of debt (convertible, for instance). They cost a lot of time to obtain and, potentially, a lot of money in the long run if you do well.
Many of the most successful startups had (have?) shitty code. Of course there's great counterexamples (i.e. WhatsApp having a 5 person backend team when they were acquired), but as a programmer and founder I find that my urge to make shitty code and ship fast is waay smaller than my urge to make fantastic beautiful code.
Of course shipping fast does not imply writing crap code in general. At all. But sometimes it does, and when it does, I've learned that hard way that maybe it's better to choose the crap code.
How does needing 40 people instead of 5 to run your infrastructure compares with the carry cost of monetary debt?
If you were only arguing that the people around here probably needs to be more open to technical debt because we here are mostly perfectionists I would probably agree. But I don't think I can agree with most of what is your post.
This is my point exactly. Until you've found product market fit, the size of the infrastructure team "once you're big" should be the least of your concerns.
Personally I don't understand how any tech company with a 5+ person infrastructure team can still be called a startup (unless the infrastructure is the product ofc, eg if you're a PaaS or something)
The idea of technical debt is that you are trading some kind of completeness/correctness in the code for something else that is valuable. Usually that valuable thing is time, but it can be something else. For example, perhaps you know that your requirements are wrong. However, the client is unhappy with answering questions. You could pester them some more, or you could implement the obviously wrong requirements and wait for them to find the problem themselves. In this way you defer the frustration of your client to a later time when they might be more approachable.
It should go without saying that you should almost never do the above: I'm only using it as an example of tech debt that isn't to do with saving time.
There are lots of times where technical debt is beneficial. The key is to understand the cost/benefit. This is where we usually go wrong. The cost/benefit analysis is usually done by someone who only understands the benefit. It's just making the analysis of borrowing money and making the assumption that you don't have to pay it back. Hmm... how much money should I borrow in that case? Obviously, as much as I can get!
That said, there are definitely early technical decisions that will be very painful to change later on, like how you authenticate users and protect PII.
Lol yeah, taking out a loan for all your future hires to pay back later.
I found the article excellent and I like the financial debt metaphor a lot. However, the quote above is somewhat confusing. I usually get rid of technical debt (post task) so that a developer can read and understand the code better. They need to do this in order to make some future unknown change. Since a "better design" would at least involve understanding the current design then a strongly disagree with leaving technical debt in code you do not think will be touched again.
I did this for one SAAS company and it worked out great. Although I had been very much a design-everything, pre-optimize excessively type of coder/designer, for this project at a company I co-founded I went completely the opposite direction.
With only a slight nod to the architecture I expected we'd need later, I just coded up the first version in the easiest high-level tools we had, separating only one module. This allowed us to get a much better handle on the actual requirements, especially since the marketing guys of course went out and sold some of the prototype. Fortunately the sales were small enough that we didn't get swamped, but it gave me real data, and strongly reinforced my plan to build a highly modular set of interacting services such that each could run on separate hardware, which could also be scaled in hours. This very scaleable architecture later allowed us to take business from competitors who were (AFAICT) were stuck in a monolithic architecture.
But after the first version had served it's purpose, no one has ever read any of the code again in the 15 yrs since. We just started on the main version and never looked back.
So as a person who formerly would have recommended the opposite, I can strongly recommend making a really brief Plan A prototype / first version that you plan to throw away. Just make sure that you do actually throw it away, and soon (code that sort-of works does have a way of sticking around too long).
Small team size means you don't have a lot of these problems.
this has a lot to do with how your code looks to begin with, bad code is generally hard to refactor, and refactoring is not a cure for bad code.
I'd think it may not apply so much with huge teams, as if you probably need to have a very extensive definition of the problem to justify a large team at the outset, so doing the long-term architecture & design up-front (vs. a throw-away version) would likely be best.
I'd hope for a situation where the initial small team built the throw-away, ran it hard, then use that knowledge to build the big version and the big team.
I've seen the insides of hundred of companies, and the number one cause of technical debt is lack of proper manpower to work ratios. Usually due to short term thinking of IT as a janitorial cost sink instead of as an investment.
any company that can break this mold is going to be light years ahead in the long run, because I have a secret to tell you all...
The infrastructure holding most businesses up is set to crumble at the slightest puff o wind, whether that wind be legal, like a lawsuit or an audit, or technical like an internal data breach...
As has been said a few times in this thread: there’s nuance to engineering, and a careful trade off. I think the metaphors are weaponized to a point where that nuance is lost in harsh memories of student-era credit card debt.