64 comments

[ 3.2 ms ] story [ 122 ms ] thread
The bad thing with tech debt is that it has a very high interest rate. The good thing is that you can just delete the software and the tech debt will be gone. So if you do not have market fit, you can take a lot of debt. But if you do gain users/customers it will cost you a lot - even if you rewrite from scratch to fix all debt that will cost you too.
So much this. Woe to you if you got to ipo on a pile of garbage. But ipo you say! Good luck scaling, adopting dev ops best practices, and keeping any sort of sla’s on that. And reducing cost or on boarding new hires? Pain is in your future.
You can get to an IPO on a pile of garbage, or you can go out of business.

I think the quote that comes to mind for me (as someone who's been with 2 startups now that have grown from less than 20 employees to more than 300) is this:

One man's trash is another man's treasure.

"Good" code written with the constraints of a tiny company won't be "good" code later anyways - the requirements will shift, the company goals will change, focus will change away from some features towards other features, abstraction levels will be different, the quality of your engineers will be different.

Trying to bake in assumptions about what "good" will be 5 years down the line is literally pain for no reason, and I promise you it will be wrong anyways.

Trying to act like the company from 5 years down the line is a guaranteed way to go out of business in the mean time.

To some degree you are right, but the sign of good quality code with low technical debt is the ability to change requirements quickly, not necessarily consider future requirements.

In my opinion, investing early on in CI/CD pays so much more than thinking about code quality, correct abstractions and future requirements. If I have a fast deployment strategy with automated testing I can trust, accommodating new features is a breeze regardless of the underlying code quality.

Of course though it's all a balancing act. I could spend a year building a solution using sophisticated tools like spinnaker, only to completely miss the market at a startup. Still, if there is a tradeoff to be made between code quality and CI/CD, I'd always side with CI/CD.

I see ci/cd mentioned every time there is a talk about tech debt, but I dont buy it. The code mess is usually in the code itself, the abstractions being used, the db schema, and etc - how does ci/cd help tackle this kind of tech debt?
It does not, but I have taken on projects that took weeks to build because the one developer who wrote it only built it on his personal computer with a nest of undocumented dependencies and unknown versions, years old.

So before fixing other things you need to be able to build it.

My opinion is the same as yours, but with one modification - I will replace "code" with "architecture". Even when doing the simplest possible thing that'd work, one can write code in such a way that will create a disaster. That is the difference between a codebase that can evolve after PMF, and one that will take the company down.
Truth. Guess the question is knowing when you need the loan or not or how much of a loan to take out. I love the metaphor.
Not building garbage software in the first place is an option
If that is the situation then sure, do that. But sometimes market realities mean corners need to be cut and that is when taking on tech debt makes sense.

Better to have a business with some tech debt than no business.

Building software is not done by making one decision -- to build garbage or not -- rather, it is done by making a series of decisions, usually by many people, with different contexts and tradeoffs.
I don’t think your comment contributes anything useful to the conversation. Nobody sets out to write garbage code. Nobody decides to add more legacy code. However it still happens despite software developers best intention. So do you have any practical solutions to solve the problem?
I think management and I agree that taking on some amount of technical debt in exchange for velocity and progress is okay.

I think we disagree on how much.

> I think we disagree on how much.

And equally importantly management and engineers disagree on “when” to service the debt.

More often than not a product feature gets prioritized over repayment. And management is perfectly happy to have on calls suffer late nights fixing random bugs that proliferate.

Worse is, as time goes on, the knowledge of why a tech debt was introduced vanishes. So everyone gets scared to touch that code with a 100 ft pole.

It’s like building on shaky foundation. Once enough floors are built it becomes impossible to fix the foundation without tearing down the building.

I bristle against most articles that posit "x isn't y, it's actually z"

(Speakers like Simon Sinek have made entire careers out of this specific rhetorical approach.)

In almost all cases, x is both y and z. Such is the truth about tech debt. Tech debt can but a strategic lever for success but is usually simultaneously a burden.

I don't find the examples in the article very on point. Twitter using publicly available storage systems until 2014 doesn't strike me as a form tech debt at all. Rather it is is a logical system design decision based on Twitter's business and organizational focus at the time.

> Tech debt can [be] a strategic lever for success but is usually simultaneously a burden.

This isn't just true, it's the point of the “debt” metaphor; financial debt is also something used because it is a strategic lever for success but which is a burden once taken on. (It's true that use of the metaphor tends to emphasize the burden part because it is often deployed where the lever of success part is already internalized without regard for downstream cost.)

I have seen more than one product need to be entirely rewritten because it was just so slow working on the legacy codebase. Simple minor features (i.e the ability to do a certain kind of promotional discount) would take forever to implement without regressions - much less the performance issues, or aged feel of the application. Tech debt can be low interest (skipping some test coverage) or absolutely crippling to velocity (i.e no tests, duplicative 1000+ line mega classes).
But would you have needed the minor features had more time been spent on the initial product? Said another way, had the initial product not been developed quickly using the available developer resources, would the product have lived long enough to need more features?
Products generally always need more features. Except maybe in some captured markets or for very specific apps you need to need to either grow the product or competitors will eat your lunch. Everything changes over time - business practices, technologies, ecosystems - so most legacy applications, even if they were implemented well, need constant changes to adapt to today's users and ecosystems.
Failed products don’t need more features, which was my point.
I’m always cynical of things needing to be entirely rewritten, especially if it’s in a different language and / or framework. It’s a very expensive thing to do and there’s no guarantee you won’t still have issues. So my question is, were any of these products completely rewritten, and if so were they still successful?
I'd say it depends on context and just how bad it is. I'm in the embedded world and I often advocate for rewrites. As an example the current code bases I maintain has code that persist certain (per product) information to flash which dates all the way back to EEPROM. That is it doesn't take into account wear leveling, it's not power loss resilient, it's copy-pasted between numerous code bases except one implementation which is shared between several products, it relies on a ton of undefined behavior, it's a bad design, has no tests, and not all of our products can even be updated. Realistically there is no fixing this. It's also just one part of a larger whole that will eventually collapse into technical bankruptcy while a competitor leap frogs us.

Embedded is certainly a different world though. The complexity really comes from the fact that 95% of the code is custom, generally poorly designed without tests, and then duplicated and modified over and over again. So rewrites can be very successful in my realm if upper management can be convinced.

(comment deleted)
I’ve spent the last 9 months doing little but rewriting indebted code that is a tar pit to all new and existing devs. Next year we can hire people who will be productive, and I can actually start to add features and resilience. Glad my employer can afford this luxury.
> would take forever

But you’re salaried, so it doesn’t matter.

The article sounds over-engineered. Tech debt is a term that helps to explain to the management why you are working on improving the internals of the system instead of adding features. You manage tech debt just like any other work item in the backlog: prioritize and execute, or delete. What else am I missing?

Dirty laundry is never an asset.

Yeah, I think you're thinking about debt wrong with the last statement.

If you can borrow at 0.5% and have a fairly reliable system to get 10% returns, is your debt and ability to borrow "dirty laundry" or an "asset"?

While I understand the borrow concept, I can't see how it is useful here. Returns come from your doing something, not from not doing something. Deciding what actions would bring best returns is the essence of planning. What's special about tech debt in this picture, vs for example product features you delay for few cycles later?
Because all tech is debt.

Even the code you're proud of is going to need a maintenance lifecycle. If you don't maintain that debt, it compounds to the point you cannot.

Debt is a useful concept because almost every business person understands the risks, but most successful people will borrow anyways.

Where the analogy fails is because we pretend like "if only we pay all the tech debt down everything will be fine". Basically our problem is we think the concept doesn't apply to tech we like.

It's just a loan. You can take on tech debt in order to grow a business, but there are risks and obligations.

Loans as an overall concept are neither inherently good nor bad.

No. Don't assume it is a loan.

'Just' a loan? What kind of loan? A mortgage? Or maybe it is like issuing a bond?

It is important to talk about our metaphors in detail. It isn't just the interest rate. It is also about how the debt is structured. Can you pay it back early? What happens when you default? And so on.

Ultimately, the tech debt metaphor should be examined and challenged! It may not 'fit' reality in the ways you might expect.

Yes. The details of said loan are extremely important. That is exactly what I am saying.

Blanket statements in abstract, professing the benefits or ills of tech debt lack any sense of nuance.

I think his point was that it is like a loan in some sense, but the problem comes from thinking that it is exactly a loan (with an unclear payment schedule, and largely illegible interest).
It is relatively easy to understand a financial debt (compared to technical debt). There are straightforward calculations to talk about how financial debts combine, change over time, and the consequences of paying them back at various times.

Less so with technical debt. How do you quantify it? Perhaps we should define a Technical Debt Unit (TDU)? How do you combine two technical debts? Do you add them? What if one compounds the other? So... we multiply them? Wait, that would result in TDU ^ 2. That breaks the units. This suggests that tech debt cannot be represented with a TDU -- or any commensurable unit.

So, money tends to be fungible. Tech debt, not so much.

We can choose our metaphors. Why is the tech debt metaphor so common? Perhaps because it fits the minds and brains of the business decision makers.

Also, businesses recognize financial debt as quantifiable and manageable. It is no surprise that the "tech debt" language is somehow comfortable, even if the connection between the two is on shaky ground.

However, too often, the hard work of conceptually mapping why and how financial debt is useful seems to be waved off.

I look at tech debt as a tax on future productivity. If you see it that way, then it's obvious that 1) it's good to take it on on the short term in order to deliver 2) your productivity asymptotically approaches zero if you don't pay it back.
1 is maybe true of real world tax at a stretch ('good not to worry about tax havens and optimal corporate structure early on'?) but I don't understand the analogy for 2?
I wasnt making an economic argument. Maybe 'drag on future work' would be a better way to put it. You could think of it as adding mass too.
Oh sorry, I thought it was an analogy to income or some other 'real world' tax.

So then I think you're saying if you don't fix it, and some percentage of everything you do adds more of it, eventually in the limit you're doing nothing more than some fixed point of progress? 'Take half the previous step forward' sort of thing.

That's pretty much it. Tech debt makes it progressively harder to change a system. As it accumulates, making a change approaches 'impossible' while the work required to do it approaches 'infinite'.

Throwing one more metaphor at it: If you don't scrub the barnacles off a ship, the ship will progressively become more difficult to steer. In fact it will eventually make sense to simply scrap the ship altogether. In software we call it a 'legacy application'.

The problem is you can’t easily ween product folks and execs away from this lever; they may have pulled it so often that no one can remember a time when any push back on tech debt was possible. When it comes time to pay the debt, the original developers and product folks may have moved on which effectively expands the debt greatly due to loss of prior knowledge. When new product folks replace the ones that pulled the original levers, they can become frustrated they can’t get their features done in any reasonable time, and push for hacks so they can have something to show to grow their career. You need very strong leadership in your engineering org to ensure the lever isn’t stuck in the on position too long.
Well put. I would love to read some case studies on orgs that recovered from bad tech debt. Does anything like that even exist?

Genuine talent tends to recognize and avoid these types of orgs, and the vicious downward cycle you describe seems unstoppable, at least in my experience.

Among the possible terms applied to this tech thing, the one that seems to make the most sense to me is Rent, which I suppose could be thought of as a private tax as well.
I actually think debt is the better analogy, but the problem is, lots of folks seem to think it only applies to tech they don't like. I think this is the wrong way to look at it.

The truth is, maintaining pretty much any tech system is debt. You're going to need to update that code, patch that system, and run that JML process. This will incur an going cost.

Like any corporate debt, you borrow to make profit. If you maintain the debt whilst making profits, everything makes sense and you're good to continue.

If you fail to maintain it, eventually you'll find yourself the unable to make payments and this will manifest as downtime, fines, or an inability to change the software to keep up with the market.

In my career I have tried hard to avoid technical debt, I've learnt the project architecture before writing much code, I've refactored a lot of stuff, kept dependencies up to date, written loads of tests. As such I've stayed in jobs 4-8 years and become invaluable to each one.

However its been terrible for my career. In job interviews people wonder what I've achieved because it really is difficult to prove how you can design and write code. The guys who spend a year or two sh**ing all over the code base as they knock out new features really are doing the right thing. Managers dont care about tech debt, they just want new features.

> Managers dont care about tech debt, they just want new features

Founders (technical ones anyway) likely do care about technical debt. They know it could be death to their future ambitions.

Managers have little reason to unless the organization has a sound culture. If they feel like hired guns who may be let go if there is a bad quarter, then their motivations are e.g juice their resume with cool new technology.

Its more like experienced tech leaders have guardrails to prevent accruing unnecessary tech debt by using code reviews, arch reviews, etc
It's the asymmetrical information problem. The worse danger for the interviewer is ending up with a bad programmer who also can't get anything done. The system is optimal for hiring barely-capable people desperately operating at their max. Especially when the software projects and startups fail in the market anyway, before they every need to deal with the mounting technical debt.
Man, you can just feel the "thought leadership" in this one...

But perhaps more constructively, I've long harbored the feeling that we are not thinking about the interplay between business and engineering in very useful terms. Phrases such as "tech debt", "culture of shipping" and "product-driven" seem to me to miss a larger point, which is this: there is an inescapable need for semi-undirected, exploratory work. Expecting engineers to always know where they're headed is counterproductive, I think.

In my experience, time spent exploring, refactoring, "paying off tech debt" and other forms of undirected "massaging" of code are significant force multipliers, when performed in addition to planned, focused dev sprints. The reason is simple: manipulating the code base leads to a better understanding of both codebase and domain logic, which in turn leads to better sprint-planning, and therefore faster execution, but also to better products! As engineers perform minor refactors and improvements, they are developing an intuition for which parts of the codebase are pliable, and which parts resist intervention. They can use this intuition to better estimate delays, and better evaluate the utility of new tools and techniques. Once in a while, they may also discover opportunities for improvement, including new product features.

The over-emphasis on "keeping focus" seems to reveal a belief that the product team should set the priorities and that engineers should execute. In some ways, this is a perfectly reasonable approach since it is obvious that the product team is better informed than the engineering team about market, customer demands, deal-flow, and other such business-facing concerns. However, it is also obvious that they are less informed about technical capabilities, i.e. about what is generally possible to do. Likewise, they are also not in a position to make accidental discoveries, and can only direct product development on the basis of known knowns.

I often worry that in an otherwise sensible attempt to keep engineers focused on profitable tasks, this "product-driven" mentality stunts the ability of companies to compete, precisely by setting conditions that inexorably lead to reproducing what already exists. This is especially pernicious for (pre-)Seed and Series A companies, which need to discover their product-market fit, and make durable their differentiation, respectively.

Companies like Google seem to have some (imperfect and perhaps subconscious) understanding of this, as evidenced by the now-famous 20% rule. By spending a significant chunk of time exploring, and bettering things along the way, opportunities for significant improvement are discovered, junior programmers become more familiar with infrastructure and code, the bus-factor increases, and the 80% of time spent on focused sprints becomes more productive. It's inevitable.

Depends on team size. With larger team the technical debt turned into a positive feedback loop. Not good. Cyclomatic complexity increased. Methods became a black box. A junior coder would copy and paste a method and change one parameter to make it work for a new context. Nearly identical methods would proliferate. Milestones would slip.

Here's how I dealt with each:

1) reduced the internal team to half its size

2) fired the offshore team and started from scratch

3) personally attempted a mass refactor / 90 hour weeks

4) refactored 49K line C++/ObjC++ to 29K Swift

Result of each:

1) product shipped, became industry standard

2) Speed improved, Apple offered to feature product

3) I failed, resigned after burnout

4) Easy and enjoyable refactoring to SwiftUI

After resigning, have you had to work a day since?
Oh yeah; burnout was reversible. In my case: sleep, ketosis, senolytics, and yoga. YMMV
I think this kind of talking about technical debt is mostly just nonsense. If there is any interest percentage to be put on this it is far higher than any sort of financial department would be prepared to accept on a loan. From which it follows that the first so-called 'preconceived notion', namely 'tech debt = bad' is actually correct. The other side of the argument is the question who one lets decide what is sound architecture is. There are many, many, many architecture astronauts and cargo cultists out there. I think one reason that in practice keeping on 'technical debt' makes any sense is that one does not know whether or not the trust the person who comes up with the payback plan.
I disagree on this. The right way to think about it is that nearly every line of code you write is almost certainly tech debt, even more so earlier on in a project where you’re just exploring the problem domain at the same time as you’re building the solution. Indeed, when I’ve looked at histories, once a developer stops being active, their name disappears from the git history relatively quickly (that’s not a perfect estimate, but it’s the best heuristic I could think of. equivalent to LOC - bad in the wrong hands, good as a general directional rule of thumb)

Also people think of tech debt as a loan which is a fine analogy but it’s important to understand it doesn’t map perfectly. Some have a high interest which means your current path is just adding this code debt to more pieces in your codebase (eg a bad convention you know you need to address). Other debt is more sporadic but doesn’t really grow (eg you have to remember to do weird thing X every time you do a Y). It may manifest as a parasitic effect on your velocity (more manual processes and/or more bugs). Additionally, unlike an actual loan balance sheet, there’s no way to quantify any of this. So agreeing on tech debt can even be hard (what one person sees as tech debt another might see as at least having consistency with the rest of the code around it). If you can’t measure then you’re all talking about an amorphous concept no one really agrees on 100% and basically paying down things that everyone agrees on as debt OR getting lucky and finding secondary quantifiable measurements that correlate with what you’re arguing for as debt.

The interest on technical debt may be high, but you don't have to pay it if the product becomes obsolete, replaced, unsuccessful or improved. Basically you only need to pay the debt if the product is successful/used and not fixed, which in terms of overall risk is very different from a loan that you need to pay back no matter what.

(It's sometimes perhaps more like VC finding - a very high cost for the provided up front money, but only in a small set of circumstances where the overall success makes the cost relatively small)

I believe most code will become tech debt over time even if was not originally acknowledged at the time of conception.

Software is a leaving breathing thing whose sole purpose is to fulfill a real world intention, it needs constant service, maintenance and upgrade to be kept alive.

At least this is my world view as a product manager of many years (before the term was coined).

Ward Cunningham coined the term to mean knowingly borrowing against the future.

You seem to be describing a metaphor more like technical depreciation. Not to be confused with "deprecation." This is not what Ward Cunningham meant.

It‘s a pity that tech inflation doesn’t reduce tech debt.
The term has a very specific meaning, but I feel some people use it very loosely...

Most people at my company seem to think tech debt = legacy code that gets in the way of modernization.

Wiki says tech debt is:

"Technical debt (also known as design debt[1] or code debt, but can be also related to other technical endeavors) is a concept in software development that reflects the implied cost of additional rework caused by choosing an easy (limited) solution now instead of using a better approach that would take longer."