> Payments couldn’t be processed in different currencies
> If the delivery system is offline, the code wouldn’t work
> Users with deactivated accounts could still access the system
> No automated testing
> And this, my friends, is what we know as Technical Debt.
I do not describe that as technical debt. That are mostly lack of features. The code could be fantastic and it is just that to support different currencies, disable accounts or similar things new code is needed.
The lack of automated testing is more in line on what I think that it is technical debt.
I wouldn't even say lack of testing is technical debt - it's just another feature that isn't implemented. Once you implement it, problem's solved.
It doesn't make the system worse/more expensive as it grows, and it doesn't become more costly to "fix" the bigger the system gets. IMO those are the operative features of tech debt.
I agree, I just wouldn't call it tech debt. It doesn't become more expensive to fix over time, except for the bump when you forget how the original code works. There's no metaphorical "interest" that you draw down on.
> I wouldn't even say lack of testing is technical deb
You are right, it depends on the kind of application and circumstances testing can be done in different ways. There is occasions were instead of automated testing there are other ways of working with quality. But, in this case automated testing is in the list of things that needs to be done. So, I think that to want automated testing and not have them is technical debt.
I said this below but I like to distinguish technical debt as having metaphorical "interest" - specifically, it becomes more expensive to pay back the longer you leave it, the ball of debt grows over time. It's stuff that gets baked into a product, and then is difficult (or impossible) to un-bake, and gets more difficult the more the product grows around it.
Aside from the initial increase in cost when you forget how the original code worked, tests are no more expensive to write in a year as they are today. Not significantly, at least. So I'd just class them as incomplete product - the effects are negative and can be expensive, but it doesn't dig you into a hole.
Yeah this is not technical debt at all. These are examples of launch-aborting bad bugs.
If we try to characterize the "perfect" example of technical debt:
- There is a spec, say: v1, of how the product must behave to the user.
- The product behaves exactly as defined in v1. I.e.: There are zero bugs.
- The very next iteration (v2) of this product has some cost to achieve.
- There is a counterfactual set of decisions that could have been made, which would make achieving v2 have the minimal possible cost.
Debt is the delta between the set of decisions made in implementing v1 vs the counterfactual set of decisions not made in implementing v1 that would have made the cost of implementing v2 minimal.
This definition has all the familiar mechanics of technical debt.
- You don't need to pay the debt to achieve v2 perfectly - it would just be more expensive.
- If all you ever want to do is v1, ie: there is no v2, then there is no technical debt, no matter how ugly the code is for v1.
- You can choose to reallocate costs between v1 and v2 freely. The distribution of costs will affect launch timelines, but will have no effect on the user-visible quality of either version.
It's dangerous not to understand the difference between bugs and technical debt. I lost interest in reading this article after it seemed like the author did not understand this basic difference.
I would totally count over engineered a technical dept. Any design choices and implementation choices that make a code base limited in its future abilities to change and evolve, and which causes it to require more time and effort to be understood, I consider technical dept.
The dept is that you take time and effort from your future to save time and effort in your present.
Agreed. I once worked on a project which was a microservices based system for managing various types of infrastructure. Progress was painfully slow.
There was a separate testing client that was used for a lot of the demos. The demos on the 'real system' would fail so often that somebody came up with the idea to just take a lot of the code from the microservices and drop it straight alongside the testing client, as a monolith.
It was so much more stable that after becoming the default way to demo, it became the default way to use the tool.
5 years on and I believe it's still running that way. In this case microservices were massive over engineering.
> In this case microservices were massive over engineering.
They often are.
Well-meaning engineers and architects seem to reach for them while forgetting that microservices are about scaling your organization, not your software.
There are very few technical deficiencies in a monolith that are solved by microservices. Indeed, they typically bring technical hurdles: distributed systems are hard to reason through and IPC over the network introduces latency. And you need a very strong ops team to implement them without major headaches.
But where they really start to shine is when regression testing takes ~days and your development cycles are screeching to a halt because you've got too many in flight features and not enough runway to land them.
What takes real skill is recognizing which of your monoliths should be grown and which ones need to be sliced down the middle.
I think a lot of today's massively tested, microservice and dependency injected systems will end up as huge piles of technical debt in the future. If you think modifying a COBOL codebase today is hard, wait another 20 years and see how much "fun" it will be to rework today's systems once their platforms are viewed as outdated.
One thing I've learned is that what often looks like overengineering is actually underengineering -- it's much "easier" to throw together a complicated system than really think things through and develop something that accomplishes the same objectives in a simpler (but possibly more complex) manner.
for instance one smart and experienced guy I knew said if there was an error, your program should just fail instead of giving a ton of messages and recovering.
and if you're just throwing code together, that might be what you do.
on the other end of the scale, a well written "second pass through everything with cleanup" might have tastefully written code, a few relevant comments and consistency.
and then I've seen a lot of code that appears to look like that, but is copy/paste garbage. (sort of the coding equivalent of a wordpress theme)
> your program should just fail instead of giving a ton of messages and recovering
I would bet that he was right in that situation, but that he was also making a nuanced statement. Not all code should fail loudly, but when it should, "recovering" can sometimes be a distraction from a very real problem that needs to be addressed by the right person.
I've come round to the "fail hard and early" view, provided you can be sure in your design that it doesn't leave junk lying around. Because otherwise contaminated data and invalid states start propagating everywhere..
This can be fun if it's a particular client request that causes a hard failure, and the client retries several times before giving up. Before you know it, all your instances are restarting and no traffic is being handled...
Interestingly I prefer failing hard specifically for data consistency reasons. I've had to deal with systems that swallowed far too many errors and let themselves get into an inconsistent state which then caused other operations to not fail but instead persist invalid data. It's very hard to clean messes like that up (restoring a nightly backup is easy, but that's a last resort).
I think poorly engineered is a better term than under engineered in this case, but I agree with you.
Logging is my go-to example for this sort of behaviour. People write stupid amounts of log statements in their code, so much that it’s hard to even read the code to understand what it does, in the hopes that it’ll make debugging easier. Use a damned debugger! Take a traffic dump! Use strace!
What’s more, it’s extremely rare that libraries provide logs themselves. So the actual complex parts of your application, like say the HTTP library or (God forbid) the TCP stack, can’t be debugged this way.
If you find yourself writing a bunch of statements like “DEBUG: updating balance from 1 to 2”, stop and write some tests instead.
> Use a damned debugger! Take a traffic dump! Use strace!
I have encountered a fair amount of cases where that does not work.
- debugging a super large program ? loading gdb may take two minutes while recompiling to add a printf only 3-4 seconds.
- not an admin on the machine you are and the person with the admin account is not around ? sorry, you can't debug on macos (and likely in some linux distros)
- likewise, no traffic dump (and I'd assume no strace) if you don't have root access
And how about highly congested multi threaded programs?
I've got a program with log statements like that all over the place. Since stepping through it with a debugger would not even be possible. My IDE takes care of hiding the debug statements, since they're all encapsulated in different regions and if-def statements to log different types of things.
I certainly wouldn't want to debug this ball of mud if there were no log statements, no :)
But, I think we're depending on different programs in our workflow. I much prefer a logfile with statements that focus on what I want to inspect at the time.
I'm quite proficient with debuggers on both Windows and Linux, but I tend to use them less when dealing with my own multi-threaded code.
> loading gdb may take two minutes while recompiling to add a printf only 3-4 seconds
To be clear I’m not talking about adding some specific print statements to the code while debugging, I’m talking about keeping those statements in production code.
Of course, there’s cases where logging might be your only option. In that case: Go for it!
> - debugging a super large program ? loading gdb may take two minutes while recompiling to add a printf only 3-4 seconds.
Relinking a super large program may take minutes, whereas loading gdb can get me a backtrace in 3-4 seconds! VS natvis files and other visualizers can hot-reload while I'm looking at a crash dump from production that took hours to trigger/gather/repro!
There are cases for logs - where sufficiently efficient conditional breakpoints are particularly painful to setup, for example - but the only time I've waited minutes for a debugger to respond has been with VS after major system updates as it refreshes a universe of symbols from the symbol server.
Do you have some really slow python scripts auto-loading in GDB or something?
> Relinking a super large program may take minutes, whereas loading gdb can get me a backtrace in 3-4 seconds!
well, we have definitely opposite experiences :) the main software I work on creates a ~1gb binary in debug mode and lld links that in a few seconds. Bud gdb and lldb, even with gdb-index, and all the optimizations I could find, both make startup slow enough that I can go for a coffee and it's not always finished loading when I'm back
On the contrary, I've spent months recently working on a system loaded with tech debt that has fallen victim to a half done rewrite. I've used a debugger on it plenty of times.
It was only when I added proper logging to the critical junctures in the code that I could finally see very clearly how it behaves and see what input results in what code paths and output. I learned more in an afternoon than I did in a month.
A blogger I read when people still did such things asserted that this sort of overengineering is fear. It's really dreadfully obvious once you know to look for it.
People are so afraid of making a bad decision that they refuse to make any decision at all, and then make a giant mess in the process. What you should do is spend your energy on finding reversible decisions, and then not spend a lot of effort on actually making them. We know where the paint store is, we know they can make up paint in 15 minutes, fuck it, paint it blue, we can always paint over it later. Hard no to black, though, since you can't paint over that shit.
People are so used to avoiding decisions that on a few occasions I've entirely flustered someone who wanted to tear into me (sometimes with an audience) but cutting them off and saying, "Yeah that was a mistake, and here's how we're going to fix it." None of them had any idea how to recover from someone saying "I was wrong" and going on to try to fix the problem. I still have a little video in my head of one guy's eyes bugging out when he realized what I just said.
Which may be a great thing if it saves you time refactoring or modifying code down the line. The hardest thing about code is not writing it but reading it, and clever succinct code can be very hard to read.
The definition of "technical debt" is so pliable that we can easily regard all the parts of the system that are overengineered to be a form of technical debt.
Yeah, to me there's a natural progression in understanding of system design.
When you start out, you just write code and don't think about how things interact with each other too much.
Then you learn that you're rewriting a lot of code, you overcompensate and start to over-engineer to avoid duplication and often to handle more scenarios than you need. You abstract out possibly too much.
Finally, you get to a phase where you realize a lot of abstractions that you've created aren't actually used and you're writing more "meta code" than you need. In this phase you learn to engineer just enough for what you need at the time and design things to be easily changed in the future. You trust that even if you're system design can't handle everything right now, you've designed options for yourself to expand as needed.
Nowadays every CS student reads HN and blogposts like this one so by the time you end up having to work with them they're so indoctrinated into the "must write Enterprise(tm) ready code" mindset trying to teach them to be more relaxed is a real pain.
Yup, adopt to the business you’re in. As in, the company. Make suggestions if you think it’s worthwhile but in the end accept that there’s a “that is the way” in any company and you’re getting paid to learn and work with that “way”, and build it out :)
this last phase in your coder evolution reminds me of a very rough translation of the wabi sabi view on manufacturing. "Nothing lasts, nothing is perfect, nothing is finished."
Agree with this 100%. I've seen both in older codebases.
It's even more frustrating when you work with engineers who refuse to believe that any code they write could become technical debt in the future. These tend to be people who overcomplicate systems to anticipate future requirements
Totally agree. As long as the system is easy to change, I don't care if it's "technical debt". I'll even go further, I don't care if it doesn't give us "flexibility" and "modularity". I don't care if it's not "generic". There is nothing that pisses me off more than people designing systems for future requirements that are not even being discussed by product leads.
It's kind of amazing that "over engineering" in software means the exact opposite of what the term should mean.
To me, engineering is taking a problem and the corresponding constraints and building the solution that satisfies it with the least amount of resources.
"over engineering" in software is about trying to make something adaptable? Or maybe just satisfying someones sense of beauty.. Or maybe it's just about programmer convenience at the expense of customers/user experience and hardware resources..
It's also a bit of an abuse of the term. Overengineering in engineering (mechanical, civil and other similar disciplines) means building a component to withstand more than the absolute minimum. So, a bridge you'd want to be overengineered by a factor of 2 or 3 at least, whereas airplanes are hardly overengineered (lower factor of safety).
But isn’t that how the term is used? The system is over engineered when it is built to scale to a million concurrent users while we current only have a hundred. Or when it is built to handle use cases that no one has required.
In this context I read it more as being excessively complicated to some unspecified end rather than engineered to withstand a load way larger than necessary.
Often it is excessively complicated because it’s divided into dozens of services to be able to scale, or having extra abstraction layers to deal with imagined future requirements.
Sure, but the fact that those requirements are imaginary instead of real is why it’s not actual overengineering. Overengineering is done with a specific dimension of scalability in mind for something that probably will happen, and there’s a budget for it too, etc.
I use the term to describe the introduction of unnecessary abstractions in the name of hypothetical future use cases. For example, using an enumerated collection of values where a boolean would suffice.
It could also refer to building resilience to load before it is necessary. Example, distributed databases with failover and recovery capabilities for a side project.
Poke holes all you want, those are bad examples, but they should illustrate the point. Over engineering is a real phenomenon.
Under and over engineering are relative to your understanding of future requirements, which by definition are both unknown and subject to educated guesses. Making such guesses correctly is what distinguishes a good engineer from a bad one. Nobody ever spells out the current requirements to any level of detail even; so forget about knowing about future ones. You just have to know, recognize, and adapt to that constantly.
A lot of over engineering is due to getting this completely wrong: you prepare for something that never happens, at great cost and while underestimating the difficulty level. So you end up wasting endless amounts of time on stuff that has no business value.
On the other hand, a lot of under-engineering is due to not having a clue about what is obviously coming next and getting caught by surprise by completely obvious requirements.
> understanding of future requirements, which by definition are both unknown and subject to educated guesses. Making such guesses correctly is what distinguishes a good engineer from a bad one
I would rephrase this as make guesses based on probabilities and urgency estimates from past experience. You can't know the future but you have stats of the past. So a good engineer is one that has relevant experience and applies it appropriately.
Seems to me the default is obviously getting things wrong whether that be over or under engineering.
How would one go about reliably making choices that strike exactly the right balance consistently?
One thing I'm starting to really internalize is that to do so requires a deep understanding of software engineering as a domain, the domain and existing system, and the technical vectors in the business.
I think I write code the way people describe Jazz. You have to listen to the notes he isn't playing.
I leave breadcrumbs and openings in my code all the time so that the next time I'm in there, I'm set up, invited even, to ask the question I left unanswered before, and maybe do something about it. I leave the option of adding a feature, instead of building a config framework to support it and then defining one implementation. Which will end up not being the correct API when (if) I write the third one.
Sometimes people beat me to it, and get excited because they had an idea they're now invested in. I usually just let them have it, don't point out that I put the idea in their head. It's so infrequent people get that invested in functionality, you have to encourage it. There's too many things I'd like to do and never enough time anyway. They've just nominated themselves as a candidate for maintaining that module when I get tired of looking at it, freeing up my time for things nobody else cares about until it's done.
It's like if you were designing a house with plans to expand it later. You'd be very careful about certain decisions, like whether you should put a bedroom in the logical spot for an addition, because now that bedroom also has to be a thoroughfare. For 20% extra effort, you've extended the calcification threshold for that house/code by 80%.
I'd call React and Redux both massively over engineered. There's a ridiculous amount of lines of code being executed to do the simplest of things. You do something like `a.b = c` and really `a` is a proxy so that setting `b` the setting gets tracked going through 100+ lines of code all of which has tracking structures that get setup and torn down between components. They claim this will make your life easier but when the facade breaks you end up having to design your entire app around the fact that `a.b = c` is not simple, it's complex and so you have to avoid it as much as possible my piling on more complexity.
All GUI frameworks hide a lot of what’s happening in the insides. There is a scale to it but personally I think that reasoning about a “reactive” program is quite easier than untangling a ball of callbacks written in say, GtK.
Reactive frameworks indeed hide a lot of complexity but that does have positive impact on the user code. (IMO)
A simple react/redux app is definitely much harder to understand than a simple vanilla js app. But a large, complex react/redux app is far easier to understand than 99% of large, complex vanilla js apps.
some lib recommended to remove all the redux boilerplate of react, uses proxies. It's mostly irrelevant though, the point is there's too much code running to do what should be simple things.
“There are two ways of constructing a software design: One way is to make it so simple that there are obviously no deficiencies, and the other way is to make it so complicated that there are no obvious deficiencies. The first method is far more difficult.”
― C. A. R. Hoare
This is how I approach software, but there are so many people out there who won't approve simple code, because it doesn't have enough configs, or classes, or whatever. It seems that the person who wants the most over-engineered code tends to get their way. Psychologically the absence of 'things' is always inferior and harder to argue for than having more 'things'.
C.A.R. Hoare: 'There are two ways of constructing a software design: One way is to make it so simple that there are obviously no deficiencies, and the other way is to make it so complicated that there are no obvious deficiencies.'
The title the author chose does a bit of injustice to the contents of the post.
The post uses a lot of words to describe an understanding that experienced engineers and managers should have. Technical debt is a trade off. It exists in nearly all software of decent size as it's being developed. It is vital both for management and engineers alike to keep it at a reasonable amount, or you'll wake up one day with something that can't release new features, function, scale or be maintained.
It is also worth stating that tech debt can have little to no siginifcance to the success of a startup.
Technical debt starts to matter when the initial project changes focus or patches start mountig.
These days rewrites are popular. They make sense in some cases (when the project is smaller or introducing new framework). But in others where the original developer and business people have left and no one knows exactly how things work aside from they need to continue to do the same. In those cases fixing technical debt is extremely important.
A startup is more likely to fall into the above category if they ignore their technical debt because rewrites are hard and messy.
Seems to me that technical debt needs to be dealt with as a function of the revenue the company/product is bringing in.
The wisdom comes from timing the investment so it’s not destroying the “mature” product due to entropy or strangling the new product due over engineering.
Startups do seem to have bigger issues than technical debt to deal with. Because relatively speaking, there are bigger mountains to climb early in the startup’s development.
On the other hand a lot of efforts to avoid technical debt end up producing more technical debt because in the end you don't know where things will be going. I think refactoring just needs to be an ongoing effort and should almost never be an explicit task.
If, for example, there is no requirement to support multiple currencies then there is no need to spend time on designing for multiple currencies "just in case".
This is not choosing an easy solution, this is doing just what needed instead of wasting time and efforts on something that may never actually be needed.
Now, the design should be sound and follow good practices. This is how to limit the rework needed when new features are added because it is impossible to design to cover all possible future features: Do the minimum but do it cleanly.
Absolutely. There are payday loans (poorly developed systems) that are clearly always a bad idea, and then there are mortgages (well-architected w/ focus on minimum viability) that get you much farther in terms of realized value than you could by saving up for a house (over-engineering).
I'm being a bit blunt here. It's easy to say Technical Debt will ruin your software when you pick a contrived thing and label it as Technical Debt.
> John knew his code had a few bugs. But nearing the deadline, another project appeared and it took a lot of John’s time, so he couldn’t go back and fix the problems.
This is a problem, but this is not inherently technical debt. That's just delivering a bad/incomplete feature (Not necessarily John's fault if he's been asked to make changes). Code with Technical debt works or mostly works, but relies on non-maintainable or less ideal than patterns.
Simplified example. Adding a "Cancel" Button to a form. This button involves adding a new "secondary" class style to the button.
* Technical Debt - Button is in place. Works correctly. Instead of a <Button> component and tweaking it to handle the new "secondary" class. John simply copies and pastes the <Button> code directly into the implementation.
* Bug/Bad Feature - John references the button correctly, but fails to handle certain state resets properly when the form is cancelled.
-----
The prior is technical debt because it works correctly, but doesn't follow a good pattern or does something unexpected (refactoring that button will be much, much harder). The latter is likely just bad code.
The case where the latter is not bad code is if the company has made an intentional decision to not handle cancel edge cases. Even here, I would argue that is product debt - not technical debt.
----
Every software product in the world has debt. Just like every software product in the world has bugs. You need to account for your debt as you grow. Otherwise it will eat you.
There's a grey area though, where you properly define a new style (yay) at the time the first Cancel button is created, but then down the road you find yourself with a whole bunch of new styles for things, and it turns out over time that the styles are maybe repetitious or subtly inconsistent or whatever. The debt has become the reality that the initial abstraction ("all our buttons are the same") hasn't kept pace with the needs of the product as it has iterated forward. Paying down the debt is creating new abstractions and updating all the buttons to be using them.
The worst case scenario is probably partial paydowns, where the abstractions get changed but some of the non-aligned stuff remains and so that you end up with a jumble. Especially combined with inadequate documentation (or a culture of inadequate documentation, where people don't even look because they don't expect it to be there), the old practices can end up being cargo culted forward even when someone attempted to create a better way.
On a slight tangent, my team had very good results when we finally brought a style guide online. Our org was big enough that people would be working on pages without context for the bigger design themes (especially related to sales/marketing). We required any UI variations to be added to the style guide, and we required a minor justification to add styles. This added just enough friction that a lot of the unintended variation went away, and a lot of the variation just to be different became explicit so discussions about the design could be more deliberate.
How did you enforce this style guide, e.g. to make sure that sales cannot simply write a proposal in Tahoma instead of your company font or marketing publishes a newsletter in Arial using underlined words (gasp)?
A couple of years ago I had to spend a considerable amount of time PoC'ing platforms and tools claiming to solve this issue and I remember there were virtually none that did this on-prem using established Office programs, without a four to five figure investment. The organisation in question ended up using password-protected Word files with forms (a royal pain). Before that, they had been using Excel files (apparently it's easier there to lock parts of the sheet you don't want users to mess with) but this was an even bigger pain as Excel still has this decade-old bug even in its latest version which distort graphics when row height is changed...).
Long story short, I'd be curious how you solved this and what software you used.
Mostly we enforced it adhoc. It helped that the whole design team was pushing for print standards at the same time.
It ended up being a bullet point that needed to be talked about during sprint planning. It helped that designers were part of some of these meetings. If our company had been bigger it would have certainly been more challenging.
Mostly we tried to make it part of the culture that we stick to what’s in the style guide. And anything that deviates had a task of adding it to the style guide.
I have to say, that I wasn’t completely on board for a long time, since I was sure company culture would make it hard to succeed. We had a champion who got us about 90% of the way there before leaving. But there was enough momentum that another dev managed to bring together the right personalities and solidify the mechanics of dealing with the style guide that it was a no brainer to make it part of the work flow.
We used git to store the html and css examples. Made it trivial to copy. And since devs controlled the repo, devs controlled the pull requests as well, and required review just like other PR. It helped that the designers we had/hired were willing to deal with git, even if they had low tech knowledge.
I’m not at that job anymore, so I don’t know how long before entropy wins. It’s the type of thing you have to keep tending for it to continue to be useful. But enough process was in place that it will likely continue to be useful for them for several years at least.
I would argue copy/pasted code is usually not technical debt because it's very isolated and easy to change, and it's cheap to consolidate it later. Crappy abstractions are a way bigger source of technical debt than repeating yourself IME. It's a lot easier to create a new abstraction than modify an existing one, so you generally want to wait as long as possible.
A single copy and paste isn't likely an issue. I've worked in code bases where that pattern ends up happening in dozens or hundreds of places - all with slightly different style/layouts. It becomes a night mare to change anything consistently across the code base because it's done differently in every place.
Copy/pasted code becomes technical debt at scale, especially if there's an unknown error in the original. Now you have 10s or 100s of variants out there, each individually tweaked and so not readily identifiable as true copy/paste repetition, all containing some variant of this subtle bug. You find out about for interface A, and fix interface A. Maybe you remember that B-C were copy/pasted versions of A and check them out and correct it there. But D-Z were all copies made by others, following your example, and you miss them.
It's not scalable, and produces severe consequences when used without care (which is most of the time, it seems).
I think you are being downvoted for the first part of your first sentence, with which I also somewhat disagree: copy/paste usually creates technical debt and even though each pair is easy to fix, with time they almost always diverge and are forgotten about. Which is bad.
But I strongly agree with your second part: bad abstractions are a much bigger source of technical debt. Abstractions that do not match the problem lead to convoluted solutions, reduced functionality and are very hard to break out of with incremental fixes.
I think that's why I'm being downvoted too, but frankly if the codebase is so bad that something is being copy/pasted hundreds of times to the point of unmaintainability, the abstractions that same team would build would probably also be disastrous, and much harder to deal with. Is that preferable?
But tbh, I'm skeptical of the premise to begin with. It's still easier to gradually consolidate 100 slightly different uses of a pattern than blindly create an abstraction that adequately covers those 100 uses. If your team is useless, it's better that their code be fragmented and atomic anyway, because big mistakes are inevitable. You don't want to make it hard to touch those mistakes. You want that code to be uncoupled & disposable.
But at that point the argument is getting pretty abstract. Depends on the specific codebase.
It seems like your definition of technical debt is whenever anything is not completed (to your satisfaction), which seems a bit broad. I look at technical debt as the results of deliberate (architecture, design, or prioritization) decisions which have adverse impacts on maintainability.
I don't think there is so much of a difference between the deliberate decision and the accidental implementation of an incomplete or broken feature.
I have written software with tests and without. The latter start like prototypes and become quite useful over time. At first, everything looked good and there where just very few minor bugs which could be fixed easily. But over time, there appeared more edge cases which cannot be tracked down. As there are no tests, there are no specific definitions of what each part of the software is supposed to do. Yes, there is a big picture of how everything should work together, but obviously it doesn't catch every edge case. The problem is, that as a developer I can't act as effectively on my code anymore, because I took some shortcuts in the past.
On the other hand, there are other projects where I have test suits and where I can be totally confident, that everything has a well defined spec of what it should do and where I can't break something accidentally without noticing it.
So in the end it doesn't matter if your shortcuts have been made deliberately (most of the time it is a mix anyway). What matters is that due to the shortcuts you can't act as effectively on your code anymore.
> where I have test suits and where I can be totally confident, that everything has a well defined spec of what it should do and where I can't break something accidentally without noticing it
I'm often in the middle. Where I have tests, they're not always (or rarely) the result of a "well defined spec". But... the tests do reference whatever the understanding of XYZ was at that time. That's often as well-defined as I can get. Clients will come back months later and say "this is broken". Well, no it's not. The tests work just fine. "No, that's wrong, it's not supposed to work that way". Well.. there's a test with notes indicating "do ABC then XYZ" and the test ensures that's working based on what was known/agreed on at that time. That's often not the same as "well defined" by a long shot.
The point of the post gets to the need to account for the debt and handle it. I think you're in agreement. And to watch out for reckless debt- it gets you into more trouble than it's worth, by definition.
The subject has been written about many times. This is a pretty good summarization of it.
Not called out:
* Reckless debt is misfeasance on the part of developers.
* "Prudent" debt is often avoidable- arbitrary deadlines are arbitrary. Management may believe that enterprise data models are as malleable as CSS. So "reckless" needs to be applicable to management as well.
I'm swimming in many years' accumulation of both reckless debt and reckless mgt. The deliberately considered debt I can handle.
Bugs can also be technical debt, but then it has to be that people make a conscious decision that the bugs are not that important - like John says this will have a problem if people do X and the decision is made that X is an edge case that won't happen and John should work on something important.
In the early days of computers the first working version was always so slow rewriting it a few times was an obvious step. You would end up improving a lot of things not necessarily related to speed. I still often crank out a really shitty version that works just to get to the points that require some deep thought. You can get a lot of work done if nothing really matters. The first rewrite is helped a lot by having done things before. The second full rewrite sadly never happens anymore. Just incremental changes.
Bad tech debt will ruin your software, but good tech debt is a smart investment in your business. It takes a good team working together to balance their "tech debt budget" against the other revenue needs of the company.
If I've run on Version X of some DB software for a while, and tomorrow Version X + 1 is released, yes, I should recognize that staying on Version X is tech debt. But that may absolutely be the right thing to do for now.
Technical debt is like credit card debt: it's high interest; it's not a huge issue if you use it wisely; and you have to track its usage and level against your total code/assets.
Like credit card debt, too many people turn around and they've got $20k of credit card debt or 20k LOC of tech debt.
I don't have too much of a problem with tech debt as long as: the benefit is clear; the debt is well understood; the payoff date/cost is planned. Just last night I was talking with a developer who works for me: "This way will require 5 queries instead of 1. Can you help me figure out this whacky ORM API to do it in 1?"; "Just do it as 5 right now and, as we learn more about the ORM API, we'll fix it later..." Tech debt: we get the product done faster; we've got a 1-5 line change to make (when we know how to make it); and we'll do so in a few months.
I love this analogy. Dealing with a minor set of updates ... that now need to be done in 60+ all slightly different codebases for this weekend. I'm using it in our next chat with the business side.
Yes, this is totally on the money. This is exactly what we want to see when assessing tech debt on an acquisition assessment (I do tech diligence for acquisitions). If you do it for good reasons, and you track it, and you proactively allocate resources on an ongoing basis or make tech debt remediation a first class road map item, it's not a problem, it might even be a good thing. If you wracked it up playing fast and loose and being all "go fast and break things", it could cost you the deal, or millions of dollars off the deal price.
I like to think of Technical Debt like an accountant or CFO would think about Financial Debt.
It provides leverage to incur debt at early stages, incur too much and it can really come back to bite, but a startup that incurs no debt will struggle to grow.
Interesting. I would say that John didn't build a solution that met all of the use cases that the business needed.
* To ensure accuracy of transactions, he should have built reporting to meet the needs of accountants.
* To support change, he should have kept his components separate and provided modest testing examples.
* To support developers and operations, he should have found and documented the external dependencies along with steps to verify that they are in place.
Whether product managers didn't uncover these requirements or the business didn't prioritize them, these aspects of the definition of "done" have more to do with thee environment the engineer is working in and less to do with his or her work.
There is bad technical debt and there is acceptable technical debt. The mark of a good engineer is knowing where to cut the corners - which you will always have to do.
Hardcoding something in a file rather than pulling it from a config or a database has a pretty good ROI in terms of time saved, and it's not that hard to change later, but tying yourself to, say, a kind of storage that you know is not going to work long-term, that will cost you later.
Also, sometimes doing MORE results in more technical debt. If you are going to spend your time building a Kubernetes cluster because "your home page has to be webscale", you are an idiot.
Discussion of technical debt always seems to devolve into an augment about what technical debt actually is.
Are there specific patterns of code behavior, metrics which can be derived from analyzing code, quality scores that are non-controversial, which can be used to define this problem? Things that can be measured and managed in a canonical way? How can some knowledge of experienced CTOs be distilled so that we can start to automate some of this wisdom?
I would say: technical debt is anything that makes the system more expensive to maintain (and grow) over time, and becomes more expensive to fix the longer you leave it. If it's just a bad or incomplete aspect of the system, that's not sufficient. It has to become more expensive over time.
So e.g. lack of functionality isn't tech debt, but a bad abstraction is usually tech debt.
There is a balancing point between under-analysis and over-analysis. It's hard to know the true requirements until the system is actually in production. You can't realistically pre-learn many domain lessons: you just have to dive in and see what floats by setting sail.
Finding the best balancing point is hard to turn into a sure-fire formula. I use experience (age) to make a best guess based on on past successes and failures and project types.
One tip is to collect and keep a list of questions and suggestions, and make sure they are sufficiently explored. You can't answer questions that have not been asked, but AT LEAST answer those that are asked. Make sure staff is not afraid to ask questions or make suggestions.
You have to release the software, with hardcoded inputs, limitations and bugs. That is fine. Having a clean code helps you then quickly polishing it and evolving it based on the customer needs. I prefer that to an over designed system, that is 80% clean and 20% crap, because that is hard to change as time goes by, and that 80% will stay the same, but will become 50% and then 10% of the total codebase.
Nobody agrees on what "clean code" is. I agree there are rules of thumb most will accept, such as try to be modular (group related things together), manage constants in a table or central class instead of hard-wire them, factor heavily repeated patterns into subroutines/methods/classes, comment at least the "big picture" of sections, and the oddities.
I started to use technical liability and technical asset in lieu of technical debt. Easier to explain and provides more insight into good engineering choices.
I like the metaphor technical tax, as it's the cruft that usually permanently sucks up x% of your development capacity. If it ratchets up permanently, after some time you're at a complete standstill.
Technical Debt is not a thing when you are coding a new feature. It’s like a stock future. It might be debt, but it might be something no one uses or gets dropped (or rolled into another feature). There’s a cost to speed that has to be considered. Maybe it’s debt, maybe you save the time and spend it on other more valuable tasks.
Does anyone have any examples of companies that died because of too much technical debt? For me its kind of one of those things I acknowledge exists but I try to avoid any projects where their goal is purely "decrease technical debt" or even when that goal is on the periphery. And I see that kind of thing way more frequently than you might think!
Not a company, but a project. It has not died yet, it is instead being rewritten while the old one is on life support.
The technical debt accrued due to:
1. Poor code practices (C++, poor use or understanding of pointers and threads created most of the actual errors in the program that make it unstable)
2. Excessive copy/paste (so bad decisions propagated quickly)
3. Poor architecture (didn't scale, performance is technically fine but development costs exploded as new capabilities were added)
4. Poor testing (almost entirely manual, as the system grew this grew with it to the point of creating major delays in releases)
They're rewriting it now keeping these lessons in mind. They're surviving because they're part of a larger org that can afford to keep them around for the rewrite, but essentially this now 10-12 year old system is dead. The other reason they're surviving is they happen to have only one competitor, whose product costs 10x as much per seat. So their customers want them to survive if they can hit the quality improvements in the rewrite, failing that they'll be disbanded.
I've become skeptical of people talking about technical debt and clean code and all the standard Fowler/Uncle Bob/PragProg-isms. Not because they're wrong. If anything they're all right, but in a rather vacuous, well-yeah way. Yes technical debt is bad. Yes we should start with a design. Yes we should do refactoring. But using the analogy of the author, that's like telling someone who's overweight and eating a lot of junk food that they should "eat healthier and exercise". Well yeah. They probably knew that already.
And so many people try to make it a dichotomy. Like that diagram of a project with a design versus without. That's not a fair comparison. What does a design even mean? Wireframes? A written document of features? How do we change this design? What's the way in which we get feedback? And what sort of projects are we talking here? A programming language is very different from a website. And what even counts as no design?
Some other notes:
- I'm not sure the author correctly used the Pareto principle. It's not just a generic "split things into 80/20" but a specific observation on how 20% of the causes result in 80% of the effects.
- The post could have used a quick proofread. There's quite a few spelling mistakes and poorly phrased sentences. I totally get if the author is not a native speaker, but they could easily enlist one to help (if the author wishes, I'd be happy to proofread—contact me).
- I get the whole "make this post fun with cats!" but honestly I'd prefer a straightforward example with no funny images. Maybe that's just my grumpiness.
Technical debt is like any debt. It's bad if you come due on it and don't have the resources to pay it off. Unlike cash debts, technical debt rarely comes due in a tangible fashion.
If getting to my next round vs. failing means leveraging technical debt, I'm going to do that by all means. It's not necessarily easier to fix tech debt at scale, but having more resources means it's less impactful on bandwidth.
This is my preferred way of thinking about it too. I wouldn't say debt in general is universally bad, and so I wouldn't say the same thing about technical debt either.
Technical debt can save a company by pushing something forward and launching a product sooner than a competitor. It can save a big customer by fixing a bug for them quickly that if not fixed immediately would have caused them to cancel a multimillion dollar contract. It can also eat away at a company if you let it accrue for decades and never pay it down.
Financial debt is okay if it gives you an important advantage now, and you can pay it off later, and you remember to actually do it.
If you get into debt so that you can have a nice new TV quickly, and you forget to pay it off, so it keeps increasing, and you only pay attention to it when the debt collectors start calling you... that is a problem.
In my experience, most software debt resembles the latter case. The goal is to meet a deadline sooner, and there is absolutely no intention of spending resources to fix it later, unless the customers start making too many tickets.
When a manager says "later" or "low priority", it usually means "never, unless we absolutely have to". Technical debt is always a "low priority" that will be addressed "later".
Yes, there are situations when meeting a specific deadline is critical, and it is a rational decision to cut some corners and fix things later. The problem is that after the deadline is met, the "fix things later" part is usually forgotten. A new deadline is set, and new corners need to be cut, etc.
I'm skeptical of any Uncle Bob-isms (and to some degree of Fowler's stuff) as well, but if you want a good description of technical debt and why it's actually a problem, Ousterhout does it nicely in his "A Philosophy of Software Design".
Yeah. The probably rarely is the debt itself, but the processes and circumstances that lead to the debt being accumulated in the first place.
Looking back and going "this is bad!" at your 'technical debt' isnt all that interesting. It's much more useful to understand what lead you (the team) to take those shortcuts in the first place.
I see an issue in the web industry that there is sometimes an economical incentive to maintain a solution with technical debt. Just ponder the idea that you hire a bunch of contractors to build you a house, and then you keep them on payroll to come back and fix every broken pipe that they installed to begin with. Even if they feel honor for their work the'll soon drift in to ugly insustainable work due to the nature of their incentives. We might need a proper system to hold engineers responsible insted of hiding behind a company, and in the long run be a solution that provides _space_ for engineers, plausibly.
Much of the discussion and worry about technical debt stems from a few fallacies programmers seem to fall into:
1. Some objectively "right" or "best" way to write software exists. We refer to "patterns" and "best practices" as if those had the force of science behind them, even though at the same time we can't, as a profession, agree on what we mean.
2. Code we (or someone else down the road) has to maintain wasn't written correctly or perfectly in the first place. Programmers tend to hate maintenance work.
3. We imagine that we could write the perfect program if only our managers and customers didn't impose time and budget constraints, or interfere with their stupid product and marketing directives.
4. Failing to write perfect code that lasts forever indicates a failure on the part of the programmer or the team.
5. Perfectionism (a symptom of obsessive-compulsive behavior) and the increasing worry among programmers about how others perceive and critique their code.
Writing perfect code that requires minimal maintenance in the future would require knowing that future, and all of the changes to requirements and constraints that will happen during the lifetime of the code. We can only work with the requirements we know, and those we can reasonably anticipate. Trying to code for requirements we don't have is usually called "overengineering," which means something else in other engineering contexts (as a few other commenters pointed out).
I have worked in software development for 40 years so I know that almost no one likes doing maintenance work, especially on someone else's code, and especially with languages and tools no longer in fashion. Maintenance programming is often given to new hires and junior programmers, while the senior developers get to write new code and play with the latest toys. This class division among programmers, often expressed in terms of what makes a programmer junior or senior, gets exacerbated by the personal quality of programming. Programming is a craft, not a science, not even engineering, but we forget that and try to express aesthetic preferences in terms of objective quality, even when we don't have agreed-upon ways to measure objective quality of code.
Most software has a short lifespan, which means the "technical debt" will disappear when the code gets rewritten or replaced by a SAAS product or the requirements change. Bridges are expected to stand for decades or even centuries. Jumbo jets have lifespans measured in decades. Very few software projects will stay in use that long. I routinely work on web sites that will stay online for less than a year, because they support short-term marketing goals. American companies have an average lifespan of 18 years, and startups are more likely to go out of business (or get acquired) than to stay in business for even a few years, so focusing on building the equivalent of the Great Pyramid in code at a startup or small company in order to avoid "technical debt" is almost always wasted effort.
Businesses factor maintenance into the lifetime cost of software just like they factor maintenance (and depreciation) into the cost of buying a fleet of trucks or a truckload of copying machines. It's the programmers who impose unrealistic goals of no/low maintenance and frame that inevitable maintenance work as a failure of the original design or implementation, i.e. technical debt.
We should try to do our best work, to make code that works (i.e. meets requirements and doesn't suffer from preventable bugs), and we should try to anticipate likely changes and make those easy for the next programmer. We should stop imagining a perfectly right way to write code, and stop thinking that maintenance work is beneath us, or a sign that the last team didn't know what they were doing.
I think programmers would enhance their value and learn a more realistic (and maybe humble) approach to their craft if they didn't think of progr...
There are many dimensions to technical debt, but the biggest one is the sociopolitical one: developers hate fixing each other's bad code, but the people who wrote it are often in a position where they can transfer the ownership of this code to someone else, either by being their manager or by quitting their job.
162 comments
[ 2.9 ms ] story [ 214 ms ] threadI do not describe that as technical debt. That are mostly lack of features. The code could be fantastic and it is just that to support different currencies, disable accounts or similar things new code is needed.
The lack of automated testing is more in line on what I think that it is technical debt.
Details aside it is an interesting read.
It doesn't make the system worse/more expensive as it grows, and it doesn't become more costly to "fix" the bigger the system gets. IMO those are the operative features of tech debt.
But testing is mainly important so a developer not familiar with the program will not break it by modifying something seemingly unrelated.
Lack of tests will absolutely bite you.
You are right, it depends on the kind of application and circumstances testing can be done in different ways. There is occasions were instead of automated testing there are other ways of working with quality. But, in this case automated testing is in the list of things that needs to be done. So, I think that to want automated testing and not have them is technical debt.
Aside from the initial increase in cost when you forget how the original code worked, tests are no more expensive to write in a year as they are today. Not significantly, at least. So I'd just class them as incomplete product - the effects are negative and can be expensive, but it doesn't dig you into a hole.
If we try to characterize the "perfect" example of technical debt:
- There is a spec, say: v1, of how the product must behave to the user.
- The product behaves exactly as defined in v1. I.e.: There are zero bugs.
- The very next iteration (v2) of this product has some cost to achieve.
- There is a counterfactual set of decisions that could have been made, which would make achieving v2 have the minimal possible cost.
Debt is the delta between the set of decisions made in implementing v1 vs the counterfactual set of decisions not made in implementing v1 that would have made the cost of implementing v2 minimal.
This definition has all the familiar mechanics of technical debt.
- You don't need to pay the debt to achieve v2 perfectly - it would just be more expensive.
- If all you ever want to do is v1, ie: there is no v2, then there is no technical debt, no matter how ugly the code is for v1.
- You can choose to reallocate costs between v1 and v2 freely. The distribution of costs will affect launch timelines, but will have no effect on the user-visible quality of either version.
It's dangerous not to understand the difference between bugs and technical debt. I lost interest in reading this article after it seemed like the author did not understand this basic difference.
Simple but under engineered systems are much easier to rewrite than to simplify over engineered ones.
The dept is that you take time and effort from your future to save time and effort in your present.
There was a separate testing client that was used for a lot of the demos. The demos on the 'real system' would fail so often that somebody came up with the idea to just take a lot of the code from the microservices and drop it straight alongside the testing client, as a monolith.
It was so much more stable that after becoming the default way to demo, it became the default way to use the tool.
5 years on and I believe it's still running that way. In this case microservices were massive over engineering.
They often are.
Well-meaning engineers and architects seem to reach for them while forgetting that microservices are about scaling your organization, not your software.
There are very few technical deficiencies in a monolith that are solved by microservices. Indeed, they typically bring technical hurdles: distributed systems are hard to reason through and IPC over the network introduces latency. And you need a very strong ops team to implement them without major headaches.
But where they really start to shine is when regression testing takes ~days and your development cycles are screeching to a halt because you've got too many in flight features and not enough runway to land them.
What takes real skill is recognizing which of your monoliths should be grown and which ones need to be sliced down the middle.
for instance one smart and experienced guy I knew said if there was an error, your program should just fail instead of giving a ton of messages and recovering.
and if you're just throwing code together, that might be what you do.
on the other end of the scale, a well written "second pass through everything with cleanup" might have tastefully written code, a few relevant comments and consistency.
and then I've seen a lot of code that appears to look like that, but is copy/paste garbage. (sort of the coding equivalent of a wordpress theme)
I would bet that he was right in that situation, but that he was also making a nuanced statement. Not all code should fail loudly, but when it should, "recovering" can sometimes be a distraction from a very real problem that needs to be addressed by the right person.
So fail early, but not too hard.
Logging is my go-to example for this sort of behaviour. People write stupid amounts of log statements in their code, so much that it’s hard to even read the code to understand what it does, in the hopes that it’ll make debugging easier. Use a damned debugger! Take a traffic dump! Use strace!
What’s more, it’s extremely rare that libraries provide logs themselves. So the actual complex parts of your application, like say the HTTP library or (God forbid) the TCP stack, can’t be debugged this way.
If you find yourself writing a bunch of statements like “DEBUG: updating balance from 1 to 2”, stop and write some tests instead.
I have encountered a fair amount of cases where that does not work.
- debugging a super large program ? loading gdb may take two minutes while recompiling to add a printf only 3-4 seconds.
- not an admin on the machine you are and the person with the admin account is not around ? sorry, you can't debug on macos (and likely in some linux distros)
- likewise, no traffic dump (and I'd assume no strace) if you don't have root access
I've got a program with log statements like that all over the place. Since stepping through it with a debugger would not even be possible. My IDE takes care of hiding the debug statements, since they're all encapsulated in different regions and if-def statements to log different types of things.
Flipping your argument, you’re basically saying that you can’t debug multithreaded code if the program doesn’t log.
But, I think we're depending on different programs in our workflow. I much prefer a logfile with statements that focus on what I want to inspect at the time.
I'm quite proficient with debuggers on both Windows and Linux, but I tend to use them less when dealing with my own multi-threaded code.
To be clear I’m not talking about adding some specific print statements to the code while debugging, I’m talking about keeping those statements in production code.
Of course, there’s cases where logging might be your only option. In that case: Go for it!
Relinking a super large program may take minutes, whereas loading gdb can get me a backtrace in 3-4 seconds! VS natvis files and other visualizers can hot-reload while I'm looking at a crash dump from production that took hours to trigger/gather/repro!
There are cases for logs - where sufficiently efficient conditional breakpoints are particularly painful to setup, for example - but the only time I've waited minutes for a debugger to respond has been with VS after major system updates as it refreshes a universe of symbols from the symbol server.
Do you have some really slow python scripts auto-loading in GDB or something?
well, we have definitely opposite experiences :) the main software I work on creates a ~1gb binary in debug mode and lld links that in a few seconds. Bud gdb and lldb, even with gdb-index, and all the optimizations I could find, both make startup slow enough that I can go for a coffee and it's not always finished loading when I'm back
It was only when I added proper logging to the critical junctures in the code that I could finally see very clearly how it behaves and see what input results in what code paths and output. I learned more in an afternoon than I did in a month.
https://quoteinvestigator.com/2012/04/28/shorter-letter/
There may be better information.
People are so afraid of making a bad decision that they refuse to make any decision at all, and then make a giant mess in the process. What you should do is spend your energy on finding reversible decisions, and then not spend a lot of effort on actually making them. We know where the paint store is, we know they can make up paint in 15 minutes, fuck it, paint it blue, we can always paint over it later. Hard no to black, though, since you can't paint over that shit.
People are so used to avoiding decisions that on a few occasions I've entirely flustered someone who wanted to tear into me (sometimes with an audience) but cutting them off and saying, "Yeah that was a mistake, and here's how we're going to fix it." None of them had any idea how to recover from someone saying "I was wrong" and going on to try to fix the problem. I still have a little video in my head of one guy's eyes bugging out when he realized what I just said.
When you start out, you just write code and don't think about how things interact with each other too much.
Then you learn that you're rewriting a lot of code, you overcompensate and start to over-engineer to avoid duplication and often to handle more scenarios than you need. You abstract out possibly too much.
Finally, you get to a phase where you realize a lot of abstractions that you've created aren't actually used and you're writing more "meta code" than you need. In this phase you learn to engineer just enough for what you need at the time and design things to be easily changed in the future. You trust that even if you're system design can't handle everything right now, you've designed options for yourself to expand as needed.
It's even more frustrating when you work with engineers who refuse to believe that any code they write could become technical debt in the future. These tend to be people who overcomplicate systems to anticipate future requirements
I don’t think I’ve ever anticipated a future requirement that did not ultimately turn out to be necessary.
Conversely, I’ve had a lot of people tell me something was not necessary only to find that, surprise, surprise, it was necessary after all.
To me, engineering is taking a problem and the corresponding constraints and building the solution that satisfies it with the least amount of resources.
"over engineering" in software is about trying to make something adaptable? Or maybe just satisfying someones sense of beauty.. Or maybe it's just about programmer convenience at the expense of customers/user experience and hardware resources..
It could also refer to building resilience to load before it is necessary. Example, distributed databases with failover and recovery capabilities for a side project.
Poke holes all you want, those are bad examples, but they should illustrate the point. Over engineering is a real phenomenon.
A lot of over engineering is due to getting this completely wrong: you prepare for something that never happens, at great cost and while underestimating the difficulty level. So you end up wasting endless amounts of time on stuff that has no business value.
On the other hand, a lot of under-engineering is due to not having a clue about what is obviously coming next and getting caught by surprise by completely obvious requirements.
I would rephrase this as make guesses based on probabilities and urgency estimates from past experience. You can't know the future but you have stats of the past. So a good engineer is one that has relevant experience and applies it appropriately.
Seems to me the default is obviously getting things wrong whether that be over or under engineering.
How would one go about reliably making choices that strike exactly the right balance consistently?
One thing I'm starting to really internalize is that to do so requires a deep understanding of software engineering as a domain, the domain and existing system, and the technical vectors in the business.
I leave breadcrumbs and openings in my code all the time so that the next time I'm in there, I'm set up, invited even, to ask the question I left unanswered before, and maybe do something about it. I leave the option of adding a feature, instead of building a config framework to support it and then defining one implementation. Which will end up not being the correct API when (if) I write the third one.
Sometimes people beat me to it, and get excited because they had an idea they're now invested in. I usually just let them have it, don't point out that I put the idea in their head. It's so infrequent people get that invested in functionality, you have to encourage it. There's too many things I'd like to do and never enough time anyway. They've just nominated themselves as a candidate for maintaining that module when I get tired of looking at it, freeing up my time for things nobody else cares about until it's done.
It's like if you were designing a house with plans to expand it later. You'd be very careful about certain decisions, like whether you should put a bedroom in the logical spot for an addition, because now that bedroom also has to be a thoroughfare. For 20% extra effort, you've extended the calcification threshold for that house/code by 80%.
Reactive frameworks indeed hide a lot of complexity but that does have positive impact on the user code. (IMO)
This is how I approach software, but there are so many people out there who won't approve simple code, because it doesn't have enough configs, or classes, or whatever. It seems that the person who wants the most over-engineered code tends to get their way. Psychologically the absence of 'things' is always inferior and harder to argue for than having more 'things'.
The post uses a lot of words to describe an understanding that experienced engineers and managers should have. Technical debt is a trade off. It exists in nearly all software of decent size as it's being developed. It is vital both for management and engineers alike to keep it at a reasonable amount, or you'll wake up one day with something that can't release new features, function, scale or be maintained.
It is also worth stating that tech debt can have little to no siginifcance to the success of a startup.
These days rewrites are popular. They make sense in some cases (when the project is smaller or introducing new framework). But in others where the original developer and business people have left and no one knows exactly how things work aside from they need to continue to do the same. In those cases fixing technical debt is extremely important.
A startup is more likely to fall into the above category if they ignore their technical debt because rewrites are hard and messy.
The wisdom comes from timing the investment so it’s not destroying the “mature” product due to entropy or strangling the new product due over engineering.
Startups do seem to have bigger issues than technical debt to deal with. Because relatively speaking, there are bigger mountains to climb early in the startup’s development.
This is not choosing an easy solution, this is doing just what needed instead of wasting time and efforts on something that may never actually be needed.
Now, the design should be sound and follow good practices. This is how to limit the rework needed when new features are added because it is impossible to design to cover all possible future features: Do the minimum but do it cleanly.
Yeah it can if you abuse it. It can also be incredibly helpful.
> John knew his code had a few bugs. But nearing the deadline, another project appeared and it took a lot of John’s time, so he couldn’t go back and fix the problems.
This is a problem, but this is not inherently technical debt. That's just delivering a bad/incomplete feature (Not necessarily John's fault if he's been asked to make changes). Code with Technical debt works or mostly works, but relies on non-maintainable or less ideal than patterns.
Simplified example. Adding a "Cancel" Button to a form. This button involves adding a new "secondary" class style to the button.
* Technical Debt - Button is in place. Works correctly. Instead of a <Button> component and tweaking it to handle the new "secondary" class. John simply copies and pastes the <Button> code directly into the implementation.
* Bug/Bad Feature - John references the button correctly, but fails to handle certain state resets properly when the form is cancelled.
-----
The prior is technical debt because it works correctly, but doesn't follow a good pattern or does something unexpected (refactoring that button will be much, much harder). The latter is likely just bad code.
The case where the latter is not bad code is if the company has made an intentional decision to not handle cancel edge cases. Even here, I would argue that is product debt - not technical debt.
----
Every software product in the world has debt. Just like every software product in the world has bugs. You need to account for your debt as you grow. Otherwise it will eat you.
The worst case scenario is probably partial paydowns, where the abstractions get changed but some of the non-aligned stuff remains and so that you end up with a jumble. Especially combined with inadequate documentation (or a culture of inadequate documentation, where people don't even look because they don't expect it to be there), the old practices can end up being cargo culted forward even when someone attempted to create a better way.
A couple of years ago I had to spend a considerable amount of time PoC'ing platforms and tools claiming to solve this issue and I remember there were virtually none that did this on-prem using established Office programs, without a four to five figure investment. The organisation in question ended up using password-protected Word files with forms (a royal pain). Before that, they had been using Excel files (apparently it's easier there to lock parts of the sheet you don't want users to mess with) but this was an even bigger pain as Excel still has this decade-old bug even in its latest version which distort graphics when row height is changed...).
Long story short, I'd be curious how you solved this and what software you used.
It ended up being a bullet point that needed to be talked about during sprint planning. It helped that designers were part of some of these meetings. If our company had been bigger it would have certainly been more challenging.
Mostly we tried to make it part of the culture that we stick to what’s in the style guide. And anything that deviates had a task of adding it to the style guide.
I have to say, that I wasn’t completely on board for a long time, since I was sure company culture would make it hard to succeed. We had a champion who got us about 90% of the way there before leaving. But there was enough momentum that another dev managed to bring together the right personalities and solidify the mechanics of dealing with the style guide that it was a no brainer to make it part of the work flow.
We used git to store the html and css examples. Made it trivial to copy. And since devs controlled the repo, devs controlled the pull requests as well, and required review just like other PR. It helped that the designers we had/hired were willing to deal with git, even if they had low tech knowledge.
I’m not at that job anymore, so I don’t know how long before entropy wins. It’s the type of thing you have to keep tending for it to continue to be useful. But enough process was in place that it will likely continue to be useful for them for several years at least.
It's not scalable, and produces severe consequences when used without care (which is most of the time, it seems).
But I strongly agree with your second part: bad abstractions are a much bigger source of technical debt. Abstractions that do not match the problem lead to convoluted solutions, reduced functionality and are very hard to break out of with incremental fixes.
But tbh, I'm skeptical of the premise to begin with. It's still easier to gradually consolidate 100 slightly different uses of a pattern than blindly create an abstraction that adequately covers those 100 uses. If your team is useless, it's better that their code be fragmented and atomic anyway, because big mistakes are inevitable. You don't want to make it hard to touch those mistakes. You want that code to be uncoupled & disposable.
But at that point the argument is getting pretty abstract. Depends on the specific codebase.
If it was a tactical failure by John, then we just call that Tuesday.
I have written software with tests and without. The latter start like prototypes and become quite useful over time. At first, everything looked good and there where just very few minor bugs which could be fixed easily. But over time, there appeared more edge cases which cannot be tracked down. As there are no tests, there are no specific definitions of what each part of the software is supposed to do. Yes, there is a big picture of how everything should work together, but obviously it doesn't catch every edge case. The problem is, that as a developer I can't act as effectively on my code anymore, because I took some shortcuts in the past.
On the other hand, there are other projects where I have test suits and where I can be totally confident, that everything has a well defined spec of what it should do and where I can't break something accidentally without noticing it.
So in the end it doesn't matter if your shortcuts have been made deliberately (most of the time it is a mix anyway). What matters is that due to the shortcuts you can't act as effectively on your code anymore.
I'm often in the middle. Where I have tests, they're not always (or rarely) the result of a "well defined spec". But... the tests do reference whatever the understanding of XYZ was at that time. That's often as well-defined as I can get. Clients will come back months later and say "this is broken". Well, no it's not. The tests work just fine. "No, that's wrong, it's not supposed to work that way". Well.. there's a test with notes indicating "do ABC then XYZ" and the test ensures that's working based on what was known/agreed on at that time. That's often not the same as "well defined" by a long shot.
The subject has been written about many times. This is a pretty good summarization of it.
Not called out: * Reckless debt is misfeasance on the part of developers. * "Prudent" debt is often avoidable- arbitrary deadlines are arbitrary. Management may believe that enterprise data models are as malleable as CSS. So "reckless" needs to be applicable to management as well.
I'm swimming in many years' accumulation of both reckless debt and reckless mgt. The deliberately considered debt I can handle.
<input type="reset" value="Cancel">
The point of incurring debt is to make a "large payment" (deliver software) quickly and immediately, in exchange for paying interest over time.
If you can deliver a working prototype, scale, get funded, and grow your business quickly in exchange of technical debt, that's not really a bad deal.
If I've run on Version X of some DB software for a while, and tomorrow Version X + 1 is released, yes, I should recognize that staying on Version X is tech debt. But that may absolutely be the right thing to do for now.
Like credit card debt, too many people turn around and they've got $20k of credit card debt or 20k LOC of tech debt.
I don't have too much of a problem with tech debt as long as: the benefit is clear; the debt is well understood; the payoff date/cost is planned. Just last night I was talking with a developer who works for me: "This way will require 5 queries instead of 1. Can you help me figure out this whacky ORM API to do it in 1?"; "Just do it as 5 right now and, as we learn more about the ORM API, we'll fix it later..." Tech debt: we get the product done faster; we've got a 1-5 line change to make (when we know how to make it); and we'll do so in a few months.
Ah, famous last words that I myself have also spoken from time to time.
s/for now/forever/g
It provides leverage to incur debt at early stages, incur too much and it can really come back to bite, but a startup that incurs no debt will struggle to grow.
* To ensure accuracy of transactions, he should have built reporting to meet the needs of accountants.
* To support change, he should have kept his components separate and provided modest testing examples.
* To support developers and operations, he should have found and documented the external dependencies along with steps to verify that they are in place.
Whether product managers didn't uncover these requirements or the business didn't prioritize them, these aspects of the definition of "done" have more to do with thee environment the engineer is working in and less to do with his or her work.
Hardcoding something in a file rather than pulling it from a config or a database has a pretty good ROI in terms of time saved, and it's not that hard to change later, but tying yourself to, say, a kind of storage that you know is not going to work long-term, that will cost you later.
Also, sometimes doing MORE results in more technical debt. If you are going to spend your time building a Kubernetes cluster because "your home page has to be webscale", you are an idiot.
Are there specific patterns of code behavior, metrics which can be derived from analyzing code, quality scores that are non-controversial, which can be used to define this problem? Things that can be measured and managed in a canonical way? How can some knowledge of experienced CTOs be distilled so that we can start to automate some of this wisdom?
So e.g. lack of functionality isn't tech debt, but a bad abstraction is usually tech debt.
Finding the best balancing point is hard to turn into a sure-fire formula. I use experience (age) to make a best guess based on on past successes and failures and project types.
One tip is to collect and keep a list of questions and suggestions, and make sure they are sufficiently explored. You can't answer questions that have not been asked, but AT LEAST answer those that are asked. Make sure staff is not afraid to ask questions or make suggestions.
If I am the one paying bills, would I really pay another developer for this refactoring? Yes: Technical debt it is. No: just busy work.
The technical debt accrued due to:
1. Poor code practices (C++, poor use or understanding of pointers and threads created most of the actual errors in the program that make it unstable)
2. Excessive copy/paste (so bad decisions propagated quickly)
3. Poor architecture (didn't scale, performance is technically fine but development costs exploded as new capabilities were added)
4. Poor testing (almost entirely manual, as the system grew this grew with it to the point of creating major delays in releases)
They're rewriting it now keeping these lessons in mind. They're surviving because they're part of a larger org that can afford to keep them around for the rewrite, but essentially this now 10-12 year old system is dead. The other reason they're surviving is they happen to have only one competitor, whose product costs 10x as much per seat. So their customers want them to survive if they can hit the quality improvements in the rewrite, failing that they'll be disbanded.
And so many people try to make it a dichotomy. Like that diagram of a project with a design versus without. That's not a fair comparison. What does a design even mean? Wireframes? A written document of features? How do we change this design? What's the way in which we get feedback? And what sort of projects are we talking here? A programming language is very different from a website. And what even counts as no design?
Some other notes:
Technical debt is like any debt. It's bad if you come due on it and don't have the resources to pay it off. Unlike cash debts, technical debt rarely comes due in a tangible fashion.
If getting to my next round vs. failing means leveraging technical debt, I'm going to do that by all means. It's not necessarily easier to fix tech debt at scale, but having more resources means it's less impactful on bandwidth.
Technical debt can save a company by pushing something forward and launching a product sooner than a competitor. It can save a big customer by fixing a bug for them quickly that if not fixed immediately would have caused them to cancel a multimillion dollar contract. It can also eat away at a company if you let it accrue for decades and never pay it down.
If you get into debt so that you can have a nice new TV quickly, and you forget to pay it off, so it keeps increasing, and you only pay attention to it when the debt collectors start calling you... that is a problem.
In my experience, most software debt resembles the latter case. The goal is to meet a deadline sooner, and there is absolutely no intention of spending resources to fix it later, unless the customers start making too many tickets.
When a manager says "later" or "low priority", it usually means "never, unless we absolutely have to". Technical debt is always a "low priority" that will be addressed "later".
Yes, there are situations when meeting a specific deadline is critical, and it is a rational decision to cut some corners and fix things later. The problem is that after the deadline is met, the "fix things later" part is usually forgotten. A new deadline is set, and new corners need to be cut, etc.
That book is a little gem, short and well-written
Looking back and going "this is bad!" at your 'technical debt' isnt all that interesting. It's much more useful to understand what lead you (the team) to take those shortcuts in the first place.
1. Some objectively "right" or "best" way to write software exists. We refer to "patterns" and "best practices" as if those had the force of science behind them, even though at the same time we can't, as a profession, agree on what we mean.
2. Code we (or someone else down the road) has to maintain wasn't written correctly or perfectly in the first place. Programmers tend to hate maintenance work.
3. We imagine that we could write the perfect program if only our managers and customers didn't impose time and budget constraints, or interfere with their stupid product and marketing directives.
4. Failing to write perfect code that lasts forever indicates a failure on the part of the programmer or the team.
5. Perfectionism (a symptom of obsessive-compulsive behavior) and the increasing worry among programmers about how others perceive and critique their code.
Writing perfect code that requires minimal maintenance in the future would require knowing that future, and all of the changes to requirements and constraints that will happen during the lifetime of the code. We can only work with the requirements we know, and those we can reasonably anticipate. Trying to code for requirements we don't have is usually called "overengineering," which means something else in other engineering contexts (as a few other commenters pointed out).
I have worked in software development for 40 years so I know that almost no one likes doing maintenance work, especially on someone else's code, and especially with languages and tools no longer in fashion. Maintenance programming is often given to new hires and junior programmers, while the senior developers get to write new code and play with the latest toys. This class division among programmers, often expressed in terms of what makes a programmer junior or senior, gets exacerbated by the personal quality of programming. Programming is a craft, not a science, not even engineering, but we forget that and try to express aesthetic preferences in terms of objective quality, even when we don't have agreed-upon ways to measure objective quality of code.
Most software has a short lifespan, which means the "technical debt" will disappear when the code gets rewritten or replaced by a SAAS product or the requirements change. Bridges are expected to stand for decades or even centuries. Jumbo jets have lifespans measured in decades. Very few software projects will stay in use that long. I routinely work on web sites that will stay online for less than a year, because they support short-term marketing goals. American companies have an average lifespan of 18 years, and startups are more likely to go out of business (or get acquired) than to stay in business for even a few years, so focusing on building the equivalent of the Great Pyramid in code at a startup or small company in order to avoid "technical debt" is almost always wasted effort.
Businesses factor maintenance into the lifetime cost of software just like they factor maintenance (and depreciation) into the cost of buying a fleet of trucks or a truckload of copying machines. It's the programmers who impose unrealistic goals of no/low maintenance and frame that inevitable maintenance work as a failure of the original design or implementation, i.e. technical debt.
We should try to do our best work, to make code that works (i.e. meets requirements and doesn't suffer from preventable bugs), and we should try to anticipate likely changes and make those easy for the next programmer. We should stop imagining a perfectly right way to write code, and stop thinking that maintenance work is beneath us, or a sign that the last team didn't know what they were doing.
I think programmers would enhance their value and learn a more realistic (and maybe humble) approach to their craft if they didn't think of progr...