Ask HN: As a CTO, what is your most frustrating problem with technical debt?

"And you know this rewrite, like others in the past, will be a significant sink of time, and likely a catastrophe."

To avoid this establish clear goals for the rewrite phase and measure results. Don't just rewrite for the sake of rewriting. Have a session where you identify issues and how to tackle them.

From a CTO-Like perspective, getting rid of technical debt should happen frequently, while keeping business goals in mind.

As a follow up, what are the most useful methods for managing technical debt without incurring too high of a cost to team/product/project velocity?

For the last 5 years I've worked primarily with startup CTOs and enterprise directors/VPs to build teams practices XP, lean product development, and user-centered design.

My advice to them, and my philosophy on this topic, is that technical debt is just that- debt. It's not a dirty four-letter word to be avoided, it's something to be managed the way you manage actual financial debt. There are some projects where the software you build will be self-contained (no dependencies), will require minimal changes in the future, or it may have a short shelf life (e.g. it's a stopgap before you cut over to a new system). In these scenarios, accruing technical debt isn't the worst thing in the world if it allows you to reap business value faster.

When you're building software that's going to live long-term, or that drives core parts of your business (and therefore will need to change frequently as your business changes), technical debt needs to be tackled early and often. It's ok to accrue technical debt in the short term ONLY if you make that decision consciously knowing that you'll need to address it later (e.g. I had a client that needed to release their software for a trade show, so we accrued tech debt and created tasks to track every piece of debt we'd want to tackle later, and then prioritized that work immediately after the trade show).

That said, it's also important to be clear about what constitutes tech debt vs what is over-optimization. Outside of libraries and frameworks, I don't really need the software to be "perfect". I need it to be clear enough, tested thoroughly enough, and easy enough to change to maintain a high velocity. If a part of the code has become spaghetti-like, or if we see code duplicated more than 3 times, or if we have disparate code that's too closely coupled to easily make changes, that's technical debt worth tackling.

The vast majority of organizations I've worked with typically think much shorter term, which is why they find themselves repeatedly accruing technical debt to the point where changes to the system occur at a snail's pace. It takes a lot of discipline, and coordination between IT and business, to practice what I'm describing, but that's the ideal that I strive for and advise my clients to strive for.

Not a CTO.

1. Not only legacy code eats resources that can be thrown on new developments. While legacy code is most troublesome, it still will be at maximum only twice as bad resource-wise, than new and pretty code.

2. As company grows, it may well be that a piece of software being used by by less than 5 people at the company, but require an own developer team to continue actively support it.

3. Maintenance resources vs resources spend on development is a never ending drama.

4. You should proactively think how to cull you internal projects, and be able to dictate to the rest of the company to stop using it.

5. Custom software for "business guys" - an easy way to multiply IT support budget. Hiring technically illiterate analysts, finance people, supply chain people costs a lot, a great great lot.

6. Last point. THERE SHOULD BE AN IRON WALL in between people making "money making code" and all other tech at the company. Can't stress this more.

The most frustrating part is explaining technical debt. From a business perspective you have a running software. You can sell it - it has all the features needed. Now you come along and tell me you need to fix something, that no client can see, that does not add any features and it might even lead to bugs. And you need two people and x weeks. I can imagine how "technical debt" is perceived as "snake oil" or "developer fun time" for management.

To avoid frustration it is best to fix technical debt in small steps, mix it in with others open issues. Devoting developer time for a big technical debt rewrite is hard to communicate.

I think a lot of the time when a developer shouts “technical debt” what they are really shouting is “code someone else wrote that I’d rather rewrite than understand”. (The rest of the time is the same but they’ve understood it enough to think it’s a disaster area.)

I have found it’s best to not take tech debt complaints very seriously and instead look at actual success metrics. For example if every change to a bit of code introduces new bugs then that might be a reason to tidy it up.

  > For instance they will insist that code that was just written fresh by a colleague is "tech debt" because they would personally have written it differently, when there are no meaningful differences between the two approaches.

  > They can claim that a design they don't understand the reasons for immediately (regardless of whether it's documented) is tech debt.

  > They can claim any feature they don't personally want to do or which is perceived as hard work will create tech debt. And so on.

As a dev, "or solve it with microservices altogether" hahaha. Good one!

Personally I find that the term tech debt gets overused. I've often heard it applied to relatively fresh projects for features/projects that just haven't been done yet, or to pitch the architecture of the day.

The times when I have seen a real breakdown where the term tech debt is legitimately used is when a scrum process breaks down deliverables below their smallest atom. E.g. different stories to write the code and unit tests, projects spread across devs such that no individual has ownership of any portion of the product etc.

In such situations we're not talking about debt as much as we're talking about junk.

I've been CTO (well, IT Director) for 3 years and I saw no particular problem with technical debt. I just handled it the same way I would handle financial debt. Repaying during good times and taking on debt in projects with high stakes and fixed deadlines.

It all depends on the industry as well. My CTO experience was in News and Media, where software is rewritten quite often and does not have any business value by itself.

I am now in a consulting position in the banking industry. Things are different there. None is willing to take on the risks associated with technical debt repayment. It accumulates endlessly. Once it reaches a point where no local developer would work on the project, the project is handed over to Asian sweatshops which support it indefinitely.

There are a few side-affects of technical debt that drive me nuts. These come from companies where developers are autonomous, self-choosing of time investment:

* continuous complaint without action (i.e. pretending to be a victim of the system you’ve built)

* lack of measurements for a system (technical debt or not, how is it working? Are you improving your measurement system after each failure to further understand the system?)

* unbuilt is better than unmaintained (this builds off the previous two — tell me what we learned from our previous attempts. If you didn’t understand the quality of the previous attempt, then how do we know if we actually improved?)

I’m down with fixing technical debt, just show me some metrics that prove that it was an improvement.

  * give the most pain AND
  * need to be frequently changed now and in the future

I think it is important to understand that technical debt is very difficult to define and quantify. Depending on your role, responsibility and skillset you will consider technical debt to be different things, and often attribute different (negative) impact of it.

Technical debt can cripple a development team, but I have also seen it being used as a continuous argument to invest into system development that really had zero positive business impact at the end. I am not saying to ignore it, but I also know that as a developer you tend to overvalue the things that are just in front of you.

I can't answer as a CTO-like person, but I'm in an Architect-type role and deal with our CTO-like person on a daily basis.

The high-level management cares only about "The Big Rewrite" because it is the only thing that gets visibility at that level of the organization and it comes with a large budget that everyone knows will be replenished for years as it gets spent. Our CTO-like person has fully bought into this idea.

We have too many projects and too few developers, so incurring technical debt is an everyday thing. It will be dealt with in "Phase 2" of the project, which all the developers know is something that never happens. From my vantage point - having one foot on the developer side and one foot on the management side, I see that the management thinks the developers are incompetent and the developers think the management is incompetent.

It's not a good position to be in, yet at the same time we have a pretty laid back culture, work less than 40 hours per week, great benefits, etc. So turnover is fairly low. Most of the developers have either a side gig or are going to school part time, or else they'd probably go insane.

The biggest problem is that Product and/or Sales are looking for constant growth via constant new features. It is impossible to get them to truly understand that by meeting some arbitrary deadline that you've had to take shortcuts. To their mind, done is done and now we need the next Y so they get their Q3 bonus.

The next problem is that too often the tech team wants to do a complete rewrite every 12 months because they want to use hindsight to remake last year's system/feature today. The issue is that in another 12 months this rewrite suffers from the same problem and now has several more things to consider. Some technical debt will always exist and I would argue always necessary. Perfect is the enemy of good enough and software that isn't live isn't adding value to the customers or company.

My perspective for this was to treat some percentage of work each quarter to resolving technical debt but do this in an agile prioritized way. Just like always focusing on the feature that will add the most value to the next release, also pick the biggest manageable issue that will reduce the debt the most to the next release. In this way, you know certain technical debt will never disappear but you have a ranked list to always decide the things that are most urgent and correct them before the house falls down. If your team is large enough, it is also helpful to have a person or standing task to update and document this list with WAG options and costs. Of course, as in all things, ymmv.

I'm one step below the CTO, but here's my perspective for what it is worth. It really depends on the stage of development your company is in. When you're first starting out, writing "good quality code" is about the worst thing you can invest in. The fact is, the code you're writing is almost certainly the wrong thing. A new startup is a hypothesis on a business model, and the first bit of code is to make the most basic tests of the hypothesis. It's VERY rare you get it right your first try. Writing good quality code is a long term investment, but if there's no one willing to buy it, it's a bad investment that will never have an ROI. On the other hand, if you're rewriting the system because now you know the market, and you know what it wants and you're scaling to meet it, now tech debt is bad. I think it's critical to understand the first system can and probably will be thrown away, so it makes little sense to build it to "scale".

I try to use an economic model for all software. All code is a loan against taken against entropy (a liability).

Best solution is no code or a simplification of the problem to existing solutions. This results in zero liability, while the revenue the solution generates is pure assets.

When a code liability does need to be incurred, consider both the amount of code written (the principal of the loan) and the architecture of the solution (the rate of compound interest). A good programmer will work to reduce the principal, an excellent one will work to reduce the rate of compound interest with a simpler architecture.

The architecture that is does more by adding complexity and conditionals / code contains a horribly high rate of compound interest, even if it seems easier to do at first.

Architectures which do more / everything by virtue of being simple may be more difficult to implement and understand (especially without some background knowledge) but can have a vanishingly small rate of interest. Double entry bookkeeping, graph theory, pipelines, normalisation/denormalization, map/reduce, unidirectional data flow etc.

For any serious business investment (not an experiment or proof of concept), architecture as rate of interest always trumps quantity of code as principal.

Order of priority is 1. Don’t take loans 2. If you must, design until you have a low and manageable rate of interest. Revisit often. 3. Reduce amount of code.

I've been the CTO at two tech startups. Here's how I think about it.

"Debt" is a good metaphor because by accumulating it, you get advantages right now in speed and ability to focus on other things, in exchange for having to pay it off later. You can either pay it off by stopping what you're doing (uncommon) and fixing stuff, or you can do lifetime installment payments of more complexity, bugs, etc.

Tech debt is just one aspect a CTO has to manage. Time to market is pretty damn important. Features are important. Many other things too. This is not to say that tech debt should be ignored, but elegance of the system is not the paramount concern. Just as startups borrow money and take financing to move quickly, so too a responsible amount of tech debt can be a good thing.

In engineering we tend to over-emphasize how important the technical qualities of the system are because that's what we see every day. There are a great many things that are also important, and various business scenarios where they become MORE important than tech debt. But if you're in the stage of trying to find product-market fit, you probably have bigger fish to fry.

The biggest tech debt I've struggled with always comes from the same patterned source. You think a system is intended to have a certain feature set for a certain user base, and so you make a set of architectural decisions to match. The world moves on quickly, you adapt, pivot, whatever you want to call it, and suddenly your carefully made architectural decisions are no longer correct or wise for your new scenario, but there is no time to go back and have a "do-over".

So in essence, failure to be able to tell the future, paired with a constant need to move forward, is the source of the problem. Both of those things are going to keep happening to startups forever.

I've been thinking how technical debt can be flagged / measured, and then managed. That is, a tool that can do a combination of:

Manual: Perhaps the person doing the code review can negotiate with the developer and agree that the code needs to be cleaned up at some point soon, but needs to be merged as is right now due to business needs, with some sensible score for the magnitude of the debt. When the debt does get paid later (stop laughing), the actual effort can be correlated with the debt score.

Semi-Automatic: The tool can take diffs from a pull request and show it to some other developers in the company and poll them on some subjective attributes like clarity etc. The score will be normalized based on other scores given by the person. Once in a while the app can even show nonsensical code in this poll to see how that is scored.

Automatic: 1. Flag the files that change frequently with issues, other pull requests, or 2. when there are long conversations and further commits to a file before a pull request affecting some files are merged (while accounting for some reviewers approving anything vs some reviewers nitpicking everything)... I have several more ideas like that.

Would you use a tool that does this? Do you already have something like that? Please feel free to email me your feedback, if you'd rather do that: techdebt@kirubakaran.com

Tech debt, whether creating it or paying it is a critical function of the business. Where this gets messy is when an engineering organization feels they own or somehow should prioritize the debt. This is especially true of paying it.

In every case that I've been involved in or had explicit oversight in creating tech debt has been when customer and revenue obligations are needed to be met. It's very rare in startups to have artificial or made up deadlines. I'm sure they exist at larger orgs, but as a startup executive, I can say with confidence that everything is constantly on the line, each day is critical, and so shipping product is paramount.

Of course, you have to pay that tech debt back at some point, but this is a business decision. It may turn out the business needs to kill that product or alter it in such a way that paying back debt is useless.

"And you know this rewrite, like others in the past, will be a significant sink of time, and likely a catastrophe."

This statement can only be true if you're using a very specific definition of "microservices". The first time I wrote about small services, in 2013 [1], I had not heard the phrase "microservices" so I used the phrase "an architecture of small apps". I was thinking of apps that talked to each via Redis, or HTTP, or didn't talk to each other and pushed everything into a database -- my idea was broad. Then in March of 2014, Martin Fowler wrote his essay on Microservices [2], which brought the phrase into widespread use. Soon after that there arose a number of startups that were promoting microservices, and they were promoting a very specific ideal: a system of apps that spoke to each other using HTTP, finding each other via auto-discovery, and offering data via a RESTful interface, with Oauth for authentication. A large number of people then began to use "microservices" for this specific set of interlocking ideas. That's fine, but it is a bit rigid. Something so comprehensive can demand a full re-write. But there is the more general idea, of slowly breaking up a monolith, and replacing small pieces with independent apps -- that does not demand a full re-write. That can be done slowly, and you have the option to keep the original monolith in a limited role, often to handle the frontend. Martin Fowler said all this in his essay, but some of the more extreme proponents of the idea have ignored the nuances that people like Fowler or myself have written about. I myself used the phrase "microservices" for a few years and now I've given up on it as too many people are associating with a very specific set of ideas, instead of the general idea of small apps cooperating with each other.

All the same, the idea can be powerful. Check out "Two months early. 300k under budget" [3]

[1] http://www.smashcompany.com/technology/an-architecture-of-sm...

[2] https://martinfowler.com/articles/microservices.html

[3] http://thoughtworks.github.io/p2/issue09/two-months-early/

I find that it doesn't get too frustrating if you don't let it get out of hand.

Routinely prioritize keeping dependencies up to date, clearly depreciate "old" / bad code, and focus on coming up with good product designs that have reasonable compromises between time-to-market and high quality engineering.

You just can't let stuff slip. Don't let 6 months go by without at least trying to upgrade to the new version of whatever or without making some some progress moving from the "old way" to the "new way" for some component.

Refactor everything all the time, duh

One big issue I have (and I'm trying various solutions around it) is code upgrade.

Let's say we are doing something one way - step 0, and we realize "hey, guess what' there's a better way" - let's call it step 1). Now going forward all new code is step 1, and touches to old code is step 0. Later you realize - there's an even better way - let's call this step 2. Now going forward you have step 2, step 1 and step 0. This process continues - but people who worked on this leave - leading to technical debt. This carries on for a few iterations and now you are on step 50 (49, 48, ...).

I have no idea how to deal with this problem.

I personally feel more micro-services would be better - but I also don't want to jump the gun since we were not too big a company at that point (and being profitable with our runaway took all priority)

As a senior sysadmin who has frequently been in the role of CTO because businesses often don't even have a CTO... my number one pain was always when the technical debt requires more personnel in order to manage properly but upper-management refuses to hire more people for the role. I know your question seems more oriented towards the SV-startup/dev cycle, so my answer is more for general business (even ones without programmers).

My pet peeve though has always been structured cabling. I don't know if yall understand just how bad the cabling situation is is many businesses, but it's bad and for some reason they really don't like spending money on that particular thing even though it's one of the most important things for keeping a business up with the times (and the data rates that match).

So, in general, it has been a frustration with the communication gap between the technical team and the C-level and the board. Too often you either get a CTO who wants to "program with you" or one who is too much MBA and not enough tech.

As a sysadmin I have realized my number one failure was not working on my MBA-side more and thinking purely technical proposals and solutions would win the day. AKA I should have spent more time golfing and going to lunch so I had more power with the Cs.

Technical debt is inevitable. If you ignore it, it becomes worse. I've never been on a project without technical debt.

Of course not every little problem is technical debt. There are plenty of nice to haves, we could do X, etc. style improvements that aren't really that critical. My advice for those: just apply the boy-scout rule and improve things when you work on them. For the bigger stuff, somebody needs to step up and take decisions. Ultimately for big chunks of work related to technical debt, that's a CTO decision that involves taking into account the interests of both tech and business. A CTO must have the power to do this and be wise enough to use that power responsibly.

Stuff to watch out for as a CTO is when seemingly simple stories snowball into a lot of firefighting, bugs, or deployment issues. That's usually a good sign something is wrong. If that happens a lot, you are definitely experiencing technical debt. Firefighting eats away development budget. If your team spends half their sprint diagnosing weird issues instead of getting stuff done, cleaning that stuff up is probably time well spent.

I don't have much frustration with tech debt itself. It's a normal, acknowledged, often understood and predictable side effect of moving fast OR weighing tradeoffs.

I actually get pretty annoyed when it's conflated with product debt. I don't know if that has a real definition, but for me that's where your market/audience has grown to need things you did not envision/never built. This is often labeled as tech debt, even though it has nothing to do with your coders. This is an organizational issue, and one I haven't fully conquered yet.

Tech debt should not be accruing every week, it should be a conscious decision. MVP's are riddled with tech debt, intentionally. You want to get the thing out there and start gathering feedback/information, you acknowledge you don't know the right thing to build, and will do so once you have more info.

I'm also annoyed at the junior dev that doesn't take time to understand a system, and shouts that it needs to be re-written. Most times that's just because the dev is junior, or lazy. It's actual tech debt when the system is too convoluted to understand, or you haven't documented.

I have no idea what "sprint commitment" or "manager's power" are supposed to mean. They both sounds kinda sick.

If you know a rewrite is going to be a time sink and catastrophe, why on earth would you do it? Also sounds sick.

Your org should be focused on business results. If you're focused on optics, things are not healthy.

You are not in a race with your competitors unless you're in a market that is itself racing to a commodity. This is a business strategy problem vs an engineering one.

Technical debt seems far less a problem to me if you have a moderately capable software engineering team. It happens, yeah. But as your experience grows you know the proper balance of paying it down vs new work. It comes down to measuring quality of your services from an end user perspective vs how much work on engineering and support teams to maintain the desired quality.

~7-8 years cto'in