Ask HN: How often does technical debt lead to the failure of a startup?
The question here comes from my concern of how some startups can actually survive with high technical debt with customers? Wouldn't that lead to their failure as a product/service sooner or later?
I work as a web architect contracting with startups on smaller complicated pieces of technologies. These days lots of startups I talk with have a very high technical debt in their product(s) and most of them don't seem to care.
My first advice to them is that things like that can kill a product, but I've had some strange feedback after passing on that advice.
So I wonder if technical debt does actually have pitfalls along the way? (Bootstrapping yeah I agree, but not when you have 8 engineers on the team)
55 comments
[ 2.7 ms ] story [ 111 ms ] threadAlso, I would say to much concern over technical debt can cause failure. I personally lost what I conservatively would say was 30% of our net revenue last year letting my prior lead dev address technical debt that never benefitted a single client and that in areas we have now replaced (or are in process of replacing) entirely in the product. I am as much (if not more so) to blame as he was because I approved the time.
Though, I'm really curious, mind if I ask what sorts of technical debt you had in your startup that led to such massive slow-down in the process?
Were the debts among any of these:
* Documentations for on boarding devs. * Lack of tests. * Delayed refactoring. * Lack of alignment to standards.
Also, almost everything you mentioned is on my list of WTF was I thinking. We wasted time adjusting code to meet new standards, adding tests to things we knew we were changing, adding scalability to places we had no scalability issues with (yet), adding new interfaces/code in places where something "might" happen etc. None of these changes benefitted a client directly, so IMO we lost money on every one of those initiatives. The reality is a lot of the code we "brought up to standards" was rewritten anyway when we started adding features and refactoring the old code to interface these features. I also allowed the team to create their own modules instead of extending some existing node.js modules that existed in the community, which to me was a waste when I saw what we did. We would have done better to extend the community modules and open source them back to the community.
As for documentation, honestly, we have pretty good documentation, although I wouldn't say it is without issues.
Overall I considered last year, the year of the "it might happen" screw ups. What really pissed me off is I spend a huge amount of time advising clients not to touch old code unless they know it is causing them a problem now or can be proven to cause an issue when testing against a new requirement. Yet, I still failed to follow my own rules, shit happens.
Yeah sometimes making these sorts of decisions could easily lead to big pitfalls on the way.
Regarding this part:
> "I also allowed the team to create their own modules instead of extending some existing node.js modules that existed in the community, which to me was a waste when I saw what we did."
This is technical debt as well by the definition of Wikipedia and some 101 CS books. Having enough contribution from a community could always be better of being used, unless that part of the technology is propriety in the product.
I totally agree I screwed the pooch and having 20 years of experience didn't save me. Ironically I had made sure we were giving back to the community but missed some opportunities that would have saved us time and money. Oh well, this year we will add the code to the modules we use. It is on our to do so we remove unnecessary dependencies we added.
IMO, if technical debt is what destroys your startup then you probably so poorly understood the customer problem early on that you engineered a solution that was going to fail either way. In all the companies I help I almost always see bad technical decisions, but their marketing is what is hurting them the most not their tech debt.
Another example is Instagram, I read recently (but can't find the link) that the founders were learning to code when they built it, and that some of that code is still in production, and apparently, it's embarassingly bad, but it gets the job done.
On the other hand, I worked for a consulting firm and one of our clients sites was absolutely riddled with technical debt. The result was that any changes they wanted to make were significantly more expensive than if they would have been on a better architected system. The thing is, it did't matter to them because the cost of re-creating the system was significantly more than the cost of working with what they had.
I hate technical debt, but have come to realise that it is a debt much like any other, and if you can cover the cost of that debt, it doesn't really matter down the line.
This isn't an excuse for bad programming, I consider that to be something slightly different from technical debt which often works but is not an idea implementation.
Thought in the context that I mentioned, it is about some startups that have surpassed the point to find their first thousands of customer and now they need reliability.
But again there are different interesting tradeoffs of these issues.
Regarding:
> Another example is Instagram, I read recently (but can't find the link) that the founders were learning to code when they built it, and that some of that code is still in production, and apparently, it's embarassingly bad, but it gets the job done.
I was at the OpenAir conference this year where Mike Krieger mentioned the above story. But I'm not sure if the Video is out.
I have never been at a startup that failed for technical/technology reasons. All failures I've seen were due to issue with delivering what customers were interested in paying money for. Most often, product owners and developers were not able to establish good communication, which led to developers basically creating a product that didn't match what customers wanted. Period.
I would be interested in your definition of technical debt though. What some people call "technical debt" or "hack", I often call "the best/fastest way to address the current business concern and deliver immediate value". In a startup environment I think these is a critical workflow.
Thanks a bunch for pointing this out! It's important for me to hear this here :).
Regarding my definition of technical debt is this:
* Lack for proper documentation where on boarding devs is becoming a burden.
* Lack of written tests to a point writing new features is almost inevitable from breaking the product.
* Delayed refactoring which has led to serious performance issues.
* Lack of alignment to standards where the engineers have written everything in Javascript like a 5 years old painting all the wall with same color, rather than using a different language for some specific purposes.
Some other Technical Debts that I meant were:
* How engineers go on a trajectory of writing code where no one knows something new have been added.
* Lack of clarity who have done what, regardless Code commits, this one is about knowing who lead a milestone or a feature to production.
When it came time for a round of financing, I was leading a team meeting when the subject of dilution came up: "Are we being screwed?" And I had to explain that a smaller percentage of a company with a higher valuation isn't "being screwed." and attendant to that we discussed cost of capital, classes of shares, preferences, ratchets, etc. You could see the light bulbs coming on as I explained.
Engineers are smart. If you explain to them that "good enough with the money we've got" will make them more money later, they are usually OK with deferring both money and correctness gratification.
Think of it as building empathy for "product owners" who might otherwise be dismissed as beancounters.
> 'Think of it as building empathy for "product owners" who might otherwise be dismissed as beancounters.'
That's true. Some engineers won't take the "just make it good enough" approach, but I believe most will - otherwise they don't belong in a startup.
I wasn't mentioning the broken communication between PO and devs in this regard though. I have seen teams where the PO were not able to describe what they wanted, what business value they wanted to add to the product, devs had to come up with their own description of feature and sure enough the market didn't care of that.
PO is such a critical position, and such a difficult one to fill. I can count on one hand the companies where I worked who understood that. A PO is not just moving tickets around in JIRA, they are in charge of describing in actionable terms the business value the team is trying to produce.
Good luck.
More often it's not "technical debt" but outright failed implementation when technology kills a startup
See how far it takes you and if you are wise enough not to press your luck, you'll have the chance to do it right with cheaper money.
My first startup that I built and sold had 0 test written, but I knew on spot what I'm giving up for what. I always say that last line of yours:
Making "the chance to do it right with cheaper money." rather than a whole team to re-do everything from scratch or break down already running product.
Sort of goes back to your point with cost of capital, indeed.
Thanks, good points :)!
Though, much of this was exacerbated my management's insistence on implementing a myriad of side jobs to bring revenue while deliberately ignoring the main part of the website being unresponsive.
http://highscalability.com/friendster-lost-lead-because-fail...
> "technical difficulties proved too pedestrian for a board of this pedigree. The performance problems would come up, but the board devoted most of its time to talking about potential competitors and new features, such as the possibility of adding Internet phone services, or so-called voice over Internet protocol, or VoIP, to the site."
Anything that delays shipping is going to contribute to the failure of a startup.
Drawing some conclusions to what technical debt could eventually lead to at the end.
> "Anything that delays shipping is going to contribute to the failure of a startup."
Edit: forgot a word.
When you get X customers, you'll have to commit Y resources to fix this one thing.
Taking on technical debt is a smart thing in early stages because it's essentially low-cost, non-dilutive financing. So you should recognize that very real benefit.
However, if they are young startups they might not even realize they are doing this smart thing. You can add value by telling them are smart and giving them the schedule of technical interest payments they will have to pay one way or the other if they are still in business.
>"the schedule of technical interest payments they will have to pay one way or the other if they are still in business."
Another reason technical debt is useful, as touched on elsewhere, is that it's something of a real option.
Options have value - think e.g. Oil futures which are the right but not the obligation to purchase X at Y price in the future.
Real options also have value - certain investment decisions can be delayed (e.g. building a plant or refactoring code) and by delaying them you create an option. Here the option is the right but not obligation to write better code that will actually work :)
I know it's hard to convince clients to change their ways, I have found that telling them they are doing great things they didn't even realize is a good way to start before you "bring the pain". Best of luck!
My definition of technical debt would be "problems in the code base that make it harder than it should be to do work," whether that work is adding new features, fixing bugs, scaling up, etc. The startup that can implement new features quickly because their code base isn't a disaster is more likely to succeed. They can try out new features quickly, seeing what customers respond to.
Customers need things like uptime, minimal bugginess, quick bug fixes, etc. Technical debt reduces uptime, increases bugginess, and makes it harder to fix bugs.
I think it would be hard to point at a startup and say "technical debt killed it". In most cases I imagine there are many inter-related causes of failure, but an inability to quickly add new features, fix bugs, and keep the system stable can't be a good thing!
Though someone else here mentioned how friendster lost their game due to technical debt:
I looked it up and it looks like that indeed it was an issue.
http://highscalability.com/friendster-lost-lead-because-fail...
> "Technical debt reduces uptime, increases bugginess, and makes it harder to fix bugs."
Yes, that is one of my concerns usually. It's good to go fast and build more, but not to an extend where it slows the process down on longer run.
The comments are interesting, but the closing one by a Mark Peterson again goes to the bottom line of unaddressed technical debt by putting a date on the best known general technical problem:
Friendster failed in the summer of 2003 when it never took less than a minute to log into the site. Where it never took less than a minute to visit any page. Even pages that should/could have been static.
Debate about any missing feature after summer 2003 is less than moot.
Especially since added features or allowing viewing by the not logged in would tend to further slow down the site.
You've basically elevated these items to the top of the list, above all other attributes, and declared them more important to all customers.
Customers will pay/use a product that delivers value. The items you listed subtract from that value but they may or may not eliminate it. It depends on the product, and how valuable those items are to the customers.
Edit: I'll add an example. Let's say you like video games. There's a particular game that you love to play. Do you love that game because "it's stable, has minimal bugginess, and the developers release patches on time"? Probably not. You love it because of the gameplay/storyline/graphics/whatever. If the game is unstable, that's irritating, and makes it less enjoyable.. but that doesn't change what you love (value) about the game.
No I didn't. I'm just saying that these things are also important to customers. Churning out tons of broken features on top of a constantly crashing/unavailable system is not a winning formula for success in many markets.
It might work to some degree for free consumer-facing apps (like Twitter), but for many other sectors reliability and data integrity are very important factors (like any B2B product).
Definition of need:
a situation in which someone or something must do or have something
Saying "these things are also important" and that it depends on the type (twitter vs b2b) of product is a lesser statement than your original statement. If you want to tone down what you originally said that's fine.
For example an API that users pay for would only be updated or relied on when it's debt free.
https://youtu.be/pqeJFYwnkjE
Essentially how well you can write a program to reflect your understanding of a problem. Though it has come to more generally mean writing crap in a lot of cases.
Along the way to delivering that value, sometimes a less than perfect solution is chosen (often because of resources--time, money, or lack of human resources) OR the perfect solution is chosen, but the product changes and that solution no longer is perfect.
Technical debt should be managed, but it can't be eliminated... unless you want to give up producing anything new, and just work on polishing your previous work forever.
When will it kill your startup? If it turns out that your product is NOT what your customer actually wants, but the shortcuts (tech debt) you took prevent you from making the necessary adjustments to meet your customers needs.
But I think at some point, this might become very difficult due to the technical debt when the startup can't deliver more value than it promised.
1) A competitive issue.. a company will less tech debt will be able to improve their product faster. That is a risk, but it depends on the competitive landscape how much risk that amounts to.
2) When you say "startup can't deliver more value than it promised".. this is more a variation on my original point. The issue isn't "delivering more value", but that the value that has been delivered is insufficient to meet customer needs.
Above that, "delivering more value" is a growth issue (your ability to meet the needs of additional segments of the market). And it's certainly true that startups with less tech debt can improve their product faster -- ie: grow faster.
A potential acquisition came our way that seemed to be exactly what we were looking for. Customer traction and revenues, the features exactly what we were looking to overhaul. There was very high internal interest in acquiring the technology and the company.
The need for this set of features was so high and the internal development estimated to be so costly (in developer time and delays in other projects) we were willing to overlook that this app was not built on our primary stack and only a few of our developers were familiar with. The language choice made by the company didn’t cool our appetite. However, technological choices made us pass on the acquisition.
They didn’t use a framework.
This means that our developers would have a very long learning curve. We also saw a lot of code that did what a framework would have taken care off and this means that we would have had to learn, maintain and expand the that code instead of working on the revenue generating features.
We found close coupling of code all over the place. This meant we couldn’t quickly extend and modify the features as we wanted without first paying back the technical debt accumulated by the developers.
It wouldn’t have mattered which framework they would have chosen as most of them have good documentation and force some sort of standard development practices. However, we couldn’t take a chance that all the behind the scenes stuff would need a rewrite if we wanted to expand and scale the platform. We passed.
Folks rarely notice how much a decision making matters on this regard and technical knowledge matters a lot on the way. Since having a company* acquired sometimes could be a good business return, but in the first place due to hitting the pedal it has brought lots of technical debt which makes an acquisition impossible.
Was the acquisition price in the 100k, 1M, 10M, 100M range?
Was the cost to replace/rewrite the systems in developer-years estimated in the single-digits, 10s, or 100s?
It was a monumental effort that took more than a year to complete, and we did the "brain transplant" (as we called it) at the same time we were moving to a new data center. Launch was shaky but overall it went very well. One of my fondest memories is of the "war room" on launch day as we triaged incoming bug reports. The team worked great together and at the end of the day we felt like we'd really accomplish something.
Sometimes of stress can be very rewarding.
To figure out which ones do, I sometimes think about the correctness vs. completeness spectrum.
Correctness is how well the architecture fits the problem, how well edge cases are handled, if it's a service then whether it's built to scale horizontally, etc.
Completeness is how shippable the thing is. Are the important features there, are the bugs not too severe, could the thing live in production without causing severe harm, etc.
Some problem domains live only at the far end of the correctness spectrum, like software that life depends on, flight control, nuclear reactors, etc.
Other domains might be all the way on the completeness side, like the throw-away prototypes, temporary stand-alone marketing sites, etc. If it works at all, you can ship it. Problems there don't count as technical debt.
Obviously most projects are somewhere in the middle.
Most high-growth companies can get away with erring a bit toward the completeness side, and ignoring correctness for many things, because a lot of stuff will be re-written every year or so to handle the changes in the business brought on by customer growth. Since it will be re-written anyway, it might not be useful to think about the lack of correctness as technical debt because you won't ever pay the debt. The real trick is figuring out which things must be perfect now and which can be improved later.
Getting that wrong either way could lead to the failure of your startup. Settle for nothing but perfectly correct architecture for everything, and it might take too long to ship. On the other hand, ignoring severe problems in a critical system might somehow limit growth, cause a legal problem, etc., and eventually sink the company.
> "correctness vs. completeness spectrum."
and yes! It's a double edge sword:
>"Getting that wrong either way could lead to the failure of your startup."
Trade off an extra day to do each release versus 30 days to fix the release process? Trade off an extra week to add a feature versus a month to refactor and a month of customer turmoil on the refactored release? Trade off a month of getting complete test coverage versus saving two days on a future feature development?
My guess would be that the startup founders are doing a better investment calculation than it might appear to you.
Though I was just pointing that out for over time. One time around the tech-debt is not that big of a deal, until it becomes a beast and hard to overcome.
It may be true that technical debt directly kills some products, but if it's bad enough that you can't get certain things done, even if what you do instead is also important to the business, the debt is a sign of bad engineering management (either in the past or the present). And bad engineering management will kill the product.