Ask HN: For a startup, when should I begin writing tests?
We're building our MVP and have got commitments from our first few customers. I'm still young and haven't managed production software before, only personal projects.
I'm hesitant to sacrifice developer velocity for tests. I believe product agility is a huge asset early on, and I think testing cuts into this.
I'm also afraid of digging an inescapable hole of untestable spaghetti code and tech debt.
I'd love to hear the thoughts of more experienced engineers and founders on this issue! How did you balance it, and what would you have done differently? When should I start testing?
My current idea is to skip testing, and instead put that effort into product development and a robust deployment pipeline with easy roll-backs. Ideally this would allow us to move quickly and revert any big mistakes, without the burden of a complex test suite.
If it's important, we're React+Go, but I'm not looking for stack-specific advice.
39 comments
[ 3.3 ms ] story [ 85.3 ms ] threadI would write tests from the get go. I would have to look at the got history to be sure, but I believe I started adding to the test suite after about a month of adding code (I did use an existing open source project as the base of the product, and they had a lot of tests written). Your fears about spaghetti code are valid and tests will help with that. At the least you can refactor with less fear, though with golang that will be less of an issue than it was with ruby. In addition they will help prevent regressions. When I (or, far more likely, our users) found a bug in the system, I would write a test before fixing it. Therefore, we had very few bugs return.
However, a deployment pipeline is critical too, because otherwise your tests won't get run often enough (especially if they are slow. Due to the way I wrote the tests, and the number of them, it took approximately 20 minutes to run through all the tests in our ci environment).
Re: your fears of slowing things down. In my experience a test suite, even a bad one, will speed things up once you are in production. This is because it will prevent regressions and doing manual testing, both of which chew up additional time. I saw this at other companies.
- Don't spend time in writing unit tests until
- You have paying customers and a steady revenue
These don't take too much time to write and speed up your development significantly because you can make significant changes and be sure that certain flows still work without doing all the manual testing over and over.
Only if your code is complex, will you need a complex test suite. If your code is complex you won't be fast implementing features anyways. TDD will help you keep your code simple.
Tests are about system integrity. If you risk losing something due to lack of integrity, tests are worth your while.
Think of it like health insurance: When are you comfortable forgoing health insurance, and when are you not?
Unit tests are important tests and are very useful for engineers. As others mentioned, thinking about test-ability keeps code simple, which is a must when you want to move fast. Test-ability (have the TDD mindset and thinking, if you don't then apply TDD until you think tests while writing your code) usually translates to readable and maintainable code. When you write good unit tests, then most assumptions are documented in an automated way and tested at every execution of the unit tests. When a unit test fails, then the error message and the test name should tell you precisely the assumption and what is broken, that ultimately makes you really fast in maintaining and (with a good overall design) changing code.
When you want to be really fast then have fewer or no integration tests. That is because integration tests usually span over several parts of your software. So a change in any part will lead to probably several changes in the corresponding integration tests. That makes it more expensive to change things fast. On top of that, many integration tests are more complex to write because of the work in mocking adjacent parts of the software.
System/Black box tests. Those test are very important for evaluating business value. One tests common happy paths from your customers and validate that business value/ feature x still works. Again like with the system tests, the process of doing, deciding and prioritising will lead to clear business values and very important a better shared vision between the developers and business. The software serves the purpose to be used by a customer, the system tests are ultimately a distilled version of what you the business is selling and it's an automated process to ensure that this value is delivered.
Even the best software developers do mistakes and forget. I have not yet seen a team which profits from not writing tests. It takes some time to write good software -> testing and approaches like TDD are some of the ways to start climbing that hill.
About your questions:
A robust deployment pipeline is the right way to go and can be even better leveraged with tests. Example: you can connect your system tests to the roll-back functionality and re-deploy the last working state.
You are right about your worries about Spaghetti code and technical dept, from my experience they are always an issue from early on. But also keep in mind that no one is perfect so even if it's only you writing code, you will find code written from you, which gives you goose bumps. A nice approach to code quality is the in-place refactoring, that is you don't do re-factoring as a task. Each time you touch code and it's not very clear or the structure/ design is not ideal then you re-factor with each and every task (piece by piece). A rule of thumb: When a 1 day tasks adds 3 days of re-factoring (only fixing the touched code), then you know that you are sliding the slope of lower productivity and you need to act and going back up the slope usually takes a lot of discipline in the team (that's an extreme example, personally I act much earlier).
Your current idea has some risks associated with it. I like to expand with an example: Usually a code base changes and grows with new features, rarely it gets smaller. When your code base grows and your team grows -> productivity declines (people: communication overhead, technology: overall more complex/bigger system, probably many more factors). In the first weeks you might deliver very fast without tests but in the future that is not given. Let's assume that you and your team are unicorns and will produce perfect code all the time. Your productivity declines naturally by having a bigger system and more people. In the case of declining productivity it is very unu...
I disagree that testing cuts into agility.
I believe that in this case "agility" means that you can change your product in significant ways quickly. Presumably, when you make these changes, you need it to still work. Having tests already in existence will inform you when you've broken something you didn't mean to change. If you don't have tests that can run frequently on your swiftly changing, rapidly pivoting codebase, you will break things and you won't even know until it takes off someone's foot.
If you don't have these tests, you can't rapidly change your product with confidence that it still works, and I suspect that you would value "working" above "not working" very very highly.
If you want agility, you need tests. Lots of automated tests. Testing doesn't cut into your agility; testing gives you agility.
Unfortunately, inexperience is going to make it take longer to write tests. That probably won't be what kills your startup though. Most companies die when they try to scale due to a variety of organizational issues, and having a few tests kicking around when you start bringing new developers in will probably help you a bit in this respect.
Your point is valid. Personally I would not join a company, as a software developer for software without tests. I would consider it if they hire me to lead their code improvement efforts, but even then no tests reflect engineering and business culture in that company (which is very hard to change). A big point I like to add is that new developers make more mistakes within a system they don't know. Having a good test suite gives young and experienced developers faster feedback about their assumptions of the system. Also unit tests act like an always up to date documentation which a developer can read to understand important assumptions about the parts of a system.
Experienced developers (the ones who made it) are generally fast, quick to identify the source of an issue or where to make a change. They have better than even odds of not mucking it up (discovered by users later because testing in-house is too poor to catch it).
New developers are either paralyzed by uncertainty (will this really do what I want, does it have an unintended effect elsewhere that I can't detect?) or make changes and have better than even odds (much better) of mucking it up and creating a mess. Hopefully it's discovered before release, but most likely it's discovered after.
One of the critical things for new developers to a project is the opportunity to make changes and see what happens, with good feedback. If you permit them to make the changes, but lack the ability to provide good feedback on the effect of those changes, then they cannot learn the system from the perspective of a developer. Along with exploratory testing, it should be possible for developers to make compilable, but arbitrary, changes and see tests fail. This lets them know the impact of a change and understand how that section of code relates to the system as a whole. That holistic view is impossible on larger projects without good testing, or more experience than you want to wait for.
The importance of this will vary based on the importance of the system and its size and scope. Smaller size and scope, then they can probably learn the system well enough (say, less than 100k lines of code, depending on language). Beyond that size, the system will become a rotten mess and your customers will leave you if they have any choice.
You will want unit tests this way and to verify error handling and logic flow. You will want integration tests or acceptance tests to verify things work together and can perform the actions your users will take. These pay dividends and prevent the manual need to verify you did not break anything on a given change.
You also don't need to exhaustedly test everything. Start with critical parts. And, like others said, add to your test suite any bugs to prevent regressions. A paying customer who reports a bug, sees it fixed, and sees it come back is not a happy camper.
It will email you a stack trace whenever an error occurs and also keeps a log of them which you can order by frequency and so on. It's an extremely useful service.
When you get bigger and have regular users and need stability then tests start to become important so that you don't break existing things when rolling out new features.
Not having tests comes with a risk though. For example, I once broke our registration flow and didn't know about it for a week. Flipside is we hardly had any users registering at that point. So it's probably a good idea to cover a few of your core user journeys. Hope that helps!
Hence you should write tests only when you want to freeze your design in a specific area. I.e. make sure that you got the domain model right, and freeze it via tests, etc.
I found this book very valuable:
https://www.manning.com/books/unit-testing
How many engineers do you have and how fast are they likely to write code? It depends more on the product that you're building rather than the stack - if the tech is the product, spend more time on making sure it works as advertised. If the tech enables the product, spend less. Any competent engineer can pick up thousands of lines of technical debt/spaghetti code and spend a few afternoons swearing at it and figure out how to modify it. Technical debt is a problem, but more of a problem when you hit millions of lines, hundreds of engineers, and you need to be sure you can change stuff without breaking it.
Debt is useful - as long as you can take it on in a calculated way, don't stress about it too much. You might need to pivot and rewrite everything anyway, or at least have your fundamental assumptions changed in such a way that requires a partial rewrite. Tests won't help you much in those scenarios.
Pros
* Established the proper culture from day one. If you end up being successful, this is extremely valuable. Culture is very, very hard to change.
* Eliminated a lot of bugs and regressions.
* Gave us confidence working in a regulated space.
Cons
* Adds a lot of time to initial development
* Product complexity doesn't really require testing in the early stages.
* Slows down ability to pivot
-----
If I were to do it again. I'd write fewer tests, but establish milestones for requiring changing/additions to be tested.
* My rule of thumb would be: If you can't easily test it in the UI, write a test. Or, if it's business critical, write a test.
* Verify authorization and authentication. It's very important.
* Test anything that makes/loses money (for example, Strip, Recurly, etc).
* Jest snapshots are quick and easy. Use them.
* Write tests for utility functions
Don't test:
* Database migrations
* UI interactions (unless critical). They take a lot of time to write and UI changes frequently.
* Vendor/3rd party API integrations beyond surface level. In my experience, quick implementations always involve a bunch of mocking/stubbing - making these types of tests less useful.
Testing the product with end-to-end tests will stop you from shipping a completely broken product to your customer, where the most damage can occur. Add the test script at the end of the build and alert to a dashboard. “Alert” can mean email, and “dashboard” can mean “PC showing inbox for testalerts@mycompany”. You can get that running immediately. Alerts won’t tell you what broke of course, though you can use version control and bisecting to isolate breakages to commits, which is almost as good.
When you find bugs, then each bug is an opportunity to put some scaffolding around smaller subsystems as smaller tests to show the actual bug, as well as the fact that the bug has been fixed. Avoid committing the test without committing the fix. It’s a bad habit. You can show how the test used to fail in your commit message.
You’ll get an idea as to what are the most problematic pieces of your app. If one part is much less reliable than the others, then you can select that subsystem for verification with bottom up unit tests. Don’t spend too much time on each one though — it’s highly likely that in exercising your APIs through unit testing, you’re going to see ways in which they need to be refactored to make more sense and, importantly, become more reliable.
On the other hand if you start early you can take testability into the design and make at least some of the above problems go away, for example by having a modular architecture that allows easy mocking.
I do recommend to minimize the use of UI tests (Selenium for web for example) as they tend to be even more fragile, focus on the most obvious sunny day scenarios and never ignore a broken test as it get rotten over time
if you are worried about tech debt you should watch "The art of destroying software". It will be hard to understand at first.
Here is TL;DR. Writing small services that will take only one week of work to rewrite. That way you can burn your services and recreate when you truly understand your domain.
Recently I worked on a REST API using go. I think the go package system has good support for writing small and independent packages. Just make sure your microservices are micro.
You can do the same for your React Codebase.
You don’t have to write thousands of unit tests. But you should absolutely have a basic test suite with integration tests that give you confidence in a new release. And it will make it easier to hire developers in the future.
A good test suite doesn’t slow you down at all, in fact, it speeds you up by helping you identify problems rapidly so you can fix code before it becomes a real problem.
I wrote these tests after launching the product publicly and just before support requests / bug fixing period started with the simple aim of avoiding regressions as I work through the improvements.
If I were to give general suggestions on the topic,
- In most cases having high level sanity tests is low effort high value decision.
- To write tests or not shouldn't be a binary decision. E.g. If you think a specific aspect is going to need refinement work then tests for those modules can help you immensely.
- Going for high % coverage very early on for the sake of mental satisfaction or bragging rights is usually not a good time investment
My only regret is not having any automated tests for frontend. Investing in some basic selenium tests after launch would have been worth it.