HN Explorer

Ask HN: Founders, how much automated tests do you write?

35 points by allie1 ↗ HN
For startup founders looking for market fit, or up to Series A/bootstrapped and growing, how often do you (or your team) write automated testing?

How do you go about it? Full coverage? End to end only? Integration only? How much dev time does it end up taking you?

41 comments

[ 3.3 ms ] story [ 82.1 ms ] thread
anderspetersson
I write tests after fixing bugs to make sure they don't happen again. Mostly integration-tests.
aswinmohanme
While working on my startup as a solo-founder I used to follow a TDD approach supported mostly by Integration tests. I mostly focused on the user happy paths with limited coverage outside the Integration tests.

The goal is to have the major functionality of the platform tested automatically so we can be confident that new features don’t introduce new bugs.

karmelapple
Precisely what I do with personal side projects, and for greenfield development at the company at which I’m a cofounder (and which has been around for over a decade now).
Rev359a
The best practice is to write the test by knowing all the scenarios.

Usually this is the practice I have seen successful in all the big projects

- unit test for each function

- Functional Integration test to test the building blocks and functions together (for example rest API with business layer and database

- BDD Behavior driven development where you write automated test to test your business scenarios by initializing the seed data in the test systems

- Performance testing

If you have freelancer who charge per hour or if you have consultants which are expensive - they will write too many test and there code will be so complex in the end that when you want to hire permanent employees or other freelancers they will say "I would like to rewrite it"

The developers now a days like to do clean coding and for simple things they discuss too much even for a program of 2 + 2. there will be endless discussions and in some companies (diverse or multi culture) developers are always under mining each other with what they like and don't like

methyl
E2E for happy paths and unit tests for algorithms that are nontrivial, that’s about it.
[deleted]
(comment deleted)
is0tope
Not a founder, but a serial side-project'er. Initial prototype typically has no tests, at least for the bits that are wired. Specific bits of complexity will have unit tests. At some stage might add integration tests using something like docker compose.

Use TDD for bits that have very limited scope and function. For instance, an order book class or similar. Otherwise code first, test bugs if needed. For smaller projects (that aren't critical espc.) overtesting can kill your momentum and motivation, which is an often overlooked resource.

nocommandline
E2E for the core & commonly used features. But I plan to add more E2E tests for less commonly used features that I’d still consider core (these are manually tested for now due to time constraints).
dgelks
Last company, no tests for the first 6 months. Current company after 2 months, have some unit tests for complicated parts in place, will start writing tests once bugs are resolved.

I guess it really depends on what your company is doing, experience of the team and the language/tech you are using.

brimstedt
Tip: resolve the bugs by writing tests
shoo
technical perspective, ignoring business perspective: resolving the bugs by writing tests is excellent advice, especially once a component is sufficiently complex. get regression tests in place first, so you can sense how may new unknown defects a "fix" for a known defect is introducing. Michael Feathers' book "Working Effectively with Legacy Code" describes one approach.

somewhat political perspective: if the org doesn't value testing or software quality, wait until there's obviously a business need to lift quality. E.g. the CTO is saying things such as "constant defects in component X are destroying most of the value of our product", users complain that the Y feature keeps getting broken in successive releases, customers refuse to renew subscriptions or upgrade to a new version until Y feature works reliably. Then propose the plan to improve QA by setting up an automated regression test suite, to allow refactor-to-test of the offending code to happen in a controlled, low risk way.

dgelks
Agree with this totally, additionally if you are unsure if you will proceed with an idea with no users, getting something done and receiving feedback as fast as possible is better than doing it perfectly the first time.

That said at my last company after 12 months we hired a few engineers so we had to transition to requiring tests for new functionality. This was just to stop regressions and make sure we could continue moving quickly.

brimstedt
I partly agree with you, but I also think that using debugger/stepping through code/test manually will take longer times in most cases. And it's time wasted that does not produce anything "permanent" - unlike tests.
Dorcy64
I only write tests for important stuff that I must continually work on. Although in the past 12 months, my tests have been written by either chatgtp or playground codex models before that. Which makes me lose 0-time testing.
atonse
Almost Never test-first. It impedes my concentration and flow and inspiration. A new idea to me is like a fresh slab of marble you have to sculpt. Over time, you play with the code and api ergonomics datamodel and abstractions and UI until it “feels” right and takes on a more defined form. Working on tests at that point would be like continuously stopping to check the foundation of the marble.

But for self contained code that is guaranteed to have edge cases (working with dates, date math, unstructured user input like SMS responses, almost always yes, but after some of the code is written and is getting cumbersome to test different cases manually.

jcmontx
In our backend team we almost exclusively write integration tests.

For the frontend we do automated ui tests with a service called Autify.

This is not by any means extensive, BUT with very little time investment we get catch most of our bugs.

kaliara
I pretty much have the same strategy. We mostly want to ensure that crucial functionality doesn’t break for users. Since we are moving fast, we try not to break things.

Ghost Inspector used to be my go to, but I’ve been looking for something in the $20-30 per month range. Makes it a much easier decision when we aren’t yet collecting much revenue.

xtalax
The second you "think" a piece of software/a feature works, you should be verifying that this is the case automatically. It's often just as easy to write an automated test as test manually, just do it and that test will be there forever saving you from your own idiocy. You don't need to write the tests before you know how the feature is going to shake down to its implementation, but you'd sure do better to test before pushing to prod. You save yourself so much hassle in the long run with automated CI.
sjducb
The answer is so dependant on what you're doing, and your industry.

The answers here are all correct for the use case of the person answering it. We don't know your use case so we can't give you a good answer.

For example a seed funded health tech company needs 100% test coverage to be compliant with regulations.

A bootstrapped online collaboration tool with no users doesn't need any tests.

A seed funded online collaboration tool needs some tests so the developers can work together without breaking each others code.

shoo
Another aspect is the cost and risk to the company if it ships defects to production (assuming it actually has customers, has largely de-risked product-market fit, etc). Suppose a company is in the hardware business and locks its cash into building and shipping a run of their v1 consumer product, then discovers after shipping there's a defect in the entire manufacturing run that means the product needs to be recalled and repaired. That's likely sufficient to bankrupt a young company that doesn't have revenue flowing in from other products or lines of business. You'd want to be doing a lot more QA than merely unit testing.
danielovichdk
Code is a liability. Founder or not I write as few tests as I feel necessary for things I don't trust in my code.

I generally write as little code as possible overall, and try to sell it as fast as possible.

I hate source code, it never ever is the result for me. So if i do not have to write code than I get to not write tests.

My goal is always to write as little code as absolutely possible. That's it

Rev359a
Are you coding in bash script - just kidding

I mean how does it even works ? What if you want to build an app for a company which is wishing for digitization of their business - you need to have code - Don't you ?

Secondly how come founder are not liability - I have seen a stage in start ups when it comes to the situation where startup is either sold or purchased by other competitive business and after some months when the startup is absorbed the founders are gone or have a silent positions... Doesn't it make them liability afterwards?

hbrn
"I hate code, and I want as little of it as possible in our product." (Jack Diederich)
lbhdc
I write tests as I go along. It helps me challenge my assumptions about what I wrote, and it helps me write more robust code. I usually end up with 80-90% coverage.
JOnAgain
Depends. For web-app and business logic, we have about 95% coverage. All our models and back-end processes are well-tested with error cases. Our front-end & E2E is much lighter and we focus on happy cases.

Firmware is mostly tested manually.

stuckinhell
Not a founder, but I work at a very large fortune 500 firm. The vast majority of our custom apps have little to no tests. However our INFRASTRUCTURE is loaded with tests, heartbeat monitors, etc. You name it, we have it.
vr46
Prototyping and figuring out product? Zero.
Axsuul
Most code is tested, even for hobby projects. I find I save more time down the road with tests.
koliber
The things that keep breaking get automated tests. The things that can't possibly break or it will cost me significant revenue or time get tests. Everything else gets to wait until it falls into one of the above two buckets.
egberts1
When working with Regex and PyParsing, I find it imperative to maintain unit tests on everything small because I am dealing with over 2,400 abstract keyword combination built out of some 423 EBNF syntax of ISC Bind9 named.conf configuration file and all in Python.

Tiny mistakes that I make are caught early on, and development gets a solid foundation in which to build the proper abstract syntax tree.

Oh, test code is about 1/4th the size of the original source code, line-wise, but it has 100% coverage on syntax linkage (and not necessarily 100% data value validation due to dependency on other field settings).

source: https://github.com/egberts/bind9_parser

muzani
On front end, almost none. Apps are very test-hostile, so much so that just setting up the framework can increase complexity by ^3.

Well, unit tests are fine and often we do it for things like testing that email parsers and unit conversion works. As long as it's easier than manual testing.

It's also easier to do tests than debug sometimes.

Back end is a different story - the ROI on tests increases drastically because tests are easier to write and failure is more expensive.

mcapodici
This is something I have been considering as I work on firestarter [0] which is a going to be an open source starting point for building a SaaS.

I decided tests are a good thing to have, and especially when already set up and you can easily add more by following the pattern.

I prefer unit tests that are narrow in scope and avoid IO interactions (for example databases)

Of course anyone using my framework can ignore the tests if they desire. But my gut feeling is tests save you time in the short run as well as long run.

[0] https://github.com/mcapodici/firestarter