Ask HN: Why focus on unit tests over end-to-end tests?
I feel that I have worked in a few companies that overly prioritized so-called unit testing, to the point of spending too much effort on it. To me, it seems that end-to-end testing, meaning the kind of testing that tries to more or less fully simulate what the software user will actually cause the software to do, is both more useful and less brittle and will generally require less test rewriting than other approaches.
Yet undoubtedly I have met many at least otherwise smart people who are big fans of unit testing even if they cannot quite define what a unit is.
What am I missing, if anything?
5 comments
[ 3.5 ms ] story [ 16.9 ms ] threadYes, but for companies that are not Google scale, it should be possible to run end-to-end tests fast without too much trouble.
>They tend to be easy to write at the same time as the code they’re testing, allowing engineers to focus their tests on the code they’re working on without having to set up and understand a larger system.
But tests maybe should not be easy to write at the same time as the code they're testing, that implies too tight coupling between code and tests.
>They promote high levels of test coverage because they are quick and easy to write. High test coverage allows engineers to make changes with confidence that they aren’t breaking anything.
Why would a good end-to-end test be harder to write than 100 unit tests that purport to cover the same code paths?
>They tend to make it easy to understand what’s wrong when they fail because each test is conceptually simple and focused on a particular part of the system."
A good end-to-end test will give you the same thing if the code logs errors well. And if it does, that will be good for you even in production, not just testing.
>They can serve as documentation and examples, showing engineers how to use the part of the system being tested and how that system is intended to work.
I can see the value of this, but in practice I think that it is usually hard to show engineers how to use some unit without them being aware of the larger system.
IMO both are to be used together...unit testing for increasing development speed by catching errors earlier, end-to-end testing for making sure the full workflow works
the failure mode is writing so many unit tests that you actually slow yourself down maintaining a bunch of tests that barely ever increase your total velocity
By automatically mutating application code, the quality of the test suite itself can be evaluated: for example, a method is mutated to return null--and tests still pass! That could indicate a real bug or nothing.
End-to-end (e2e) is ideal to really exercise everything, but they can take a long time to finish. The feedback cycle is an eternity compared to unit tests.
One way to improve e2e is to have an agent on the app server and an agent on the test server, building a probabilistic model of what e2e tests to actually run; then it's a real time-saver to only execute those tests. At least one company offers this.
In the end, it's preferable to have both unit tests for microservices and e2e tests that run against one or more browsers.
Ideally, you'd test everything via the public interface (be it an API, a UI, or an SDK.) That would make it so you only change your tests when the user-facing behavior changes.
But your public interfaces are typically doing slow, non-deterministic things (e.g., calling external services.) Therefore, they aren't easy to test. Some people fix this by testing individual classes. It's better (but harder) to isolate slow/non-deterministic dependencies to small services that can be faked/stubbed out. Then you can have your cake and eat it too: Your tests are fast/deterministic, and they can still be end-to-end (and therefore not need to be updated whenever internals change.)
Your intuition is leading you in the right direction. There are ways to write "end to end" tests that have all the benefits of "unit" tests. The (free) book "Software Engineering at Google" has chapters on testing that describe the techniques better than I can. (They reach the same conclusion about how testing internals ends up being less useful than testing the public interface, and then they explain techniques/tradeoffs.)