HN Explorer

Ask HN: Why reusing e2e tests output is bad?

1 points by fedeb95 ↗ HN
I understand that the consensus is that each test should be independent in order to avoid side effects affecting other tests, but consider the following scenario.

I have n features f_1, ..., f_n. Each of them requires other features (a subset of them) to be executed in order to test the ith f_i. For each feature there is an end to end test t_i, which executes the corresponding feature and its dependencies and then makes some assertions on the feature under test.

Argument: the all set of test executions is a boolean expression t_1 AND ... AND t_n. So, for each t_i, we can describe it as f_i AND f_j AND ... for each dependency f_j of feature f_i, because if a dependency fails, the all test fails (it can't arrive to evaluating its assertions).

So the whole test execution could be represented as a BDD (https://en.wikipedia.org/wiki/Binary_decision_diagram) and reduced. Basically when I make a test, i reuse the output of a previous test which encompasses all the dependencies of the feature under test, for which assertions have already passed. Of course this means tests should have a precise scheduling order.

What's wrong with my idea? Thanks for the dedicated time.

4 comments

[ 2.8 ms ] story [ 17.7 ms ] thread
austin-cheney
I use end to end testing my node application. The interface is in the browser so for me testing must mean doing what the user can in the browser. That also means progressively altering the interface one test at a time.

You can try out how I do it if your are interested. The application is at:

https://github.com/prettydiff/share-file-systems

The test command is:

    share test_browser
fedeb95
yes, I don't mean not doing what a user does. My main concern is test execution speed. When you have a system that's not that fast and a lot of long tests, that can quickly add up. My idea is to avoid redo redundant steps.

Let's say that a user can book a flight and after it's booked he can manage his flight. I can have two tests: one to book a flight, and one to manage a booked flight. The second one doesn't need to book another flight as precondition, it can use the one booked when executing the first test. Hope this makes my point clearer.

austin-cheney
That depends on whether the combination of those two tests can result in different outcomes when combined or executed in slightly different ways. You want to cover as many permutations as possible in your test automation because test automation is intended to be faster than human testing and cover repetition that humans are bad at.

To increase performance I use websockets for everything. The Google answer for maximum network speed is gRPC with protobuf. That is supposedly 10x faster than JSON over HTTP, but protobuf is a pain in the ass. The way I execute websockets is 8x faster than HTTP and more convenient. Since I am not dealing with binary transform to protobuf my websocket approach with JSON serialization might actually be faster than (or at least as fast as) the Google solution.

fedeb95
thanks this is interesting. It's not something I'm an expert at but I'll research the topic