if you are fine releasing broken things, by all means, skip testing. Automated tests are faster and more repeatable than manual ones and allows you to more confidently release code. I like knowing that when I deploy that things basically still work.
At my last gig, acceptance testing that were blackbox were required before code could be merged to main. Sure, issues happened. But by far and large, we could release code and let alerts and other monitoring work and not have to check everything by hand. We seldom utterly broke things. We could confidently merge code and have it in production in like 15 minutes.
At my current gig, we lack acceptance tests or good integration tests for $crappy_reasons and even though we are a public company worth billions of dollars, every week the site is broken and people can't log in. Reverts happen regularly. The test suite takes over an hour and it is mostly unit level with some weak integration tests. The system itself is never actually exercised before it is released to customers. It took me 18 hours of test running to validate a schema change because you can't run the integration tests locally and so you have to let the build server do its thing. If tests were a priority here and treated well, my 18 hours would have been like 2. With no testing, I would have broken customer data and had to clean up tables after the fact.
If you prioritize fast, high level tests, you can really unlock velocity. If you have shitty tests that take a long time, your gonna have a bad time.
Our customers pay us and don't want to pay for a broken experience. I respect our customers too much to ship untested code.
Heck, even if we were free, I'd expect far more users to experience bugs than report them. Why would those users and customers continue to trust the reliability of your software? Thus reducing future conversions and giving reason for churn.
Nonsense. The argument isn't 100% proof based tests vs yolo shipping directly to prod. Our tests are based on user actions and expected error conditions (from bad input to network errors). We don't cover the combinatorial explosion of potential inputs but we don't need 100% mathematical proof based confidence.
Tests boost the confidence from "i think the code works" with no tests to "i know this code works in these N cases." It is risk management. Could we miss an unknown percentage of potential errors? Of course. But the errors that happen become fewer and fewer as the deployment safety net grows. And we monitor bug reports, errors, and incidents over time to gauge the effectiveness.
This has allowed us to confidently ship code that has earned our company, literally, hundreds of millions of dollars and has evolved over time. I know my tests mean the software works as expected because my tests are my expectations. Maybe there is a cache bug or a memory leak or a logic bug or a misunderstood business requirement, but I _know_ that when my code deploys, standard usage of it will absolutely work. If it didn't, then a test should have failed.
I've been developing software for a couple of decades now and I have a good feel on what to test at what layer. If the argument is only "let my valued customers experience bugs and report them to me" vs tests before code release, I'm siding on tests before releasing.
I was hoping for some interesting insight to be shared, but no it’s just a shallow opinion piece on why E2E is better than unit tests for web interfaces, without really explaining why.
The author misses that component tests don’t have to be like unit tests at all, they are more like integration tests if you test entire pages (which are components too), use actual user interactions and mock only the I/O (mostly network) layer. It’s like E2E without spinning up a backend.
As someone advocates for and runs E2E tests, you're going to be hard pressed to find any one writing about it.
The main objection heard will be: "It takes x hours and n servers to set up an E2E environment."
Infrastructure is hard, and with cloud functionally unlimited, so people write a mock and move on with their life.
To do E2E well your culture needs to support it. You need to be able to spin up, run and summarize an E2E test in less than 20 minutes. Set your test up to END on the hour (not start, end) SO that if there is an outage no one can make the excuse that the "meeting already started". If the test fails every one who has changes in that build drops what they are doing and treats it like a production outage. The technological magpies, collecting shiny things and adding them to the stack has to stop, because you dont want to add "new tech" to build time. You may need to run some of your own infra to get this done. Some servers in a closet some where will do wonders for your bills.
Because you're always building your environment from scratch, in the event of a real disaster you know what it takes to put a large chunk of your environment back together. Your like going to have a pretty lean stack, one that scales diagonally (at this point go look at AWS spot pricing and how much you can save if you can scale that way).
That isn't to say you should not unit test. Things that are clearly stand alone, like well defined validation, things that are ugly (that 20 condition switch statement that powers some esoteric edge case of business "logic") should get unit tests. Places where you can leverage fuzzing easily should have unit tests! Anything that has a dependency tree or needs a mock, should be skipped and addressed with an end to end test.
What will you get in the end? A production environment with every bit of unneeded complexity stripped from it (and probably a lower cloud bill). An aversion to adding anything that doesn't have VALUE to your stack. A ruthless eye for what qualifies as "bloat" (hint, a LOT of it is bloat). The removal of code for codes sake (mocks) and layers of duplication (an E2E create user test can exercise a lot of code, and replace a lot of unit tests at various levels). The whole team having a much better understanding of how your infrastructure is built and deployed.
Does this logic extend to /parts of software/? Most lines of code are not UIs, so they are not used by humans, meaning they should be tested by machines?
The 'user' being the entity that is using the system at the 'commercial boundary' - where your customer interfaces with your system. So for a web site, it is the human, whereas for a banking API, it is the machine.
It may be useful for development purposes to create artificial internal 'commercial boundaries', but unit tests are far too fine grained.
I shutter thinking my boss could read that article because unit testing is already viewed as extra work that is unnecessary.
It’s apparent that the author has never worked on a legacy product.
End-to-end testing works great until your project grows in complexity and something doesn’t work right. Then you play the game of looking for where and why while your boss is frustratingly asking you how long it’s going to take while telling all the other bosses that you’re incompetent. Worst case, you break something, and you don’t realize it, or your coworkers break something you wrote and they don’t realize it.
E2E is not a replacement for unit tests. E2E tests are more difficult to write, as they can require a lot of setup to get all the exact scenario needed for each piece of functionality we need to test, they tend to be flaky, they can be very slow, etc.
The purpose of unit testing is to isolate the various scenarios by making sure that each unit/component/whatever correctly responds to its various inputs. In this way, we only need to set up the relevant scenario for that unit. If we tested each of these piece of functionality in an E2E test, then each test would also need to set up the user session, settings, etc. It would be a nightmare to author and maintain, compared to the simple, straight forward unit tests.
In contrast, E2E tests are good at making sure that the whole system works well together, and that each unit is getting inputs and providing outputs, and that the page doesn't break in the process. Both are needed to maintain confidence, and each solves a different problem.
Agreed. An e2e test suite can continue passing with significant bug rot throughout the system, as they’re often given generous timeouts, retries for flakes, etc. So once you realize something is really, truly broken; it’s going to be a nightmare to track down and properly fix if the e2e suite is your only data point.
I have seen projects where people push back on making major functional changes and the only rational explanation is that they don’t want to have to rewrite a bunch of brittle E2E tests.
That really took the value of E2E down several notches in my estimation.
>> I have seen projects where people push back on making major functional changes and the only rational explanation is that they don’t want to have to rewrite a bunch of brittle E2E tests.
Same for any testing. it should be part of the cost and Cost Benefit Analysis.
There are ways to write unit tests that make it a lot cheaper to change requirements. But I can still count the number of people I've worked with who write them that way.
Right now I'm still trying to convince people that fakes and mocks are not the same thing.
If your E2E test is flaky, then how is your production environment reliable?
> they can require a lot of setup
Setup vs mock.
> they can be very slow, etc.
We have a good handle on scaling, slow is now a product of the testing time being smeared across your org like peanut butter.
> The purpose of unit testing...
Does E2E replace unit testing, no. Things that are already units or esoteric (business "logic", you know it's in your code base). But if your writing mocks your better off un fucking your environment that E2E testing is viable, and make it a first class citizen. Everything is a crud app of some sort: web, api, mobile, smoke signal... Dropping data in these front ends, can be validated out of the back end, and in the app itself.
To put a fine point on it: Unit tests make me feel good about checking in. End to end tests make me confident to deploy!
>> If your E2E test is flaky, then how is your production environment reliable?
IME, E2E tests are flaky despite the underlying system not being flaky because E2E has to do a series of things, often including waiting/polling, and in real life people will wait longer or compensate but in tests you want to do it all perfectly and fast.
I think in this scenario, different people will have a different perspective on whether asking people to "wait longer" constitutes flakiness or not.
From an end-user's perspective, there's very little difference between "the task didn't work" and "the task worked, but took longer than your patience to wait." Both appear to be broken.
The flakiness comes from needing an automated browser interaction acting against a page with changing input. If your page is static, sure, the E2E test is probably straightforward and quick. But the moment any dynamic functionality is introduced to the page, the test's interactions need to account for the page's current context (or wait for an intended context). And that has to be coordinated amongst the actual test code, the testing browser, the driver, etc. If anyone one of them hiccups, you may end with a false negative, and those happen more often than we'd like.
> The flakiness comes from needing an automated browser interaction acting against a page with changing input.
This isn't an an end to end testing problem, this is a testing problem. Mocking isn't a fix for this (it's not a network issue). And testing in another environment (node/bun as units) is akin to deploying a server to linux boxes after having tested on windows (and I would argue far worse because the whole purpose of the front end is the dam UI).
I'm somewhat at a loss as to what point you're trying to make. Can you clarify what you consider to be E2E tests, and what you consider to be unit tests, with examples?
Browser test: front end -> api -> db, full transaction life cycle inter coverage of the whole interface (crud operations).
Mobile: same as browser...
API tests: test client -> api -> db, full transaction lifecycle, should give coverage to the backend. It isn't the job of a browser test to hit every variant of every endpoint however, they should all be exercised from the front end.
You should be calling out to as many actual running services durning E2E testing, the system should behave in this environment the same way it will in production. NO mocks no synthetics, "full" data paths.
Unit testing:
Validation screams for "unit test". Does your "email" field allow for + encoding? Do you have frontend CC validation (does the card number hash) vs back end where it "auths" the card (auth is outside the scope of a unit). None of these sorts of test should need a mock. We're not faking another part of our stack (those should be end to end tests). Does that mean you should NEVE mock, and have "full" unit tests, no. But they should be sparse, for those things that have to be hard to understand or are complex (business "logic").
The bonus is you also have a wet blanket for scaling issues (if you do it right). Need to refactor or re-platform a service/endpoint? The tests are agnostic from the code, and you should have a "thin" set of unit tests to port!
>> From an end-user's perspective, there's very little difference between "the task didn't work" and "the task worked, but took longer than your patience to wait." Both appear to be broken.
I disagree. Assume it usually takes 1sec but 1% of the time it takes 120sec. Does your test suite wait 120sec and fail 1% of the time or does it wait 1sec and speed up the test suite? These are test trade-offs IMHO
Yeah it's almost like you join a team and that's not your call, and it;s almost
you might join a team that works an SPA application that's so complicated it would need real time updates.
>> Only if you run pure server side applications that’s simple.
End to End testing is less about "making the front end do all the work" and more about being free of mocks and synthetics.
Just because the front end exercises my server, does not mean server devs get a pass on testing API down. The tests I write should inform the data your sending and receiving, they should function as an example of inputs and outputs of what to expect in the system. Those timing issues are either a real front end problem, or a server problem (and my own tests should have noted that I did not meet the SLA).
This is my experience as well. Unit test suites tend to be huge, which costs effort to maintain, and they generally tell you little about the actual quality of the software.
A good end to end test suite will give you a much truer indication of the state of your systems.
Of course I don’t say do zero unit tests, they have value, but I think too many orgs are too zealous about unit tests and too soft on higher level testing.
Great. Go over in that corner and argue with the other testers and some managers about defining "better" while the rest of us get some work done.
> strong emphasis on unit tests, and while they have their place, let’s not forget about a category of tests that I consider much more valuable when it comes to testing complex web application user interfaces: End-to-End
First, testing UI doesn't mean the tests have to be End-to-End(E2E).
Second, I have the opposite experience, where teams seem to want test devs to write slow, brittle E2E tests that require maintaining fragile environments that often need to cross-coordinate among many repos deploying to it (plus any manual changes that always seem to happen).
I see very little emphasis on unit tests. And I see almost none on contract tests, the fast way to break E2E tests into grok-able pieces that can run on every push (just like unit tests).
> let’s shift our focus from output to outcome
I don't really see that a focus on E2E tests outcomes. I just see that as slow, brittle, high-maintenance, and something requiring an environment (thus not running the tests on every push - even slower feedback).
Maybe try to focus on fast, meaningful feedback instead?
My first real software job I worked primarily in Python and diligently added and maintained unit tests for every code path. This effort really paid off as my team and I could be so much more certain that changes and refactors wouldn't break things in prod.
A few years into that job, gradual typing came to Python and I realized that a substantial fraction of the unit tests we were writing were in fact just making up for a lack of static analysis in Python. Since then I've come to believe that with decent typing and static analysis, unit tests are most useful during development of new code and only marginally useful after that. Integration tests are much harder to set up initially but are much more valuable.
Same background as you, still work a lot in Python to this day to a lesser degree. A couple of caveats: first, for strictly web-based integration tests, I agree that frequently they will give you what you need faster than units. Many popular API framework libs even give you a way to test the framework components as if they were being hit over http without having to “really” run the server via test clients. All of this is great. When you are working outside of a strictly HTTP paradigm, however, this can get complicated. When using Python for data-intensive tasks outside the realm of http (I.e. worker/task paradigm, event-driven, etc.) unit tests are sometimes the only feasible way of validating an atomic piece of work, as a task may need to invoke multiple functions/grow to do more, and a simple “fire task X in Docker and hope that you end up with the desired data store state afterwards” can get complicated and obscure lower level bugs that a unit test can account for.
Second: the rise of pedantic has given the Python ecosystem a great way to enforce typing and make up for python’s lack of SA. It can be really helpful to document what types are fed to which layers of the system. Since Pydantic will aggressively explode if users try to start throwing arbitrary models/data packaged around, it can really help if you’re working in a system where a model needs to change in one place rather than 10, and can lead to better hygiene than in the old days, where it was more common to pass less strict data wrappers around or have many layers of software defining their own interfaces rather than exporting that to a model which can be reused globally. In theory this is also possible with POPOs but in reality, duck-typing and mocks can lead developers into some really bad places where they think they’re typing things correctly until they get to the integration tests, which catch the bug and throw any unit work out the window for another iteration.
E2E tests are useful, but I definitely do not agree that they should be the primary target when writing tests. Setting up the correct data conditions for an E2E test to accurately exercise the code is often challenging or sometimes impossible, particularly since E2E tests often execute against a hosted environment with shared resources. Unit tests probably often have room for improvement, but in my experience they are extremely useful for checking code during development before I've produced a final product that could be tested E2E.
I was just looking at the unit tests I have in a game-related Rust program. "cargo test" runs these. So what did I test?
- A complicated index of objects in a virtual world. The test creates some dummy objects, moves them around, and runs a consistency check on the data structures after each change. This has caught bugs in the past.
- A lock-by-string primitive, used when I don't want two threads working on the same URL.
- Serializers and deserializers, usually where some data comes in from the outside. Quoted strings in the test cases deserialize, reserialize, and check.
- There are geometry generators which generate spheres, boxes, cylinders, etc. Those all have unit tests.
- Some math functions, such as angle between two quaternions, and some computations around transformation matrices and the viewing frustrum that tell what's visible on screen.
- Some transformations on meshes.
- Some simple parsers.
Mostly things with algorithmic complexity, data format dependency, or non-trivial internal consistency.
Hard parts that the Rust compiler won't catch, and which are hard to debug in a larger context.
With those tests in place, and in 100% safe Rust, not much mysterious happens that's internal to the program.
External relationships with the game server are another matter. That's handled with lots of internal checks and logging.
Of course context matters but _generally_, end to end tests should be mostly avoided:
- the execution time of end to end tests are orders of magnitude larger than unit tests. In a medium ish application your end to end tests will take several seconds to spin up, for example by spinning up fresh databases (even with rollback in gets slow) and other detachable services. A unit test is usually a malloc away and thousands can be run in parallel on your local box in a second. End to end tests easily take hours of compute time time to run in the real world. Good luck running the suite on your laptop. Nothing kills productivity quicker than a test suite that runs slowly
- unit tests are also independent of other code so it’s easy to get some confidence that your code does the right thing by only testing the touched files in your diff before running the entire suite on CI.
- unit tests encourage good code quality since it’s easier to test injectable code. E2E instead incentivises copy pasted test suites that are thousands of lines of code long.
- in the general case end to end tests are asynchronous and require orders of magnitude more effort to run. Asynchronous tests are flaky. Flaky builds slow down your velocity.
- E2E tests rot really fast. One change leads to thousands of broken E2E tests.
- E2E is not a replacement for monitoring your application. The class of bugs you will catch in your E2E suite is a strict subset of the class of bugs you will catch with monitoring and metrics. Build the latter instead and roll out your code incrementally to your users.
- E2E tests are complicated to set up and maintain in ops. Unit tests not so much. Somewhat unexpectedly your development velocity scales pretty much linearly with stack complexity.
In summary (and again context matters and there aren’t any absolutes, but …):
- Don’t do automated E2E tests.
- Do unit tests
- Do metrics and monitoring
You have a limited budget, spend it where you get a good bang for your buck.
Unit tests and smoke/E2E are like peanut butter and jelly, they're better together. And yeah they're expensive to build but it's almost always cheaper than deploying crap and hurting your customers.
The drama is mostly because lots of people don't like writing tests, and lots of folks at one point or another have seen unit tests that are questionably valuable or past the point of diminishing returns I think of tests that require a bunch of scaffolding to mock out any meaningful logic in order to test a single, simple if statement that invokes some other business logic that is already meaningfully tested on its own. At that point, it's easy to say "Wow, a couple of E2E tests would have caught the whole shebang!" Maybe I've just been burned by how easily one (read: I) can accidentally write pointless unit tests in Python.
I think the unit testing "debate" when it comes up tends to be very handwavey, like this article (which I still think is thoughtful.)
E2E also has the potential of being a cross-team nightmare when multiple teams own individual components of a process rather than a single team owning the entire tech stack. So focusing on outcomes rather than output doesn't necessarily work because the teams doing the work are only given responsibility for an output. This is mostly an organizational problem but it does trickle down to the team level. And yah, it's not 'agile' which I feel like this post is just reiterating one of those pillars.
I think this post glosses over the reality that tech workers primarily want to make things work correct, and they generally would write good tests (whichever kind that means for their problem) if they had the time and resources. But the final words seem to indicate doing a check on if the test is valuable... But that's not really where the problem or difficulty lies. It's often an organizational/team barrier (at least in the places I've worked) not a tech problem. And the commentary is often 'well this test is as good as we can do without engaging with all of these known pains'
All testing can ever do is increase your confidence that you’re delivering value and not destroying it. I think it’s easy to believe that if you’re just pure enough and follow some prescribed process or another then you’re doing a good job, but that’s just not possible. If you’re thinking about this stuff at all and you have some of your own principles, good on you.
Personally I’ve settled on this:
1) Whatever style of tests you write, they need all the love and attention your other code has. You need reuse, abstractions, constant refactoring. This makes me wary of external E2E frameworks in languages separate from your application codebase, but I’m aware most people don’t get to use a single stack across front and back end.
2) Functional tests are great. At the very least your codebase should have somewhere you can see its use cases enumerated (and preferably verified) and it’s rare for this to exist without functional tests (but good on you if you explicitly model use cases in your code). These are the things that actually make you money and should be your primary concern.
3) You can’t test every possible code path in functional tests, or at least not easily and quickly. I’m happy to test the bones of the system end to end but isolate complex logic the same way you would external dependencies. With this isolation you can easily write property based tests to verify the behaviour, and these to me are by far the most (dare I say only) valuable form of unit test.
4) I’m happy for end to end tests to hit mock/fake components but I always mock out the other side of the protocol. That is, tests that query a database always query a real embedded database not a mocked class that returns fake results. Embedded Postgres is extremely fast (20ms overhead in my tests) and perfectly reliable. The same is true of network services running in process on localhost.
5) Any time you mock/fake something you also need some way of verifying you’re not lying to yourself so you should have a minimum number of tests against real dependencies to verify your assumptions.
6) Unit test an implementation too early and you’ve done twice as much work to throw away when you refactor. You need to spike and stabilise based on your judgement. Personally I’m not concerned if I have no unit tests at all in code that is exercised by integration or functional tests, with the caveat about property based tests of complex logic above.
7) If you’re working on a legacy codebase or in a dysfunctional organisation then just do your best, obviously. Most of the time making things easier to test improves their architecture anyway but it’s very hard to pin revenue on that.
55 comments
[ 3.3 ms ] story [ 113 ms ] threadAt my last gig, acceptance testing that were blackbox were required before code could be merged to main. Sure, issues happened. But by far and large, we could release code and let alerts and other monitoring work and not have to check everything by hand. We seldom utterly broke things. We could confidently merge code and have it in production in like 15 minutes.
At my current gig, we lack acceptance tests or good integration tests for $crappy_reasons and even though we are a public company worth billions of dollars, every week the site is broken and people can't log in. Reverts happen regularly. The test suite takes over an hour and it is mostly unit level with some weak integration tests. The system itself is never actually exercised before it is released to customers. It took me 18 hours of test running to validate a schema change because you can't run the integration tests locally and so you have to let the build server do its thing. If tests were a priority here and treated well, my 18 hours would have been like 2. With no testing, I would have broken customer data and had to clean up tables after the fact.
If you prioritize fast, high level tests, you can really unlock velocity. If you have shitty tests that take a long time, your gonna have a bad time.
Tests are are very weak at proving the correctness of your code anyway. So why bother, there will be always non covered cases
Heck, even if we were free, I'd expect far more users to experience bugs than report them. Why would those users and customers continue to trust the reliability of your software? Thus reducing future conversions and giving reason for churn.
If your code is critical you need proof of correctness, not feelings.
Tests boost the confidence from "i think the code works" with no tests to "i know this code works in these N cases." It is risk management. Could we miss an unknown percentage of potential errors? Of course. But the errors that happen become fewer and fewer as the deployment safety net grows. And we monitor bug reports, errors, and incidents over time to gauge the effectiveness.
This has allowed us to confidently ship code that has earned our company, literally, hundreds of millions of dollars and has evolved over time. I know my tests mean the software works as expected because my tests are my expectations. Maybe there is a cache bug or a memory leak or a logic bug or a misunderstood business requirement, but I _know_ that when my code deploys, standard usage of it will absolutely work. If it didn't, then a test should have failed.
I've been developing software for a couple of decades now and I have a good feel on what to test at what layer. If the argument is only "let my valued customers experience bugs and report them to me" vs tests before code release, I'm siding on tests before releasing.
But your customers will want those features and you'll be glad you had fast, meaningful tests.
He has a lot of haters. I do kind of see why.
The author misses that component tests don’t have to be like unit tests at all, they are more like integration tests if you test entire pages (which are components too), use actual user interactions and mock only the I/O (mostly network) layer. It’s like E2E without spinning up a backend.
The main objection heard will be: "It takes x hours and n servers to set up an E2E environment."
Infrastructure is hard, and with cloud functionally unlimited, so people write a mock and move on with their life.
To do E2E well your culture needs to support it. You need to be able to spin up, run and summarize an E2E test in less than 20 minutes. Set your test up to END on the hour (not start, end) SO that if there is an outage no one can make the excuse that the "meeting already started". If the test fails every one who has changes in that build drops what they are doing and treats it like a production outage. The technological magpies, collecting shiny things and adding them to the stack has to stop, because you dont want to add "new tech" to build time. You may need to run some of your own infra to get this done. Some servers in a closet some where will do wonders for your bills.
Because you're always building your environment from scratch, in the event of a real disaster you know what it takes to put a large chunk of your environment back together. Your like going to have a pretty lean stack, one that scales diagonally (at this point go look at AWS spot pricing and how much you can save if you can scale that way).
That isn't to say you should not unit test. Things that are clearly stand alone, like well defined validation, things that are ugly (that 20 condition switch statement that powers some esoteric edge case of business "logic") should get unit tests. Places where you can leverage fuzzing easily should have unit tests! Anything that has a dependency tree or needs a mock, should be skipped and addressed with an end to end test.
What will you get in the end? A production environment with every bit of unneeded complexity stripped from it (and probably a lower cloud bill). An aversion to adding anything that doesn't have VALUE to your stack. A ruthless eye for what qualifies as "bloat" (hint, a LOT of it is bloat). The removal of code for codes sake (mocks) and layers of duplication (an E2E create user test can exercise a lot of code, and replace a lot of unit tests at various levels). The whole team having a much better understanding of how your infrastructure is built and deployed.
If the system is used by humans, then it needs to be tested by humans. Ideally but not necessarily not the customers.
If it is used by just machines then machine testing is fine.
(Per the article, unit tests are clown coding really - nobody cares)
Do we consider webbrowsers to be humans here?
It may be useful for development purposes to create artificial internal 'commercial boundaries', but unit tests are far too fine grained.
It’s apparent that the author has never worked on a legacy product.
End-to-end testing works great until your project grows in complexity and something doesn’t work right. Then you play the game of looking for where and why while your boss is frustratingly asking you how long it’s going to take while telling all the other bosses that you’re incompetent. Worst case, you break something, and you don’t realize it, or your coworkers break something you wrote and they don’t realize it.
Unit tests can protect you from those situations.
The purpose of unit testing is to isolate the various scenarios by making sure that each unit/component/whatever correctly responds to its various inputs. In this way, we only need to set up the relevant scenario for that unit. If we tested each of these piece of functionality in an E2E test, then each test would also need to set up the user session, settings, etc. It would be a nightmare to author and maintain, compared to the simple, straight forward unit tests.
In contrast, E2E tests are good at making sure that the whole system works well together, and that each unit is getting inputs and providing outputs, and that the page doesn't break in the process. Both are needed to maintain confidence, and each solves a different problem.
That really took the value of E2E down several notches in my estimation.
Same for any testing. it should be part of the cost and Cost Benefit Analysis.
Right now I'm still trying to convince people that fakes and mocks are not the same thing.
If your E2E test is flaky, then how is your production environment reliable?
> they can require a lot of setup
Setup vs mock.
> they can be very slow, etc.
We have a good handle on scaling, slow is now a product of the testing time being smeared across your org like peanut butter.
> The purpose of unit testing...
Does E2E replace unit testing, no. Things that are already units or esoteric (business "logic", you know it's in your code base). But if your writing mocks your better off un fucking your environment that E2E testing is viable, and make it a first class citizen. Everything is a crud app of some sort: web, api, mobile, smoke signal... Dropping data in these front ends, can be validated out of the back end, and in the app itself.
To put a fine point on it: Unit tests make me feel good about checking in. End to end tests make me confident to deploy!
IME, E2E tests are flaky despite the underlying system not being flaky because E2E has to do a series of things, often including waiting/polling, and in real life people will wait longer or compensate but in tests you want to do it all perfectly and fast.
From an end-user's perspective, there's very little difference between "the task didn't work" and "the task worked, but took longer than your patience to wait." Both appear to be broken.
The flakiness comes from needing an automated browser interaction acting against a page with changing input. If your page is static, sure, the E2E test is probably straightforward and quick. But the moment any dynamic functionality is introduced to the page, the test's interactions need to account for the page's current context (or wait for an intended context). And that has to be coordinated amongst the actual test code, the testing browser, the driver, etc. If anyone one of them hiccups, you may end with a false negative, and those happen more often than we'd like.
This isn't an an end to end testing problem, this is a testing problem. Mocking isn't a fix for this (it's not a network issue). And testing in another environment (node/bun as units) is akin to deploying a server to linux boxes after having tested on windows (and I would argue far worse because the whole purpose of the front end is the dam UI).
Browser test: front end -> api -> db, full transaction life cycle inter coverage of the whole interface (crud operations).
Mobile: same as browser...
API tests: test client -> api -> db, full transaction lifecycle, should give coverage to the backend. It isn't the job of a browser test to hit every variant of every endpoint however, they should all be exercised from the front end.
You should be calling out to as many actual running services durning E2E testing, the system should behave in this environment the same way it will in production. NO mocks no synthetics, "full" data paths.
Unit testing:
Validation screams for "unit test". Does your "email" field allow for + encoding? Do you have frontend CC validation (does the card number hash) vs back end where it "auths" the card (auth is outside the scope of a unit). None of these sorts of test should need a mock. We're not faking another part of our stack (those should be end to end tests). Does that mean you should NEVE mock, and have "full" unit tests, no. But they should be sparse, for those things that have to be hard to understand or are complex (business "logic").
The bonus is you also have a wet blanket for scaling issues (if you do it right). Need to refactor or re-platform a service/endpoint? The tests are agnostic from the code, and you should have a "thin" set of unit tests to port!
I disagree. Assume it usually takes 1sec but 1% of the time it takes 120sec. Does your test suite wait 120sec and fail 1% of the time or does it wait 1sec and speed up the test suite? These are test trade-offs IMHO
Whether you use playwright or cypress getting the correct wait events figured out is a huge pain.
Only if you run pure server side applications that’s simple.
End to End testing is less about "making the front end do all the work" and more about being free of mocks and synthetics.
Just because the front end exercises my server, does not mean server devs get a pass on testing API down. The tests I write should inform the data your sending and receiving, they should function as an example of inputs and outputs of what to expect in the system. Those timing issues are either a real front end problem, or a server problem (and my own tests should have noted that I did not meet the SLA).
A good end to end test suite will give you a much truer indication of the state of your systems.
Of course I don’t say do zero unit tests, they have value, but I think too many orgs are too zealous about unit tests and too soft on higher level testing.
Great. Go over in that corner and argue with the other testers and some managers about defining "better" while the rest of us get some work done.
> strong emphasis on unit tests, and while they have their place, let’s not forget about a category of tests that I consider much more valuable when it comes to testing complex web application user interfaces: End-to-End
First, testing UI doesn't mean the tests have to be End-to-End(E2E).
Second, I have the opposite experience, where teams seem to want test devs to write slow, brittle E2E tests that require maintaining fragile environments that often need to cross-coordinate among many repos deploying to it (plus any manual changes that always seem to happen).
I see very little emphasis on unit tests. And I see almost none on contract tests, the fast way to break E2E tests into grok-able pieces that can run on every push (just like unit tests).
> let’s shift our focus from output to outcome
I don't really see that a focus on E2E tests outcomes. I just see that as slow, brittle, high-maintenance, and something requiring an environment (thus not running the tests on every push - even slower feedback).
Maybe try to focus on fast, meaningful feedback instead?
A few years into that job, gradual typing came to Python and I realized that a substantial fraction of the unit tests we were writing were in fact just making up for a lack of static analysis in Python. Since then I've come to believe that with decent typing and static analysis, unit tests are most useful during development of new code and only marginally useful after that. Integration tests are much harder to set up initially but are much more valuable.
Second: the rise of pedantic has given the Python ecosystem a great way to enforce typing and make up for python’s lack of SA. It can be really helpful to document what types are fed to which layers of the system. Since Pydantic will aggressively explode if users try to start throwing arbitrary models/data packaged around, it can really help if you’re working in a system where a model needs to change in one place rather than 10, and can lead to better hygiene than in the old days, where it was more common to pass less strict data wrappers around or have many layers of software defining their own interfaces rather than exporting that to a model which can be reused globally. In theory this is also possible with POPOs but in reality, duck-typing and mocks can lead developers into some really bad places where they think they’re typing things correctly until they get to the integration tests, which catch the bug and throw any unit work out the window for another iteration.
- A complicated index of objects in a virtual world. The test creates some dummy objects, moves them around, and runs a consistency check on the data structures after each change. This has caught bugs in the past.
- A lock-by-string primitive, used when I don't want two threads working on the same URL.
- Serializers and deserializers, usually where some data comes in from the outside. Quoted strings in the test cases deserialize, reserialize, and check.
- There are geometry generators which generate spheres, boxes, cylinders, etc. Those all have unit tests.
- Some math functions, such as angle between two quaternions, and some computations around transformation matrices and the viewing frustrum that tell what's visible on screen.
- Some transformations on meshes.
- Some simple parsers.
Mostly things with algorithmic complexity, data format dependency, or non-trivial internal consistency. Hard parts that the Rust compiler won't catch, and which are hard to debug in a larger context.
With those tests in place, and in 100% safe Rust, not much mysterious happens that's internal to the program.
External relationships with the game server are another matter. That's handled with lots of internal checks and logging.
- the execution time of end to end tests are orders of magnitude larger than unit tests. In a medium ish application your end to end tests will take several seconds to spin up, for example by spinning up fresh databases (even with rollback in gets slow) and other detachable services. A unit test is usually a malloc away and thousands can be run in parallel on your local box in a second. End to end tests easily take hours of compute time time to run in the real world. Good luck running the suite on your laptop. Nothing kills productivity quicker than a test suite that runs slowly
- unit tests are also independent of other code so it’s easy to get some confidence that your code does the right thing by only testing the touched files in your diff before running the entire suite on CI.
- unit tests encourage good code quality since it’s easier to test injectable code. E2E instead incentivises copy pasted test suites that are thousands of lines of code long.
- in the general case end to end tests are asynchronous and require orders of magnitude more effort to run. Asynchronous tests are flaky. Flaky builds slow down your velocity.
- E2E tests rot really fast. One change leads to thousands of broken E2E tests.
- E2E is not a replacement for monitoring your application. The class of bugs you will catch in your E2E suite is a strict subset of the class of bugs you will catch with monitoring and metrics. Build the latter instead and roll out your code incrementally to your users.
- E2E tests are complicated to set up and maintain in ops. Unit tests not so much. Somewhat unexpectedly your development velocity scales pretty much linearly with stack complexity.
In summary (and again context matters and there aren’t any absolutes, but …):
- Don’t do automated E2E tests.
- Do unit tests
- Do metrics and monitoring
You have a limited budget, spend it where you get a good bang for your buck.
Unit tests and smoke/E2E are like peanut butter and jelly, they're better together. And yeah they're expensive to build but it's almost always cheaper than deploying crap and hurting your customers.
Why all the drama?
I think the unit testing "debate" when it comes up tends to be very handwavey, like this article (which I still think is thoughtful.)
I think this post glosses over the reality that tech workers primarily want to make things work correct, and they generally would write good tests (whichever kind that means for their problem) if they had the time and resources. But the final words seem to indicate doing a check on if the test is valuable... But that's not really where the problem or difficulty lies. It's often an organizational/team barrier (at least in the places I've worked) not a tech problem. And the commentary is often 'well this test is as good as we can do without engaging with all of these known pains'
Personally I’ve settled on this:
1) Whatever style of tests you write, they need all the love and attention your other code has. You need reuse, abstractions, constant refactoring. This makes me wary of external E2E frameworks in languages separate from your application codebase, but I’m aware most people don’t get to use a single stack across front and back end.
2) Functional tests are great. At the very least your codebase should have somewhere you can see its use cases enumerated (and preferably verified) and it’s rare for this to exist without functional tests (but good on you if you explicitly model use cases in your code). These are the things that actually make you money and should be your primary concern.
3) You can’t test every possible code path in functional tests, or at least not easily and quickly. I’m happy to test the bones of the system end to end but isolate complex logic the same way you would external dependencies. With this isolation you can easily write property based tests to verify the behaviour, and these to me are by far the most (dare I say only) valuable form of unit test.
4) I’m happy for end to end tests to hit mock/fake components but I always mock out the other side of the protocol. That is, tests that query a database always query a real embedded database not a mocked class that returns fake results. Embedded Postgres is extremely fast (20ms overhead in my tests) and perfectly reliable. The same is true of network services running in process on localhost.
5) Any time you mock/fake something you also need some way of verifying you’re not lying to yourself so you should have a minimum number of tests against real dependencies to verify your assumptions.
6) Unit test an implementation too early and you’ve done twice as much work to throw away when you refactor. You need to spike and stabilise based on your judgement. Personally I’m not concerned if I have no unit tests at all in code that is exercised by integration or functional tests, with the caveat about property based tests of complex logic above.
7) If you’re working on a legacy codebase or in a dysfunctional organisation then just do your best, obviously. Most of the time making things easier to test improves their architecture anyway but it’s very hard to pin revenue on that.