> Only isolate your code from truly external services
That makes tests more trustworthy but also sometimes harder to maintain I think. I have seen cases where small changes on the code base created strong ripple effects with many tests to update.
Arguably, the tests were not very well written or organized and with too many high level tests. Still, this and the very large execution time of the test collection made me realized that for medium to large projects, I will be much more careful in the future before going all in with the no-mock approach.
Or, write code that _can_ work in isolation. (Pure functions). You can bet when GM builds fuel injectors, they have a harness that they can place them in to make sure that JUST the injector gets tested... putting it in a car and driving around is the wrong approach.
The same is often true for software. Ensuring that things integrate is vital. Ensuring that functions run in isolation also has value.
Absolutely, whenever there is hardware in the loop, the more of the system you test with the more expensive the test is and the harder it is to isolate issues.
That’s not to say you can skip end to end testing, but the more testing pushed to the complete system level will drive up cost and schedule.
If I were to put a fuel injector (system under test) into a vehicle, and drive it around a parking lot (E2E test), only to have the vehicle stall out. Is the fuel injector at fault? How would you prove that?
No, you pull out the component and you "bench test" it. Believe me, after many MANY hours spend fighting bugs where the extensive E2E suite passed, we finally had to go back to adding in unit tests for the small units under the surface. Not all... but many places.
This makes sense when the cost of running the integration tests is high due to real hardware being in the loop. But for most business software this isn't the case and e2e tests aren't actually costly.
Looking at a previous project we ran full e2e tests from an end user perspective in three major browsers in less than 15 minutes at a cloud cost of only a few hundred dollars per month. Compared to the cost of developers on the project that was a negligible amount.
Adding more unit tests would make the feedback cycle shorter on some changes, but also add development time to create and maintain those tests. So in line with the article we did so only sparingly on a few critical paths that were likely to cause issues. The rest was well covered by e2e tests.
Agree w/the author that the concept of "unit" often hurts test quality.
You should be striving to balance the long-term usefulness of your tests with the debuggability of those tests. In my experience, those tests are what most people would call "integration tests" (although that name, like so much terminology in the testing world, is confusing and poorly defined.)
You want to get the tests up at as high a level of abstraction as possible where the API and correctness assertions are likely to survive implementation detail changes (unlike many unit tests) while at the same time avoiding the opaque and difficult to debug errors that come with end-to-end testing (again, the language here is confusing, I assume you know what I mean.)
Eh, I knew this would somehow come back to that Kent C Dodds stuff. I can respect the guy, but his opinion on valuable testing differs from mine and quite a lot of others. His opinion is "the more your tests represent the way your users use your app, the better they are". For E2E tests, yes, absolutely. For unit tests, no... not at all. As a very short example, an E2E test will prove that yes, I did indeed show a button on the screen. A unit test will prove the code that put it there worked. The code can "fail open" for many reasons. (In JavaScript for example, forgetting an `await` keyword can return a truthy value!).
The worst part about it is that he called himself a thought leader, called his approach a "best practice" and had nothing really to back that up. Now people go around repeating it all the time. It's frustrating.
That’s not in line with the original definition of unit test. It also sounds like a pointless test using your definition. There is no advocacy for writing pointless tests.
It's pointless if one is just trying to put "points on the board" and see a bunch of green in their console. Knowing what to test is important. Should every function be tested? Probably not. Should every function that has conditional logic be tested? Maybe? I totally get that integration tests have immense value. Where I disagree is that, "the closer your tests are to the way a user would use your app" is a good way to use testing. Functional/Acceptance/E2E tests are but one tool in an aresenal. Telling new software developers to ignore 25 years of history because React components were hard to test in isolation (Enzyme - the package Kent's React Test Library is built to replace) is irresponsible. Telling them that it's "best practice" is disingenuous.
If you have an E2E test that tests the button behavior thoroughly (i.e. not just showing it, but the fact that, when it's clicked, the app state changes accordingly), then you have a test showing that "code that put it there worked".
```
function setButtonState() {
const res = this.fetchSomeResult(); // returns an unfulfilled promise
if (res) {
return true;
}
return false;
}
```
as a unit test would quickly fail when you want it to. An E2E test would pass, but for the wrong reasons. This has nothing to do with app state, and everything to do with not being scientific, and testing all sorts of variable things at once just because, "that's what the user sees".
The E2E test would test the user-observable behavior that hinges on this function returning true or false.
If there's no such observable behavior, then why does it matter?
The appeal to "being scientific" doesn't make much sense to me, either. But if you really want to go there, being scientific is all about app state since that's literally all there is to the app.
Respectfully, I disagree, though maybe we haven’t yet articulated enough details to know exactly where we disagree.
I think underlying Kent’s statement is an observation that is undeniably true. The closer a test is to the actual way the software will be used, the better it is in a number of ways, _all else being equal_. All else is never fully equal, which is why everything is about tradeoffs.
But when a test aligns with how the software will be used, there are a whole bunch of synergies and benefits. Simplicity increases, clarity increases, you start getting multiple payoffs for each bit of effort you invest.
I’m sure people cargo cult it and lose track of the tradeoffs, like anything else, but there’s a solid point under there. And I agree with him, it is valuable enough to be called a “best practice.”
The point I often try to make is just how easy it is to get the right result, for the wrong reasons. (especially in loose languages like JavaScript). It goes from making more scientific proofs to, "I saw what I wanted to see on the screen - I can't exactly say why, but I think it's because my code is sound".
It's the equivalent to, "this new flea killer kills twice as many fleas". "What did you change?" "Four different things", "which one ended up being the thing that actually made it work?", "dunno, but now we're seeing twice as many fleas dead!".
The thing that drives me nuts about these threads is that people vastly overestimate their ability to write code that isn't broken. I distrust myself, and therefore I write basic unit tests which validate the code I've written to prove that I haven't screwed up.
Such unit tests are quick to write and don't change frequently because they run against low-level building blocks — so they are not a major maintenance burden. The ROI on this kind of testing is very high.
I don't like being told by Dodds and these other unit-test haters that I shouldn't be doing this. In my experience, organizations suffer much more when developers eschew such tests, being quite naturally overly optimistic about the reliability of their work.
Writing low-level unit tests doesn't prevent you from writing integration or end-to-end tests, and those are important too. I'm not here to devalue higher-level testing, and I'd appreciate it if advocates for higher-level testing likewise didn't go out of their way to devalue unit tests.
My tests might of course be faulty, but statistically speaking adding some will reduce the number of bugs in the main code that get through. The tests don’t have to be perfect to have a positive impact; in fact my impression is that the lowest-effort tests have an outsized impact and a very high ROI.
Always write functional tests first. Doesn't matter if they are slow - you still want something that faithfully captures the specified behavior and allows you to detect regressions automatically.
Then, if your resulting test suite is too slow, add finer-grained tests in areas where the perf benefits of doing so dwarf the cost of necessary black-boxing.
Getting down to the level of individual classes, never mind functions - i.e. the traditional "unit tests" - should be fairly rare in non-library code.
I'm not trying to be confrontational, but even "business logic" inside a project is often clearly separated between "library" and "glue, aka controllers or whatever" - and that's kinda the point where I disagree.
Not accepting "but the unit tests work" - 100% agreed. Not being able to slice it that the thing that could be accessed by the current public API or a different public API be tested on its own (and often relatively pure) is not how I want to write my software or tests.
Thing is, trying to make everything individually testable in tiny bits takes a lot of effort that wouldn't otherwise be needed. Furthermore, with all the mocking this usually requires, you end up with tons of extraneous code that now needs to be maintained as well (and by very nature of mocks, you'll have to fix them every time the corresponding real code changes behavior observable across the boundary). In practice, the ostensible benefit of knowing exactly what the problem is as soon as you see the test failure is usually dwarfed by all these costs. And, at the same time, the more mocks you have, the less likely your tests are to catch real issues.
Now, as you rightly note, in many cases there's already an obvious component boundary that is inherent in the design. And then it makes perfect sense to write functional tests for individual components along those boundaries. And in some cases, the component may be "library-like", and those tests will end up looking a lot like your typical unit test, where individual classes and even methods are covered one by one. But most classes and methods in a typical app aren't like that.
Most of the systems I build use a database on which all logic depends, and often a network connection.
I've worked on systems where these aspects were mocked, and they eventually grind to a halt because of the effort required to make the tiniest change.
First of all you need a way to create a pristine database from code, preferably in memory. Second nested nested transactions are nice, since you can simply rollback the outer transaction per test case; otherwise you need to drop/create the database which is slower.
For networked servers, an easy way to start/stop servers in code and send requests to them is all you need.
Given these pieces, it's easy to write integration tests that run fast enough and give a lot of bang for the buck.
TDD is even more rare for me, I typically only do that when designing API's I'm unsure about, which makes imagining user code difficult. And fixing bugs, because it makes total sense to have a failing test to verify that you fixed it, and that it remains fixed.
I still find the skepticism around TDD weird. Except for a few pretty niche scenarios (e.g. it's experimental code or manual testing is cheaper for some obscure reason) i dont really see the point of not doing it.
I especially dont see what is gained by writing the test after.
I hate coding fundamentalism with a passion too. The only thing I get really religious about in coding is the importance of trade offs.
The cost/benefit of writing a test before just consistently exceeded doing it after for me.
Same for integration, e2e or unit tests (there's never been a rule that says you can only TDD with a unit test).
The cost/benefit trade off for tests with mocks vs. database is a different topic - orthogonal to the practise of red/green/refactor, and one where IMO the trade offs are much less obvious.
I honestly can't see how writing integration tests before code would even look in practice, that puzzle usually isn't even close to finished at that point in time.
It sometimes makes sense for unit tests; I'll occasionally do that when I'm unsure about the API of the code I'm writing since it allows me to spend some time in the user's shoes.
> I especially dont see what is gained by writing the test after.
I assume you mean versus writing it first, rather than versus not writing it at all.
I've found that TDD works well for bottom-up coding, but not so well for top-down.
With bottom up, I can write the test for a piece at the bottom, write the code to pass the test, and move on. With top-down, if I write the test first, it might be a long while before I have that top-level working, because the bottom bits don't exist yet.
When I feel it's better to write things top-down, I'll often use TDD for the bottom bits I need to write, but for the bits above that, I'll write the tests "on my way back up".
"I especially dont see what is gained by writing the test after."
The greatest value in tests is that they help prevent future changes from breaking existing functionality. Writing the test after you write the implementation is equally useful for achieving that as writing the test before you write the implementation.
Sometimes you need the unit before you can unit test.
So you end up writing the unit's boilerplate, that doesn't yet do anything but needs to compile/run for a suite to test it throwing a adhoc error of the notImplemented sort or whatever, then break flow to set the test suite and check the reds (that aren't telling you anything useful because of course it's red), then actually write the code.
I find it flow-breaking and cumbersome for little to no gain.
For additions to existing code I find TDD more useful, but even then it's not unusual to decide to move logic around until deciding a final structure, so the units you end up with might be solidified later in time. Writing tests for scraped units is a waste of time then.
Not necessarily. On plenty of projects I have done 100% TDD and never written a single low level unit test.
The type of test is, in my mind, a completely different topic to red-green-refactor and for the decision about which one to write I follow a set of rules which is also unconnected.
TDD is just red-green-refactor. It works with any test.
If you value red green refactoring then you should write the tests first.
I only use that technique for pieces of code that really fit that well - usually functions that have a very strong relationship between their input and output - so I'll write tests first for those, but not for most of my other stuff.
Well ok...but then what kind of code doesnt it fit well?
Almost every user story I follow in production code follows the form of given/when/then scenario which can always be transformed into a test of some kind (e2e, integration, sometimes even unit).
Where it's something like "do x, y and z and then a graph appears" I find TDD with a snapshot test with, say, playwright works best.
I'm talking about strict test-first development here, where you write the tests before you write the implementation.
If you're using snapshot tests (a technique I really like) surely you can't write the tests before the implementation, because you need the implementation in order to generate the snapshot?
(This is what I hate about the term TDD: sometimes it means test-first, sometimes it doesn't - which leads to frustrating conversations where people are talking past each other.)
You need the final implementation before taking the final snapshot but you can write the entire test up front (given/when). The snapshot artefact is generated not written (often in a different file entirely), so Id argue it still fits the definition cleanly.
I agree that "unit test"/"integration test" as a definition sucks horribly and leads to people talking past each other, but I think with TDD the main issue is that lots of people have developed a fixed and narrow idea of the kind of test you are "supposed" to write with it which makes the process miserable if the type of code doesnt fit that type of test.
The whole idea of a unit test being "the" kind of "default" test and being "tests a class/method as a unit" definitely needs to die.
I need something to work with before I can write the test. So my order tends to be: get the code working first with the simplest case, and by using it I know that simple case is working, then use that to write the first couple of tests. Only then would I expand the tests to the cases not written yet and to a TDD style.
This order also helps verify I didn't typo something in the test itself and end up TDD-ing myself into broken code.
> Im sensing a pattern in the answers to my question though. I keep getting "well, if you assume TDD is only done with low level unit tests..."
Completely wrong.
Even with your example, there's an initial exploratory stage where you're still figuring out the interface that the tests would use. I, personally, am not capable of using something that doesn't exist. I have to make that initial version first before I can use it in a test.
Quick edit aside: This is also why I rarely work top-down or bottom-up, I work mostly throughline - following the data flow and jumping up and down the abstraction stack as needed.
Im not sure quite why you feel you always need to write code before sussing out what an API or UI should look like but it seems like a very expensive habit to me.
What happens when you then show it to stakeholders (e.g. other teams consuming your API, customers or UX people) or and they tell you to change it again?
Rewrite everything again?
Thats gonna be reaaaaaaaalllly labor intensive and could damage your code base too.
Im equally perplexed about why people dont try to build top down. It's one of those few things in programming that always makes sense regardless of circumstance.
If Izkata is anything like me they write code as part of their exploratory design process with no intention of showing it to anyone else until they've iterated their way to a design that they like.
> What happens when you then show it to stakeholders (e.g. other teams consuming your API, customers or UX people) or and they tell you to change it again?
> Rewrite everything again?
> Thats gonna be reaaaaaaaalllly labor intensive and could damage your code base too.
Why would I do that? Only the thing they have issue with would need to be changed, it wouldn't take any longer than another way of doing it.
You seem to have forgotten what I said, something needs to exist for me to work with. Well, in this "stakeholders want something changed", something exists. It's not a rewrite from scratch.
If you change the spec (e.g. changing the contract on a REST API), you will probably need to consult to make sure it aligns with everybody's expectations. Does the team calling it even have the customer ID you've just decided to require on, say, this new endpoint?
>You seem to have forgotten what I said, something needs to exist for me to work with.
No. I'm assuming here that a code base exists and that you are mostly (if not 100%) familiar with it.
> Red-green-refactor can also provides live feedback about whether your code is behaving correctly as you write it.
No, it provides live feedback about whether your code is passing your tests
If you have written your tests poorly then set out to make the tests pass, then your tests become the target rather than the correct behavior
If you are continuously updating your tests while your code evolves because you missed test cases or your understanding of the behavior has improved, then writing the tests first didn't actually give you any value. In fact it just wasted a lot of your time
Write the code
Manually test to verify correctness and to identify the test cases you have to write
THEN write tests to protect against regressions
A small community of programmers, with a disproportionately large audience, foretold that practicing test-driven development would produce great benefits; over twenty five years the audience has found that not to be the case.
Compare with "continuous integration" - here, the immediate returns of trying the proposed discipline were so good that pretty much everybody who tried the experiment got positive returns, and leaned into it, and now CI (and later CD) are _everywhere_.
As for what is gained, try this spelling: test driven development adds load to your interfaces at a time when you know the least about the problem you are trying to solve, which is to say the period where having your interfaces be flexible is valuable.
And thus, the technique gets criticism from both ends -- that design work that should have been done up front is deferred (making the design more difficult to change, therefore introducing costs/delays), and that the investment is being made in testing before you have a clear understanding for which tests are going to be sensitive to the actual errors that you introduce creating the code (thereby both increasing the amount of "waste" in the test suite, in addition to increasing the risk of needing test rewrites).
The situation is further not improved by (a) the fact that most TDD demonstrations are problems that are small, stable problems that you can solve in about an hour with any technique at all and (b) the designs produced in support of the TDD practice aren't clearly an improvement on "just doing it", and in some notable cases have been much much worse.
So if it is working for you: GREAT, keep it up; no reason for you not to reap the benefits if your local conditions are such that TDD gives you the best positive return on your investment.
>As for what is gained, try this spelling: test driven development adds load to your interfaces at a time when you know the least about the problem you are trying to solve
If Im writing a single line of production code I should know as much as possible what requirements problem Im actually trying to solve with it first, no?
This is actually dovetails into a benefit to writing the test first. If you flesh out a user story scenario in the form of an executable test it can provoke new questions ("hm, actually I'd need the user ID on this new endpoint to satisfy this requirement...") and you can more quickly return to stakeholders ("can you send me a user ID in this API call?") and "fix" your "requirements bugs" before making more expensive lower level changes to the code.
This outside-in "flipping between one layer and the layer directly beneath it" is very effective at properly refining requirements, tests and architecture.
>And thus, the technique gets criticism from both ends -- that design work that should have been done up front is deferred
I dont think "design work" should be done up front if you can help it. I've always felt that the very best architecture emerges as a result of aggressive refactoring done within the confines of a complete set of tests that made as few architectural assumptions as possible. Why? Coz we're all bad at predicting the future and it's better if we dont try.
> Most of the systems I build use a database on which all logic depends, and often a network connection.
Why do you believe this is relevant? Unit tests target units of code, and check if the unit of code verifies specific invariants for specific combinations and ranges of input values. The concept of a network or a database does not exist in unit tests. In fact, you are expected to abstract away these aspects.
> I've worked on systems where these aspects were mocked, and they eventually grind to a halt because of the effort required to make the tiniest change.
This scenario is totally unrealistic if you knew what you are doing.
The fact alone that you're talking about spinning up a database in the context of unit tests already raises red flags.
Even if somehow you're talking about integration and end-to-end tests, none of the perceived problems you're mentioning are even a challenge, let alone a problem.
> Given these pieces, it's easy to write integration tests that run fast enough and give a lot of bang for the buck.
That's perfectly fine. It's besides the point though, and completely ignores the whole point of unit tests, which is to have a bunch of fast tests that check if your code continues to do the right thing between changes. Unit tests also compel developers to think through how they create/update code to prevent them from introducing handled scenarios.
There are many reasons why the test pyramid is a thing, and why the unit test layer includes far more tests than any other layer.
This works fine if the difficult part of your software is in nicely abstracted boxes, but in my experience the content of the boxes is usually fairly trivial: you can write a huge number of tests for them, but they're rarely where problems occur. The integration and interaction are where you get by far the most value out of testing (it's where most errors will occur: units are usually small enough to fit in your head while working on them, but the system as a whole it not), and for that you should abstract out the bare minimum needed to control your test environment and get acceptable performance, not mock absolutely everything by default.
Unit tests are the last tests that should be added to any system, and I've yet to see any system where the effort isn't better spent on integration testing (maybe once you're as thorough as sqlite?).
> This works fine if the difficult part of your software is in nicely abstracted boxes, but in my experience the content of the boxes is usually fairly trivial: you can write a huge number of tests for them, but they're rarely where problems occur.
That's perfectly fine. It's also the basis of the whole test pyramid concept.
One of the main reasons why you have some problems caught in integration tests instead of unit tests is that developers introduced bugs by failing to characterize how some components actually work and thus fail to design both the units of code and the tests that go along with them.
This is not a problem caused by unit tests. This is a problem caused by developers failing to design and implement components, which consequently means they failed to add relevant tests.
> The integration and interaction are where you get by far the most value out of testing (...)
It's perfectly fine to think like that. The test pyramid concept exists for a reason. However, when looking at failures you need to understand their root causes. If a fault is triggered with an integration test but not with a unit test, that tells you two things:
- one or more unit of code is broken,
- your unit tests fail to cover a very predictable failure mode.
Each of these items by themselves reflect a fundamental failure in the way developers implemented their test strategy.
> Unit tests are the last tests that should be added to any system (...)
This take is fundamentally wrong, to the point that your software design process needs to be completely and fundamentally broken to believe it's reasonable that unit tests can be added post-facto after skipping the whole design process.
I mean, I design my code, I'm just uninterested in distorting the design of the code to make it a little bit easier to write the least valuable kinds of tests (you can always unit test some code, it's just a matter of how extreme you want to go in your test harness - I believe it's the test harness that should change to fit the natural design of the code, not the other way around).
>If a fault is triggered with an integration test but not with a unit test, that tells you two things:
In most cases the interaction between units is far less predictable than the behavior of a given unit. And it's often not obvious which unit is 'broken'. Usually the units are doing what they were intended to do, it's the combination of behaviors that are broken (which yes, means 'the design is broken' - that's usually the harder part than writing the code!). No amount of unit tests will catch that. The integration tests will catch problems units, in contrast (admittedly, not 'accidentally working' cases, but I find it's rare anyone else cares about those).
> I mean, I design my code, I'm just uninterested in (...) the design of the code (...) to write (...) tests
You're just admitting you're the root cause of these problems, and basic testing strategies play no role in that failure.
> In most cases the interaction between units is far less predictable than the behavior of a given unit.
Irrelevant. You write tests to verify how units of code should respond to specific input values, because your units of code will for a fact handle those sets of input values in Production and you want to prove they can and will handle them.
> And it's often not obvious which unit is 'broken'.
It is, if you know what you're doing. That's why unit tests matter, because they prevent you from hand-waving over the problems you're creating for yourself. You need to think through what are the preconditions and post-conditions of each unit of code, add checks, and make sure your unit of codes comply with those.
These errors can only be a mystery if you have no idea what are these preconditions and post-conditions, enforce none, test none, and just allow failures to cascade and errors to be swallowed without having any idea of what's going on and where the problem starts and finishes.
> No amount of unit tests will catch that.
This is simply wrong.
Consider for example null reference exceptions. Do you think it's impossible to tell with tests where a null reference is passed where it shouldn't?
My first job interview after college. It was a pair programming session but I would not touch the keyboard. I would dictate and the interviewer would type. I had to implement a dynamic array in java using TDD in one hour. From time to time I would break protocol and jump into implementation before the tests. Overall it went good. I finish the job. The interviewer however assesed that he was afraid I would not actually use TDD on the job, he was spot on. I did not get the job.
This is hilarious because it's a perfect interview (detects precisely the thing they're testing for) but also provides total adverse selection because the thing they're testing for is ridiculous.
Ive done this too. The exercise wasnt arrays (Im militant about only setting very realistic tasks). My task required modifying existing production-like code and tests.
My hope was always that the candidates would do TDD where it seemed simple and obvious to do so. It was actually pretty rare but the candidates that defaulted to doing that always ended up being better in my opinion. They were always made offers higher than my company could afford elsewhere (so i guess in others' opinions too).
In this thread https://news.ycombinator.com/item?id=43060636 I pondered why most people dont default to TDD for production code and the answer invariably seemed to be "we didnt think TDD was a thing you could do with integration/e2e tests".
I think having tests for all your diffs at the level of published commits/change lists/etc is totally reasonable for software you really care about. What's counterproductive is practicing TDD at the level of individual editor operations.
If I'm fixing a bug, I start by writing a test that reproduces the bug. If I can't do that, I fix the test harness until I can. Then I implement the change, making mental notes of each intermediate bug I think about along the way - things like "I should be careful to name this distinctly so that it's not confused with this other value in scope that has the same type". After that, I cull down that list until it's reasonable and not totally paranoid, and write tests covering those cases. Same thing for any bugs in in-progress code caught by manual testing, fuzzers, etc.
If you have discipline and use version control, you don't need to write tests before you write the actual code to get the same level of coverage as TDD and you waste a lot less time. I've often figured out late in the game how to make something a compile time failure rather than a runtime one - time to delete all those tests written along the way? Encode them all as negative compilation tests? Fundamentally the goal of testing is to describe what behaviors of the software are intentional rather than incidental, and to detect bugs that might be introduced by future changes to the software - TDD mixes both concerns and doesn't put any emphasis on preventing future bugs specifically.
Maybe other people work on different types of things and TDD is great for them, but I write primarily infrastructure code where correctness is critical and I have the luxury of time, and TDD doesn't produce better results for me. This is a case TDD feels like it should work well for, but in my experience it doesn't improve correctness, maintainability, or speed of delivery - at least compared to the alternative I described. I'm sure there's a universe of teams with sloppy practices out there that TDD would be an improvement for, but it's not helpful for me.
>I've often figured out late in the game how to make something a compile time failure rather than a runtime one
This is actually a good (albeit somewhat niche) reason to not write a test scenario at all, but it's still not a great reason to write a test after instead of before.
>Fundamentally the goal of testing is to describe what behaviors of the software are intentional rather than incidental
Yup. A test scenario which is of no interest to at least some stakeholders probably shouldnt be written at all.
This is again about whether to write a test at all, though, not whether to write it first.
>TDD mixes both concerns
I dont think writing a test after helps unmix those concerns any better.
In fact it's probably a bit easier to link intentional behavior to a test while you have the spec in front of you and before the code is written.
I find people who write test after tend to (not always, but strong tendency) fit the test to the code rather than the requirement. This is really bad.
>Maybe other people work on different types of things and TDD is great for them, but I write primarily infrastructure code where correctness is critical and I have the luxury of time
Assuming Im understanding you correctly (you're building something like terraform?), integration tests which run scenarios matching real features against fake infra would seem to be pretty useful to me.
So...why wont you write tests with that harness before the code? Im still unsure.
The only thing "special" about that type of code that i can see (which isnt even all that special) is that unit tests would often be useless. But so what?
>This is actually a good (albeit somewhat niche) reason to not write a test scenario at all, but it's still not a great reason to write a test after instead of before.
But the before-test is strictly negative - it's a waste of time (deleted code, never submitted) and it possibly slowed down development (had to update the test as I messed with APIs).
>Yup. A test scenario which is of no interest to at least some stakeholders probably shouldnt be written at all.
And yet I see TDD practitioners as the primary source of such tests - if you are dogmatically writing a test for every intermediate change, you will end up with lots of extra tests that assert things in order to satisfy the TDD dogma rather than the specific needs of the problem. Obviously this can be avoided with judgement - but if you have sound independent judgement you don't need to adhere to specific philosophies about the order you make changes in.
>In fact it's probably a bit easier to link intentional behavior to a test while you have the spec in front of you and before the code is written.
When implementing to a spec you are absolutely right, but a very small amount of software is completely or even mostly specified in advance.
>I find people who write test after tend to (not always, but strong tendency) fit the test to the code rather than the requirement. This is really bad.
I agree this can lead to brittle tests and lack of spec adherence, but if you are iterating on intermediate state and writing tests as you go, the structure of the code you wrote 30 seconds ago is very much influencing the test you're writing now.
Another issue is that fault injection tests basically require coupling to the implementation - "make the Nth allocation fail" etc. The way I prefer to write these is to write the implementation first, then write the fuzz test - add a few bugs in the implementation, and fix/enhance the fuzz test until it catches them. Fuzz testing is one of the best bang-for-buck testing methodologies there is, and in my experience it's very hard to write a really good fuzz test unless you already have most of your implementation, so you can ensure your fuzz tester is actually exercising the stuff you want it to.
>Assuming Im understanding you correctly (you're building something like terraform?),
I write library code for mobile phones, mostly in Java/Kotlin. I recently did some open source work (warning: I am not actually very proficient with C, any good results are from enormous time spent and my code reviewers, constructive criticism very much welcome). Here's a few somewhat small, contained changes of mine, so we can talk about something concrete:
This change alters a lock-free data structure to add a monotonicity invariant, when the space allocated is queried on an already-fused arena while racing with another fuse. I didn't add tests for this - I spent a fair bit of time thinking about how to do it, and decided that the type of test I would have to write to reliably reproduce this was not going to be net better at preventing a future bug, given its cost, than a comment in the implementation code and markdown documentation of the data structure. I don't know how I would really have made this change with a TDD methodology.
This change moves a memory layout - again, I don't know how I would have written a test for this, besides something wild like querying smaps (not portable) to see if the final page of the arena allocation had faulted in.
I actually did this the other day on a piece of code. I was feeling a bit lazy. I didn't write the test and I figured that making the type checker catch it was enough. I still didn't write a test after either though.
Anecdotally I've always found that tests which cover real life bugs are in the class of test with the highest chance of catching future regressions. So even if it does exist, I'm still mildly skeptical of the idea that tests that catch bugs that compilers have been provoked into also catching are strictly negative.
>And yet I see TDD practitioners as the primary source of such tests
I find the precise opposite to be true. TDD are more likely to tie requirements to tests because they write them directly after getting requirements. Test-after practitioners more likely tie implementation to the test.
It's always possible to write a shit implementation-tied test with TDD, but the person who writes a shit test with TDD will write a shit implementation tied test after too. What did TDD have to do with that? Nothing.
>if you are dogmatically writing a test for every intermediate change, you will end up with lots of extra tests that assert things in order to satisfy the TDD dogma rather than the specific needs of the problem.
I find that this only really happens when you practice TDD with very loose typing. If you practice strict typing, the tests will invariably be narrowed down to ones which address the specific needs of the problem.
Again - without TDD and writing the test after, loose typing is still a shit show. So, I see this as another issue which is about something separate to TDD.
>Obviously this can be avoided with judgement - but if you have sound independent judgement you don't need to adhere to specific philosophies
I think this is conflating "TDD is a panacea" with "if it is valuable to write a test, it's always better to write it before". I've never thought the former, but the examples you've listed here look to me only like examples of where TDD didn't save somebody from making a mistake that was about a separate issue (types, poor quality test). None of them are examples of "actually writing the test after would have been better".
>When implementing to a spec you are absolutely right, but a very small amount of software is completely or even mostly specified in advance.
Why on earth would you do that? If I write even a single line of production code I have specified what that line of code is going to do. I have watched juniors flail around and do this when getting vague specs but seniors generally try to nail down a user story tight with a combination of code investigation, spiking and dialog with stakeholders before writing code that they would otherwise have to toss in the trash can if it wasn't fit for purpose.
To me this isn't related to TDD either. Whether or not I practice TDD, I don't fuck around writing or changing production code if I don't know precisely what result it is I want to achieve. Ever.
Future requirements will probably remain vague but never the ones I'm implementing right now.
>I agree this can lead to brittle tests and lack of spec adherence, but if you are iterating on intermediate state and writing tests as you go, the structure of the code you wrote 30 seconds ago is very much influencing the test you're writing now.
Only if the spec is changing too. This sometimes happens if I discover some issue by looking at the code, but in general my test remains relatively static while the code underneath it iterates.
This obviously wouldn't happen if you wrote implementation-tied tests rather than specification-tied tests but... maybe just don't do that?
>Another issue is that fault injection tests basically require coupling to the implementation
All tests requiring coupling to implementation in some way. The goal is to lo...
>Anecdotally I've always found that tests which cover real life bugs are in the class of test with the highest chance of catching future regressions. So even if it does exist, I'm still mildly skeptical of the idea that tests that catch bugs that compilers have been provoked into also catching are strictly negative.
Maybe this is a static typing thing - if the test won't build because you've made the bug inexpressible in the type system, what test can you even write?
>If I write even a single line of production code I have specified what that line of code is going to do.
>combination of code investigation, spiking and dialog with stakeholders
Does "spiking" in this context mean writing code without TDD?
>I think this is conflating "TDD is a panacea" with "if it is valuable to write a test, it's always better to write it before".
That's a weaker formulation of TDD than I've seen espoused and practiced, which is usually something more like "before you make any behavioral changes to the code under test, you must write a test that fails; then make your edit so the test passes, and repeat". The problem with your approach at least for me is that until I've messed around a bit in the code seeing what's the best approach is to solving a problem, I don't know what the best structure for a final test is, or whether I can make the change in a way that leverages existing tests to detect the bug.
> I'm not really sure why fault injection should be treated as special.
Suppose you write a test reproducing a bug that reacts to allocation failing. One common way to do that is to inject an allocator that fails on the specific allocation call you had a bug with. But how do you target that specific call? One way is to make the Nth allocation in a test fail - but even slight changes to the production code will make this test start testing something completely different. The solution is to have a fuzz test that injects failed allocations randomly per run - now you can be reasonably confident that even if the prod code changes over time, your fuzz test will still all the allocation sites, preventing regression. I don't see why it's beneficial to write this first or last. Under your model the instrumented allocator setup is a replacement for the single test case that reproduces the original bug.
>If I'm reading it correctly, this looks like a class of bugs we discussed that is fixed by tightening up the typing. In which case, no test is strictly necessary, although I'd argue that it probably would not hurt either.
No, the original "bug" is that if you have two calls to SpaceAllocated and the second one races with a Fuse call on a other thread, you can see a smaller value result than before. This behavior wasn't guaranteed by the public API (so not a bug) but it's desirable to add. The fix is replacing a singly linked list with a doubly linked list, using tagged pointers to avoid adding storage cost for the second set of links. A test could be written for this; I could inspect the allocated addresses of three arenas and fuse them in a specific order, but I could not actually reproduce the ordering required without a test harness that allows full control of thread execution interleaving - which would be a huge amount of work and in my opinion not actually prevent future bugs, since that interleaving only meaningfully exists in the previous implementation. The full set of possible interleavings is way too large to productively explore. If I had started by writing the test, I would have spent a bunch of time messing around before giving up; because I did the implementation first, I had a much more informed idea of what would be required to test it, and changed my plan for preventing future bugs to documentation.
>I can't tell if this is refactoring or you're fixing a bug. Is there a scenario which would reproduce a bug?
Disclaimer: I don't have a lot of comments on TDD as a whole, other than that most software I write is very exploratory (I'll throw away the first 3-10 drafts, and by no means does that mean it takes 3-10x as long to write), and the best language for me for that exploration is often the language of the actual software I'm writing. TDD, in that environment, doesn't seem very applicable since the whole point is that we don't know what's actually possible (or, when it's possible, if the tradeoffs are worth it).
The author has a lot of opinions about testing though which conflict with what I've found to work in even that sort of dynamic environment. Their rationale makes sense on the surface (e.g., I've never seen a "mock"-heavy [0] codebase reap positive net value from its tests), but the prescription for those observed problems seems sub-optimal.
I'll pick on one of those complaints to start with, IMO the most egregious:
> Now, you change a little thing in your code base, and the only thing the testing suite tells you is that you will be busy the rest of the day rewriting false positive test cases.
If changing one little thing results in a day of rewriting tests, then either (a) the repo is structured such that small functional changes affect lots of code (which is bad, but it's correct that you'd therefore have to inspect all the tests/code to see if it actually works correctly afterward), or (b) the tests add coupling that doesn't exist otherwise in the code itself.
I'll ignore (a), since I think we can all agree that's bad (or at least orthogonal to testing concerns). For (b) though, that's definitely a consequence of "mock"-heavy frameworks.
Why?
The author's proposal is to just test observable behavior of the system. That's an easy way to isolate yourself from implementation details. I don't disagree with it, and I think the industry (as I've seen it) discounts a robust integration test suite.
What is it about "unit" tests that causes problems though? It's that the things you're testing aren't very well thought through or very well abstracted in the middle layers. Hear me out. TFA argues for integration tests at a high level, but if you (e.g.) actually had to implement a custom sorting function at your job would you leave it untested? Absolutely not. It'd be crammed to the gills with empty sets, brute-force checking every permutation of length <20, a smattering of large inputs, something involving MaxInt, random fuzzing against known-working sorting algorithms, and who knows what else the kids are cooking up these days.
Moreover, almost no conceivable change to the program would invalidate those tests incorrectly. The point of a sorting algorithm is to sort, and it should have some performance characteristics (the reason you choose one sort over another). Your tests capture that behavior. As your program changes, you either say you don't need that sort any more (in which case you just delete the tests, which is O(other_code_deleted)), or you might need a new performance profile. In that latter case, the only tests that are broken are associated with that one sorting function, and they're broken _because_ the requirements actually changed. You still satisfy O(test_changes) <= O(code_changes); the thing the author is arguing doesn't happen because of mocks.
Let's go back to the heavily mocked monstrosities TFA references. The problem isn't "unit" testing. Integration tests (the top of a DAG), and unit tests (like our sorting example, the bottom of a DAG) are easy. It's the code in between that gets complicated, and there might be a lot of it.
What do we do then?
At a minimum, I'd personally consider testing the top and bottom of your DAG of code. Even without any thought leadership or whatever garbage we're currently selling, it's easy to argue that tests at those levels are bot...
> The argument for isolating the units from each other is that it is easier to spot a potential bug. (...) In my opinion, this does not pay out because of the huge amount of false positive test cases you get and the time you need to fix them. Also, if you know the code base a little you should have an idea where the problem is. If not, this is your chance to get to know the code base a little better.
This is at best specious reasoning, and to me reflects that the blogger completely misses the point of having tests.
To start off, there is no such thing as a false positive test. Your tests track invariants, specially those which other components depend on. The whole point of having these tests is to have a way to automatically check for them each and every single time we touch the code, so that the tests warn us that a change we are doing will cause the application to fail.
If you somehow decide to change your code so that a few invariants break, these are not "false positives". This is your tests working as expected and warning you that you must pay attention to what you are doing so that you do to not introduce regressions.
It's also completely mind-boggling and absurd to argue that "knowing the code" is any argument to avoid tracking invariants. The whole point of automated test suites is that you do not want the app to fail because you missed any detail or corner case or failure mode. Knowing the code does not prevent bugs or errors or regressions.
I'm perplexed by the way we have people write long articles on unit tests when they don't really seem to understand what they are supposed to achieve.
Maybe in the ideal case, but in practice, a lot of unit tests track much more than invariants other components depend on.
Example: a function makes two RPC calls to query two independent statuses, which are of course mocked for the unit tests. Due to the the test library used, unit tests expect query A first and query B second. The function is refactored, and it now queries B first, so unit tests start to fail. This is 100% false positive - nothing in the real world cares if A or B are queried first, those actually go to systems which don't even know each other. And yet, the change author now has to fix dozens of failed tests.
Another example: there is an event-based system, and unit tests hardcode number of main loop executions ("emit event A; poll; poll; poll; ensure event B arrives"). A change which adds one more poll cycle will have zero effect on the real app, but causes many tests to break. It's a false positive.
Yet another example: there are random numbers involved, so test driver makes sure to seed RNG to known value at start of each test. A single call to random() is added somewhere early in the code and boom! each test explodes. Another hundreds of false positives.
I could go on an on... Should those tests be rewritten to be more robust and only check things that really matter? Yep. Does real-life code has fragile unit tests like this? Yep, almost every single codebase I have seen.
It mostly comes from observing the reality of how tests are used rather than idealism. In a lot of "test-first" codebases, the majority of tests are garbage. All they check is that the code is still structured identically to the original implementation. This is the reality that people encounter, so of course it's what they talk about. Why would they talk about some ivory tower idea instead?
> It mostly comes from observing the reality of how tests are used rather than idealism. In a lot of "test-first" codebases, the majority of tests are garbage.
At most, you can only arrive at that conclusion by observing your personal reality. If your teams crank out crappy code that's error-prone and untestable with their crappy tests, and can't manage to fix either their code or the tests, then naturally they end up living in the reality they created for themselves.
Back in the real world, some projects renowned by their stability and robustness go out of their way to single out their test-drive approach to software design and development as the fundamental reason their software is stable and robust.
What do you think is the difference?
> All they check is that the code is still structured identically to the original implementation.
Unit tests don't test structure. Their whole point is that they do not reflect structe
Unit tests only cover the behavior that's expected from a specific unit of code, and they only check for the invariants relevant to that specific unit of code.
Perhaps this is a telltale sign you should rethink what you're doing.
A shit test written before writing the code is still a shit test. Mimetic tests arent be any better written after the code either.
If I had to choose between 1) always writing specification-linked tests that make as few architectural assumptions as possible and 2) TDD, sure, I'd pick 1 every time.
People don't like to write tests so any argument saying you need fewer of them goes down real smooth. The details don't matter so much.
But when software actually needs to be made at scale and with high quality you almost always have lots of tests.
Statistically, the average developer is not writing software anymore where corner cases matter. They're writing various permutations on standard web apps. Most web apps are pretty buggy, even ones from major corporations. So stakes are relatively low for having failures.
People writing file systems or crypto routines are going to have an easier time understanding the value of tests.
> This is at best specious reasoning, and to me reflects that the blogger completely misses the point of having tests.
I mean, yeah. Most devs don't understand the point. It turns into a check-boxing exercise. And most devs are vaguely smart and can figure out how to do a check-boxing exercise with minimal effort while evading the point of doing it.
Relying on people who don't care enough to write good code to care enough to write good tests has a certain irony to it.
> To start off, there is no such thing as a false positive test. Your tests track invariants, specially those which other components depend on. The whole point of having these tests is to have a way to automatically check for them each and every single time we touch the code, so that the tests warn us that a change we are doing will cause the application to fail.
I'll push back on this. I'm a veteran of a codebase where the tests broke all the time with tons of false positives. This was mostly because of over-mocking, and an abundance of assert X called with Y and Z tests. Even pure functions were mocked.
Every time I re-wrote a function implementation, even if it was a pure function and produced the same results, it would break tens of tests. Bonus points if the implementation of the thing that was mocked out had changed since the last time someone reviewed the tests and the way it was calling the function didn't do what the test assumed anymore.
> To start off, there is no such thing as a false positive test. Your tests track invariants, specially those which other components depend on.
They were just talking about test doubles / mocks above that. Those cause all sorts of risks with false positives and false negatives, for example by mocking out a function the code under test relies on, but the mock not being changed when the contract between those two pieces of code changes.
> They were just talking about test doubles / mocks above that. Those cause all sorts of risks with false positives and false negatives, for example by mocking out a function the code under test relies on, but the mock not being changed when the contract between those two pieces of code changes.
The point is that it's absurd to try to characterize the regressions you're introducing as "false positive"/"false negative".
Again, this is at best specious reasoning, and to me reflects that the blogger completely misses the point of having tests.
Your code has behavior. Your tests verify invariants associated with this behavior. If you purposely change that behavior, it stands to reason that your task also involves updating how your code checks for that behavior.
If you change behavior but fail to change how that behavior is checked, that is not a false positive or false negative. That represents a few problems you introduced: failing to update how that behavior is checked, and failing implement new behavior without checking if it works. Both reflect problems created by a developer failing to perform the basics of their work, and worse: blaming working tests for the mess he's creating.
Think about the issue: if a highway administration decides to redesign an intersection but fails to move/update its stoplights, does this means the old stoplights cause accidents with their false positive greens?
> Your code has behavior. Your tests verify invariants associated with this behavior. If you purposely change that behavior, it stands to reason that your task also involves updating how your code checks for that behavior.
If your tests don't fail before you update them, that's one false positive. Usually introduced by mocks that hide part of the behavior.
If your tests were passing but a bug is discovered in the code that should have been caught by the given tests, that's another false positive. Also possibly introduced by mocks, but hopefully is more like just a missing assert.
If your tests start failing with a correct code change and just straight need to be updated, that's a false negative (this is your example, halfway through the work).
In either false positive case if it was caused by mocks, those are bad tests and should be rewritten and not just updated.
> Think about the issue: if a highway administration decides to redesign an intersection but fails to move/update its stoplights, does this means the old stoplights cause accidents with their false positive greens?
Lights at an intersection aren't tests, they're part of the working system. Tests in this example would be something more like a central reporting system that's supposed to show what's going on at the intersection. The system (the intersection and lights) can be working totally fine, but the reporting system is showing it messed up (false negative) or messed-up lights can be causing crashes but the reporting system shows it's working fine (false positive). In both cases it should be fixed, but the fix is independent of what - if anything - should be done with the intersection and lights themselves.
Wow. If "unit" in "unit test" does indeed mean that the test itself should be able to run independent of the other tests then maybe I can get over my avoidance of calling them "unit tests"!
I dislike that term because the most valuable tests I write are inevitably more in the shape of integration tests - tests that exercise just one function/class are probably less than 10% of the tests that I write.
So I call my tests "tests", but I get frustrated that this could be confused with manual tests, so then I call them "automated tests" but that's a bit of a mouthful and not a term many other people use.
I'd love to go back to calling them "unit tests", but I worry that most people who hear me say that will still think I'm talking about the test-a-single-unit-of-code version.
"I call them 'unit tests' but they don't match the accepted definition of unit tests very well." -- Kent Beck, __Test Driven Development By Example__
The short version is that "unit test" did actually mean something (see Glenford Myers, __The Art of Software Testing__ or Boris Beizer, __Software Testing Techniques__), although it wasn't necessarily clear how those definitions applied to object-oriented programming (see Robert Binder, __Testing Object-Oriented Systems__).
The Test-First/TDD/XP community later made an effort to pivot to the language of "programmer test", but by the time that effort began it was already too late.
So I think you should continue to call your tests "tests" (or "checks", if you prefer the framing of James Bach and Michael Bolton).
As best I can tell - there's no historicity to the idea that "unit test" was a reference to the isolation of a tests from its peers; it's just a ret-con.
Ive had this experience with team-specific vocab where certain terms organically end up having terms with two or more conflicting meanings and it was horrendous. It led to all sorts of bugs, misunderstandings and even arguments.
Even worse, most people didnt realize there was a problem coz they always knew what they meant.
The only time I managed to work past it was by convincing everyone to never use that term again - burning it to the ground - and agreeing to replace it with two or more new, unambiguous terms.
Id love to burn "unit test" and "integration test" to the ground but nobody outside my team listens to me :)
> The argument for isolating the units from each other is that it is easier to spot a potential bug.
That's not the only argument. The important result of this, is ensuring the "unit" of code is written to be testable. This happens to require it be simple and extensible. It does not enforce making the code or tests comprehensible.
When you don't trust someone's code, have them write detailed unit tests. They will find most of their problems on their own and learn better practices, along the way.
I am, in no way, implying that unit tests are a replacement for integration or behavioral or E2E testing et al...depending on how you want to define those.
85 comments
[ 3.3 ms ] story [ 153 ms ] thread> Only isolate your code from truly external services
That makes tests more trustworthy but also sometimes harder to maintain I think. I have seen cases where small changes on the code base created strong ripple effects with many tests to update. Arguably, the tests were not very well written or organized and with too many high level tests. Still, this and the very large execution time of the test collection made me realized that for medium to large projects, I will be much more careful in the future before going all in with the no-mock approach.
The same is often true for software. Ensuring that things integrate is vital. Ensuring that functions run in isolation also has value.
That’s not to say you can skip end to end testing, but the more testing pushed to the complete system level will drive up cost and schedule.
If I were to put a fuel injector (system under test) into a vehicle, and drive it around a parking lot (E2E test), only to have the vehicle stall out. Is the fuel injector at fault? How would you prove that?
No, you pull out the component and you "bench test" it. Believe me, after many MANY hours spend fighting bugs where the extensive E2E suite passed, we finally had to go back to adding in unit tests for the small units under the surface. Not all... but many places.
Looking at a previous project we ran full e2e tests from an end user perspective in three major browsers in less than 15 minutes at a cloud cost of only a few hundred dollars per month. Compared to the cost of developers on the project that was a negligible amount.
Adding more unit tests would make the feedback cycle shorter on some changes, but also add development time to create and maintain those tests. So in line with the article we did so only sparingly on a few critical paths that were likely to cause issues. The rest was well covered by e2e tests.
You should be striving to balance the long-term usefulness of your tests with the debuggability of those tests. In my experience, those tests are what most people would call "integration tests" (although that name, like so much terminology in the testing world, is confusing and poorly defined.)
You want to get the tests up at as high a level of abstraction as possible where the API and correctness assertions are likely to survive implementation detail changes (unlike many unit tests) while at the same time avoiding the opaque and difficult to debug errors that come with end-to-end testing (again, the language here is confusing, I assume you know what I mean.)
See https://grugbrain.dev/#grug-on-testing
The worst part about it is that he called himself a thought leader, called his approach a "best practice" and had nothing really to back that up. Now people go around repeating it all the time. It's frustrating.
``` function setButtonState() { const res = this.fetchSomeResult(); // returns an unfulfilled promise if (res) { return true; }
} ```as a unit test would quickly fail when you want it to. An E2E test would pass, but for the wrong reasons. This has nothing to do with app state, and everything to do with not being scientific, and testing all sorts of variable things at once just because, "that's what the user sees".
If there's no such observable behavior, then why does it matter?
The appeal to "being scientific" doesn't make much sense to me, either. But if you really want to go there, being scientific is all about app state since that's literally all there is to the app.
I think underlying Kent’s statement is an observation that is undeniably true. The closer a test is to the actual way the software will be used, the better it is in a number of ways, _all else being equal_. All else is never fully equal, which is why everything is about tradeoffs.
But when a test aligns with how the software will be used, there are a whole bunch of synergies and benefits. Simplicity increases, clarity increases, you start getting multiple payoffs for each bit of effort you invest.
I’m sure people cargo cult it and lose track of the tradeoffs, like anything else, but there’s a solid point under there. And I agree with him, it is valuable enough to be called a “best practice.”
The point I often try to make is just how easy it is to get the right result, for the wrong reasons. (especially in loose languages like JavaScript). It goes from making more scientific proofs to, "I saw what I wanted to see on the screen - I can't exactly say why, but I think it's because my code is sound".
It's the equivalent to, "this new flea killer kills twice as many fleas". "What did you change?" "Four different things", "which one ended up being the thing that actually made it work?", "dunno, but now we're seeing twice as many fleas dead!".
Such unit tests are quick to write and don't change frequently because they run against low-level building blocks — so they are not a major maintenance burden. The ROI on this kind of testing is very high.
I don't like being told by Dodds and these other unit-test haters that I shouldn't be doing this. In my experience, organizations suffer much more when developers eschew such tests, being quite naturally overly optimistic about the reliability of their work.
Writing low-level unit tests doesn't prevent you from writing integration or end-to-end tests, and those are important too. I'm not here to devalue higher-level testing, and I'd appreciate it if advocates for higher-level testing likewise didn't go out of their way to devalue unit tests.
This drives me nuts.
You don't trust yourself to write code that isn't broken but you trust yourself to write quality tests that ensure the code isn't broken?
I don't understand this at all. Automated tests are just like any code, they are as prone to mistakes and bugs as anything else
Always write functional tests first. Doesn't matter if they are slow - you still want something that faithfully captures the specified behavior and allows you to detect regressions automatically.
Then, if your resulting test suite is too slow, add finer-grained tests in areas where the perf benefits of doing so dwarf the cost of necessary black-boxing.
Getting down to the level of individual classes, never mind functions - i.e. the traditional "unit tests" - should be fairly rare in non-library code.
Not accepting "but the unit tests work" - 100% agreed. Not being able to slice it that the thing that could be accessed by the current public API or a different public API be tested on its own (and often relatively pure) is not how I want to write my software or tests.
Now, as you rightly note, in many cases there's already an obvious component boundary that is inherent in the design. And then it makes perfect sense to write functional tests for individual components along those boundaries. And in some cases, the component may be "library-like", and those tests will end up looking a lot like your typical unit test, where individual classes and even methods are covered one by one. But most classes and methods in a typical app aren't like that.
I believe links are significantly more useful when they include descriptive text like the title or author, rather than just 'here'.
Most of the systems I build use a database on which all logic depends, and often a network connection.
I've worked on systems where these aspects were mocked, and they eventually grind to a halt because of the effort required to make the tiniest change.
First of all you need a way to create a pristine database from code, preferably in memory. Second nested nested transactions are nice, since you can simply rollback the outer transaction per test case; otherwise you need to drop/create the database which is slower.
For networked servers, an easy way to start/stop servers in code and send requests to them is all you need.
Given these pieces, it's easy to write integration tests that run fast enough and give a lot of bang for the buck.
TDD is even more rare for me, I typically only do that when designing API's I'm unsure about, which makes imagining user code difficult. And fixing bugs, because it makes total sense to have a failing test to verify that you fixed it, and that it remains fixed.
I especially dont see what is gained by writing the test after.
I will write as many tests as I need to feel confident, which depends on context.
And integration tests give me a lot more confidence than mocked unit tests.
The cost/benefit of writing a test before just consistently exceeded doing it after for me.
Same for integration, e2e or unit tests (there's never been a rule that says you can only TDD with a unit test).
The cost/benefit trade off for tests with mocks vs. database is a different topic - orthogonal to the practise of red/green/refactor, and one where IMO the trade offs are much less obvious.
It sometimes makes sense for unit tests; I'll occasionally do that when I'm unsure about the API of the code I'm writing since it allows me to spend some time in the user's shoes.
But like I said, I don't do fundamentalism.
I assume you mean versus writing it first, rather than versus not writing it at all.
I've found that TDD works well for bottom-up coding, but not so well for top-down.
With bottom up, I can write the test for a piece at the bottom, write the code to pass the test, and move on. With top-down, if I write the test first, it might be a long while before I have that top-level working, because the bottom bits don't exist yet.
When I feel it's better to write things top-down, I'll often use TDD for the bottom bits I need to write, but for the bits above that, I'll write the tests "on my way back up".
The greatest value in tests is that they help prevent future changes from breaking existing functionality. Writing the test after you write the implementation is equally useful for achieving that as writing the test before you write the implementation.
Requiring the test before writing the code also ensures you dont forget to write a test to match the scenario.
So what is gained by test after... is that it is almost as good?
I still dont get it.
So you end up writing the unit's boilerplate, that doesn't yet do anything but needs to compile/run for a suite to test it throwing a adhoc error of the notImplemented sort or whatever, then break flow to set the test suite and check the reds (that aren't telling you anything useful because of course it's red), then actually write the code.
I find it flow-breaking and cumbersome for little to no gain.
For additions to existing code I find TDD more useful, but even then it's not unusual to decide to move logic around until deciding a final structure, so the units you end up with might be solidified later in time. Writing tests for scraped units is a waste of time then.
Right. In those situations I TDD with an e2e or integration test.
I dont get why youd restrict yourself to doing TDD with just with low level unit tests.
The type of test is, in my mind, a completely different topic to red-green-refactor and for the decision about which one to write I follow a set of rules which is also unconnected.
TDD is just red-green-refactor. It works with any test.
I only use that technique for pieces of code that really fit that well - usually functions that have a very strong relationship between their input and output - so I'll write tests first for those, but not for most of my other stuff.
Almost every user story I follow in production code follows the form of given/when/then scenario which can always be transformed into a test of some kind (e2e, integration, sometimes even unit).
Where it's something like "do x, y and z and then a graph appears" I find TDD with a snapshot test with, say, playwright works best.
If you're using snapshot tests (a technique I really like) surely you can't write the tests before the implementation, because you need the implementation in order to generate the snapshot?
(This is what I hate about the term TDD: sometimes it means test-first, sometimes it doesn't - which leads to frustrating conversations where people are talking past each other.)
I agree that "unit test"/"integration test" as a definition sucks horribly and leads to people talking past each other, but I think with TDD the main issue is that lots of people have developed a fixed and narrow idea of the kind of test you are "supposed" to write with it which makes the process miserable if the type of code doesnt fit that type of test.
The whole idea of a unit test being "the" kind of "default" test and being "tests a class/method as a unit" definitely needs to die.
This order also helps verify I didn't typo something in the test itself and end up TDD-ing myself into broken code.
You don't need something to work with to write it. You can, by definition, write an e2e test against an app that doesnt exist.
This test isnt special as far as TDD is concerned - red-green-refactor works the same way.
Im sensing a pattern in the answers to my question though. I keep getting "well, if you assume TDD is only done with low level unit tests..."
Completely wrong.
Even with your example, there's an initial exploratory stage where you're still figuring out the interface that the tests would use. I, personally, am not capable of using something that doesn't exist. I have to make that initial version first before I can use it in a test.
Quick edit aside: This is also why I rarely work top-down or bottom-up, I work mostly throughline - following the data flow and jumping up and down the abstraction stack as needed.
What happens when you then show it to stakeholders (e.g. other teams consuming your API, customers or UX people) or and they tell you to change it again?
Rewrite everything again?
Thats gonna be reaaaaaaaalllly labor intensive and could damage your code base too.
Im equally perplexed about why people dont try to build top down. It's one of those few things in programming that always makes sense regardless of circumstance.
> Rewrite everything again?
> Thats gonna be reaaaaaaaalllly labor intensive and could damage your code base too.
Why would I do that? Only the thing they have issue with would need to be changed, it wouldn't take any longer than another way of doing it.
You seem to have forgotten what I said, something needs to exist for me to work with. Well, in this "stakeholders want something changed", something exists. It's not a rewrite from scratch.
If you change the spec (e.g. changing the contract on a REST API), you will probably need to consult to make sure it aligns with everybody's expectations. Does the team calling it even have the customer ID you've just decided to require on, say, this new endpoint?
>You seem to have forgotten what I said, something needs to exist for me to work with.
No. I'm assuming here that a code base exists and that you are mostly (if not 100%) familiar with it.
No, it provides live feedback about whether your code is passing your tests
If you have written your tests poorly then set out to make the tests pass, then your tests become the target rather than the correct behavior
If you are continuously updating your tests while your code evolves because you missed test cases or your understanding of the behavior has improved, then writing the tests first didn't actually give you any value. In fact it just wasted a lot of your time
Write the code Manually test to verify correctness and to identify the test cases you have to write THEN write tests to protect against regressions
A small community of programmers, with a disproportionately large audience, foretold that practicing test-driven development would produce great benefits; over twenty five years the audience has found that not to be the case.
Compare with "continuous integration" - here, the immediate returns of trying the proposed discipline were so good that pretty much everybody who tried the experiment got positive returns, and leaned into it, and now CI (and later CD) are _everywhere_.
As for what is gained, try this spelling: test driven development adds load to your interfaces at a time when you know the least about the problem you are trying to solve, which is to say the period where having your interfaces be flexible is valuable.
And thus, the technique gets criticism from both ends -- that design work that should have been done up front is deferred (making the design more difficult to change, therefore introducing costs/delays), and that the investment is being made in testing before you have a clear understanding for which tests are going to be sensitive to the actual errors that you introduce creating the code (thereby both increasing the amount of "waste" in the test suite, in addition to increasing the risk of needing test rewrites).
The situation is further not improved by (a) the fact that most TDD demonstrations are problems that are small, stable problems that you can solve in about an hour with any technique at all and (b) the designs produced in support of the TDD practice aren't clearly an improvement on "just doing it", and in some notable cases have been much much worse.
So if it is working for you: GREAT, keep it up; no reason for you not to reap the benefits if your local conditions are such that TDD gives you the best positive return on your investment.
If Im writing a single line of production code I should know as much as possible what requirements problem Im actually trying to solve with it first, no?
This is actually dovetails into a benefit to writing the test first. If you flesh out a user story scenario in the form of an executable test it can provoke new questions ("hm, actually I'd need the user ID on this new endpoint to satisfy this requirement...") and you can more quickly return to stakeholders ("can you send me a user ID in this API call?") and "fix" your "requirements bugs" before making more expensive lower level changes to the code.
This outside-in "flipping between one layer and the layer directly beneath it" is very effective at properly refining requirements, tests and architecture.
>And thus, the technique gets criticism from both ends -- that design work that should have been done up front is deferred
I dont think "design work" should be done up front if you can help it. I've always felt that the very best architecture emerges as a result of aggressive refactoring done within the confines of a complete set of tests that made as few architectural assumptions as possible. Why? Coz we're all bad at predicting the future and it's better if we dont try.
This is a mostly separate issue from TDD though.
Why do you believe this is relevant? Unit tests target units of code, and check if the unit of code verifies specific invariants for specific combinations and ranges of input values. The concept of a network or a database does not exist in unit tests. In fact, you are expected to abstract away these aspects.
> I've worked on systems where these aspects were mocked, and they eventually grind to a halt because of the effort required to make the tiniest change.
This scenario is totally unrealistic if you knew what you are doing.
The fact alone that you're talking about spinning up a database in the context of unit tests already raises red flags.
Even if somehow you're talking about integration and end-to-end tests, none of the perceived problems you're mentioning are even a challenge, let alone a problem.
> Given these pieces, it's easy to write integration tests that run fast enough and give a lot of bang for the buck.
That's perfectly fine. It's besides the point though, and completely ignores the whole point of unit tests, which is to have a bunch of fast tests that check if your code continues to do the right thing between changes. Unit tests also compel developers to think through how they create/update code to prevent them from introducing handled scenarios.
There are many reasons why the test pyramid is a thing, and why the unit test layer includes far more tests than any other layer.
Spinning up a fresh database instance for each testing run is trivially easy in every testing framework worth its salt.
Unit tests are the last tests that should be added to any system, and I've yet to see any system where the effort isn't better spent on integration testing (maybe once you're as thorough as sqlite?).
That's perfectly fine. It's also the basis of the whole test pyramid concept.
One of the main reasons why you have some problems caught in integration tests instead of unit tests is that developers introduced bugs by failing to characterize how some components actually work and thus fail to design both the units of code and the tests that go along with them.
This is not a problem caused by unit tests. This is a problem caused by developers failing to design and implement components, which consequently means they failed to add relevant tests.
> The integration and interaction are where you get by far the most value out of testing (...)
It's perfectly fine to think like that. The test pyramid concept exists for a reason. However, when looking at failures you need to understand their root causes. If a fault is triggered with an integration test but not with a unit test, that tells you two things:
- one or more unit of code is broken,
- your unit tests fail to cover a very predictable failure mode.
Each of these items by themselves reflect a fundamental failure in the way developers implemented their test strategy.
> Unit tests are the last tests that should be added to any system (...)
This take is fundamentally wrong, to the point that your software design process needs to be completely and fundamentally broken to believe it's reasonable that unit tests can be added post-facto after skipping the whole design process.
>If a fault is triggered with an integration test but not with a unit test, that tells you two things:
In most cases the interaction between units is far less predictable than the behavior of a given unit. And it's often not obvious which unit is 'broken'. Usually the units are doing what they were intended to do, it's the combination of behaviors that are broken (which yes, means 'the design is broken' - that's usually the harder part than writing the code!). No amount of unit tests will catch that. The integration tests will catch problems units, in contrast (admittedly, not 'accidentally working' cases, but I find it's rare anyone else cares about those).
You're just admitting you're the root cause of these problems, and basic testing strategies play no role in that failure.
> In most cases the interaction between units is far less predictable than the behavior of a given unit.
Irrelevant. You write tests to verify how units of code should respond to specific input values, because your units of code will for a fact handle those sets of input values in Production and you want to prove they can and will handle them.
> And it's often not obvious which unit is 'broken'.
It is, if you know what you're doing. That's why unit tests matter, because they prevent you from hand-waving over the problems you're creating for yourself. You need to think through what are the preconditions and post-conditions of each unit of code, add checks, and make sure your unit of codes comply with those.
These errors can only be a mystery if you have no idea what are these preconditions and post-conditions, enforce none, test none, and just allow failures to cascade and errors to be swallowed without having any idea of what's going on and where the problem starts and finishes.
> No amount of unit tests will catch that.
This is simply wrong.
Consider for example null reference exceptions. Do you think it's impossible to tell with tests where a null reference is passed where it shouldn't?
My hope was always that the candidates would do TDD where it seemed simple and obvious to do so. It was actually pretty rare but the candidates that defaulted to doing that always ended up being better in my opinion. They were always made offers higher than my company could afford elsewhere (so i guess in others' opinions too).
In this thread https://news.ycombinator.com/item?id=43060636 I pondered why most people dont default to TDD for production code and the answer invariably seemed to be "we didnt think TDD was a thing you could do with integration/e2e tests".
If I'm fixing a bug, I start by writing a test that reproduces the bug. If I can't do that, I fix the test harness until I can. Then I implement the change, making mental notes of each intermediate bug I think about along the way - things like "I should be careful to name this distinctly so that it's not confused with this other value in scope that has the same type". After that, I cull down that list until it's reasonable and not totally paranoid, and write tests covering those cases. Same thing for any bugs in in-progress code caught by manual testing, fuzzers, etc.
If you have discipline and use version control, you don't need to write tests before you write the actual code to get the same level of coverage as TDD and you waste a lot less time. I've often figured out late in the game how to make something a compile time failure rather than a runtime one - time to delete all those tests written along the way? Encode them all as negative compilation tests? Fundamentally the goal of testing is to describe what behaviors of the software are intentional rather than incidental, and to detect bugs that might be introduced by future changes to the software - TDD mixes both concerns and doesn't put any emphasis on preventing future bugs specifically.
Maybe other people work on different types of things and TDD is great for them, but I write primarily infrastructure code where correctness is critical and I have the luxury of time, and TDD doesn't produce better results for me. This is a case TDD feels like it should work well for, but in my experience it doesn't improve correctness, maintainability, or speed of delivery - at least compared to the alternative I described. I'm sure there's a universe of teams with sloppy practices out there that TDD would be an improvement for, but it's not helpful for me.
This is actually a good (albeit somewhat niche) reason to not write a test scenario at all, but it's still not a great reason to write a test after instead of before.
>Fundamentally the goal of testing is to describe what behaviors of the software are intentional rather than incidental
Yup. A test scenario which is of no interest to at least some stakeholders probably shouldnt be written at all.
This is again about whether to write a test at all, though, not whether to write it first.
>TDD mixes both concerns
I dont think writing a test after helps unmix those concerns any better.
In fact it's probably a bit easier to link intentional behavior to a test while you have the spec in front of you and before the code is written.
I find people who write test after tend to (not always, but strong tendency) fit the test to the code rather than the requirement. This is really bad.
>Maybe other people work on different types of things and TDD is great for them, but I write primarily infrastructure code where correctness is critical and I have the luxury of time
Assuming Im understanding you correctly (you're building something like terraform?), integration tests which run scenarios matching real features against fake infra would seem to be pretty useful to me.
So...why wont you write tests with that harness before the code? Im still unsure.
The only thing "special" about that type of code that i can see (which isnt even all that special) is that unit tests would often be useless. But so what?
But the before-test is strictly negative - it's a waste of time (deleted code, never submitted) and it possibly slowed down development (had to update the test as I messed with APIs).
>Yup. A test scenario which is of no interest to at least some stakeholders probably shouldnt be written at all.
And yet I see TDD practitioners as the primary source of such tests - if you are dogmatically writing a test for every intermediate change, you will end up with lots of extra tests that assert things in order to satisfy the TDD dogma rather than the specific needs of the problem. Obviously this can be avoided with judgement - but if you have sound independent judgement you don't need to adhere to specific philosophies about the order you make changes in.
>In fact it's probably a bit easier to link intentional behavior to a test while you have the spec in front of you and before the code is written.
When implementing to a spec you are absolutely right, but a very small amount of software is completely or even mostly specified in advance.
>I find people who write test after tend to (not always, but strong tendency) fit the test to the code rather than the requirement. This is really bad.
I agree this can lead to brittle tests and lack of spec adherence, but if you are iterating on intermediate state and writing tests as you go, the structure of the code you wrote 30 seconds ago is very much influencing the test you're writing now.
Another issue is that fault injection tests basically require coupling to the implementation - "make the Nth allocation fail" etc. The way I prefer to write these is to write the implementation first, then write the fuzz test - add a few bugs in the implementation, and fix/enhance the fuzz test until it catches them. Fuzz testing is one of the best bang-for-buck testing methodologies there is, and in my experience it's very hard to write a really good fuzz test unless you already have most of your implementation, so you can ensure your fuzz tester is actually exercising the stuff you want it to.
>Assuming Im understanding you correctly (you're building something like terraform?),
I write library code for mobile phones, mostly in Java/Kotlin. I recently did some open source work (warning: I am not actually very proficient with C, any good results are from enormous time spent and my code reviewers, constructive criticism very much welcome). Here's a few somewhat small, contained changes of mine, so we can talk about something concrete:
https://github.com/protocolbuffers/protobuf/pull/19893/files
This change alters a lock-free data structure to add a monotonicity invariant, when the space allocated is queried on an already-fused arena while racing with another fuse. I didn't add tests for this - I spent a fair bit of time thinking about how to do it, and decided that the type of test I would have to write to reliably reproduce this was not going to be net better at preventing a future bug, given its cost, than a comment in the implementation code and markdown documentation of the data structure. I don't know how I would really have made this change with a TDD methodology.
https://github.com/protocolbuffers/protobuf/pull/19933/files
This change moves a memory layout - again, I don't know how I would have written a test for this, besides something wild like querying smaps (not portable) to see if the final page of the arena allocation had faulted in.
I actually did this the other day on a piece of code. I was feeling a bit lazy. I didn't write the test and I figured that making the type checker catch it was enough. I still didn't write a test after either though.
Anecdotally I've always found that tests which cover real life bugs are in the class of test with the highest chance of catching future regressions. So even if it does exist, I'm still mildly skeptical of the idea that tests that catch bugs that compilers have been provoked into also catching are strictly negative.
>And yet I see TDD practitioners as the primary source of such tests
I find the precise opposite to be true. TDD are more likely to tie requirements to tests because they write them directly after getting requirements. Test-after practitioners more likely tie implementation to the test.
It's always possible to write a shit implementation-tied test with TDD, but the person who writes a shit test with TDD will write a shit implementation tied test after too. What did TDD have to do with that? Nothing.
>if you are dogmatically writing a test for every intermediate change, you will end up with lots of extra tests that assert things in order to satisfy the TDD dogma rather than the specific needs of the problem.
I find that this only really happens when you practice TDD with very loose typing. If you practice strict typing, the tests will invariably be narrowed down to ones which address the specific needs of the problem.
Again - without TDD and writing the test after, loose typing is still a shit show. So, I see this as another issue which is about something separate to TDD.
>Obviously this can be avoided with judgement - but if you have sound independent judgement you don't need to adhere to specific philosophies
I think this is conflating "TDD is a panacea" with "if it is valuable to write a test, it's always better to write it before". I've never thought the former, but the examples you've listed here look to me only like examples of where TDD didn't save somebody from making a mistake that was about a separate issue (types, poor quality test). None of them are examples of "actually writing the test after would have been better".
>When implementing to a spec you are absolutely right, but a very small amount of software is completely or even mostly specified in advance.
Why on earth would you do that? If I write even a single line of production code I have specified what that line of code is going to do. I have watched juniors flail around and do this when getting vague specs but seniors generally try to nail down a user story tight with a combination of code investigation, spiking and dialog with stakeholders before writing code that they would otherwise have to toss in the trash can if it wasn't fit for purpose.
To me this isn't related to TDD either. Whether or not I practice TDD, I don't fuck around writing or changing production code if I don't know precisely what result it is I want to achieve. Ever.
Future requirements will probably remain vague but never the ones I'm implementing right now.
>I agree this can lead to brittle tests and lack of spec adherence, but if you are iterating on intermediate state and writing tests as you go, the structure of the code you wrote 30 seconds ago is very much influencing the test you're writing now.
Only if the spec is changing too. This sometimes happens if I discover some issue by looking at the code, but in general my test remains relatively static while the code underneath it iterates.
This obviously wouldn't happen if you wrote implementation-tied tests rather than specification-tied tests but... maybe just don't do that?
>Another issue is that fault injection tests basically require coupling to the implementation
All tests requiring coupling to implementation in some way. The goal is to lo...
Maybe this is a static typing thing - if the test won't build because you've made the bug inexpressible in the type system, what test can you even write?
>If I write even a single line of production code I have specified what that line of code is going to do.
>combination of code investigation, spiking and dialog with stakeholders
Does "spiking" in this context mean writing code without TDD?
>I think this is conflating "TDD is a panacea" with "if it is valuable to write a test, it's always better to write it before".
That's a weaker formulation of TDD than I've seen espoused and practiced, which is usually something more like "before you make any behavioral changes to the code under test, you must write a test that fails; then make your edit so the test passes, and repeat". The problem with your approach at least for me is that until I've messed around a bit in the code seeing what's the best approach is to solving a problem, I don't know what the best structure for a final test is, or whether I can make the change in a way that leverages existing tests to detect the bug.
> I'm not really sure why fault injection should be treated as special.
Suppose you write a test reproducing a bug that reacts to allocation failing. One common way to do that is to inject an allocator that fails on the specific allocation call you had a bug with. But how do you target that specific call? One way is to make the Nth allocation in a test fail - but even slight changes to the production code will make this test start testing something completely different. The solution is to have a fuzz test that injects failed allocations randomly per run - now you can be reasonably confident that even if the prod code changes over time, your fuzz test will still all the allocation sites, preventing regression. I don't see why it's beneficial to write this first or last. Under your model the instrumented allocator setup is a replacement for the single test case that reproduces the original bug.
>If I'm reading it correctly, this looks like a class of bugs we discussed that is fixed by tightening up the typing. In which case, no test is strictly necessary, although I'd argue that it probably would not hurt either.
No, the original "bug" is that if you have two calls to SpaceAllocated and the second one races with a Fuse call on a other thread, you can see a smaller value result than before. This behavior wasn't guaranteed by the public API (so not a bug) but it's desirable to add. The fix is replacing a singly linked list with a doubly linked list, using tagged pointers to avoid adding storage cost for the second set of links. A test could be written for this; I could inspect the allocated addresses of three arenas and fuse them in a specific order, but I could not actually reproduce the ordering required without a test harness that allows full control of thread execution interleaving - which would be a huge amount of work and in my opinion not actually prevent future bugs, since that interleaving only meaningfully exists in the previous implementation. The full set of possible interleavings is way too large to productively explore. If I had started by writing the test, I would have spent a bunch of time messing around before giving up; because I did the implementation first, I had a much more informed idea of what would be required to test it, and changed my plan for preventing future bugs to documentation.
>I can't tell if this is refactoring or you're fixing a bug. Is there a scenario which would reproduce a bug?
In this case the "bug" is not...
The author has a lot of opinions about testing though which conflict with what I've found to work in even that sort of dynamic environment. Their rationale makes sense on the surface (e.g., I've never seen a "mock"-heavy [0] codebase reap positive net value from its tests), but the prescription for those observed problems seems sub-optimal.
I'll pick on one of those complaints to start with, IMO the most egregious:
> Now, you change a little thing in your code base, and the only thing the testing suite tells you is that you will be busy the rest of the day rewriting false positive test cases.
If changing one little thing results in a day of rewriting tests, then either (a) the repo is structured such that small functional changes affect lots of code (which is bad, but it's correct that you'd therefore have to inspect all the tests/code to see if it actually works correctly afterward), or (b) the tests add coupling that doesn't exist otherwise in the code itself.
I'll ignore (a), since I think we can all agree that's bad (or at least orthogonal to testing concerns). For (b) though, that's definitely a consequence of "mock"-heavy frameworks.
Why?
The author's proposal is to just test observable behavior of the system. That's an easy way to isolate yourself from implementation details. I don't disagree with it, and I think the industry (as I've seen it) discounts a robust integration test suite.
What is it about "unit" tests that causes problems though? It's that the things you're testing aren't very well thought through or very well abstracted in the middle layers. Hear me out. TFA argues for integration tests at a high level, but if you (e.g.) actually had to implement a custom sorting function at your job would you leave it untested? Absolutely not. It'd be crammed to the gills with empty sets, brute-force checking every permutation of length <20, a smattering of large inputs, something involving MaxInt, random fuzzing against known-working sorting algorithms, and who knows what else the kids are cooking up these days.
Moreover, almost no conceivable change to the program would invalidate those tests incorrectly. The point of a sorting algorithm is to sort, and it should have some performance characteristics (the reason you choose one sort over another). Your tests capture that behavior. As your program changes, you either say you don't need that sort any more (in which case you just delete the tests, which is O(other_code_deleted)), or you might need a new performance profile. In that latter case, the only tests that are broken are associated with that one sorting function, and they're broken _because_ the requirements actually changed. You still satisfy O(test_changes) <= O(code_changes); the thing the author is arguing doesn't happen because of mocks.
Let's go back to the heavily mocked monstrosities TFA references. The problem isn't "unit" testing. Integration tests (the top of a DAG), and unit tests (like our sorting example, the bottom of a DAG) are easy. It's the code in between that gets complicated, and there might be a lot of it.
What do we do then?
At a minimum, I'd personally consider testing the top and bottom of your DAG of code. Even without any thought leadership or whatever garbage we're currently selling, it's easy to argue that tests at those levels are bot...
> The argument for isolating the units from each other is that it is easier to spot a potential bug. (...) In my opinion, this does not pay out because of the huge amount of false positive test cases you get and the time you need to fix them. Also, if you know the code base a little you should have an idea where the problem is. If not, this is your chance to get to know the code base a little better.
This is at best specious reasoning, and to me reflects that the blogger completely misses the point of having tests.
To start off, there is no such thing as a false positive test. Your tests track invariants, specially those which other components depend on. The whole point of having these tests is to have a way to automatically check for them each and every single time we touch the code, so that the tests warn us that a change we are doing will cause the application to fail.
If you somehow decide to change your code so that a few invariants break, these are not "false positives". This is your tests working as expected and warning you that you must pay attention to what you are doing so that you do to not introduce regressions.
It's also completely mind-boggling and absurd to argue that "knowing the code" is any argument to avoid tracking invariants. The whole point of automated test suites is that you do not want the app to fail because you missed any detail or corner case or failure mode. Knowing the code does not prevent bugs or errors or regressions.
I'm perplexed by the way we have people write long articles on unit tests when they don't really seem to understand what they are supposed to achieve.
Example: a function makes two RPC calls to query two independent statuses, which are of course mocked for the unit tests. Due to the the test library used, unit tests expect query A first and query B second. The function is refactored, and it now queries B first, so unit tests start to fail. This is 100% false positive - nothing in the real world cares if A or B are queried first, those actually go to systems which don't even know each other. And yet, the change author now has to fix dozens of failed tests.
Another example: there is an event-based system, and unit tests hardcode number of main loop executions ("emit event A; poll; poll; poll; ensure event B arrives"). A change which adds one more poll cycle will have zero effect on the real app, but causes many tests to break. It's a false positive.
Yet another example: there are random numbers involved, so test driver makes sure to seed RNG to known value at start of each test. A single call to random() is added somewhere early in the code and boom! each test explodes. Another hundreds of false positives.
I could go on an on... Should those tests be rewritten to be more robust and only check things that really matter? Yep. Does real-life code has fragile unit tests like this? Yep, almost every single codebase I have seen.
At most, you can only arrive at that conclusion by observing your personal reality. If your teams crank out crappy code that's error-prone and untestable with their crappy tests, and can't manage to fix either their code or the tests, then naturally they end up living in the reality they created for themselves.
Back in the real world, some projects renowned by their stability and robustness go out of their way to single out their test-drive approach to software design and development as the fundamental reason their software is stable and robust.
What do you think is the difference?
> All they check is that the code is still structured identically to the original implementation.
Unit tests don't test structure. Their whole point is that they do not reflect structe Unit tests only cover the behavior that's expected from a specific unit of code, and they only check for the invariants relevant to that specific unit of code.
Perhaps this is a telltale sign you should rethink what you're doing.
If I had to choose between 1) always writing specification-linked tests that make as few architectural assumptions as possible and 2) TDD, sure, I'd pick 1 every time.
1 and 2 is still better though.
But when software actually needs to be made at scale and with high quality you almost always have lots of tests.
Statistically, the average developer is not writing software anymore where corner cases matter. They're writing various permutations on standard web apps. Most web apps are pretty buggy, even ones from major corporations. So stakes are relatively low for having failures.
People writing file systems or crypto routines are going to have an easier time understanding the value of tests.
I mean, yeah. Most devs don't understand the point. It turns into a check-boxing exercise. And most devs are vaguely smart and can figure out how to do a check-boxing exercise with minimal effort while evading the point of doing it.
Relying on people who don't care enough to write good code to care enough to write good tests has a certain irony to it.
> To start off, there is no such thing as a false positive test. Your tests track invariants, specially those which other components depend on. The whole point of having these tests is to have a way to automatically check for them each and every single time we touch the code, so that the tests warn us that a change we are doing will cause the application to fail.
I'll push back on this. I'm a veteran of a codebase where the tests broke all the time with tons of false positives. This was mostly because of over-mocking, and an abundance of assert X called with Y and Z tests. Even pure functions were mocked.
Every time I re-wrote a function implementation, even if it was a pure function and produced the same results, it would break tens of tests. Bonus points if the implementation of the thing that was mocked out had changed since the last time someone reviewed the tests and the way it was calling the function didn't do what the test assumed anymore.
They were just talking about test doubles / mocks above that. Those cause all sorts of risks with false positives and false negatives, for example by mocking out a function the code under test relies on, but the mock not being changed when the contract between those two pieces of code changes.
The point is that it's absurd to try to characterize the regressions you're introducing as "false positive"/"false negative".
Again, this is at best specious reasoning, and to me reflects that the blogger completely misses the point of having tests.
Your code has behavior. Your tests verify invariants associated with this behavior. If you purposely change that behavior, it stands to reason that your task also involves updating how your code checks for that behavior.
If you change behavior but fail to change how that behavior is checked, that is not a false positive or false negative. That represents a few problems you introduced: failing to update how that behavior is checked, and failing implement new behavior without checking if it works. Both reflect problems created by a developer failing to perform the basics of their work, and worse: blaming working tests for the mess he's creating.
Think about the issue: if a highway administration decides to redesign an intersection but fails to move/update its stoplights, does this means the old stoplights cause accidents with their false positive greens?
If your tests don't fail before you update them, that's one false positive. Usually introduced by mocks that hide part of the behavior.
If your tests were passing but a bug is discovered in the code that should have been caught by the given tests, that's another false positive. Also possibly introduced by mocks, but hopefully is more like just a missing assert.
If your tests start failing with a correct code change and just straight need to be updated, that's a false negative (this is your example, halfway through the work).
In either false positive case if it was caused by mocks, those are bad tests and should be rewritten and not just updated.
> Think about the issue: if a highway administration decides to redesign an intersection but fails to move/update its stoplights, does this means the old stoplights cause accidents with their false positive greens?
Lights at an intersection aren't tests, they're part of the working system. Tests in this example would be something more like a central reporting system that's supposed to show what's going on at the intersection. The system (the intersection and lights) can be working totally fine, but the reporting system is showing it messed up (false negative) or messed-up lights can be causing crashes but the reporting system shows it's working fine (false positive). In both cases it should be fixed, but the fix is independent of what - if anything - should be done with the intersection and lights themselves.
I dislike that term because the most valuable tests I write are inevitably more in the shape of integration tests - tests that exercise just one function/class are probably less than 10% of the tests that I write.
So I call my tests "tests", but I get frustrated that this could be confused with manual tests, so then I call them "automated tests" but that's a bit of a mouthful and not a term many other people use.
I'd love to go back to calling them "unit tests", but I worry that most people who hear me say that will still think I'm talking about the test-a-single-unit-of-code version.
The short version is that "unit test" did actually mean something (see Glenford Myers, __The Art of Software Testing__ or Boris Beizer, __Software Testing Techniques__), although it wasn't necessarily clear how those definitions applied to object-oriented programming (see Robert Binder, __Testing Object-Oriented Systems__).
The Test-First/TDD/XP community later made an effort to pivot to the language of "programmer test", but by the time that effort began it was already too late.
So I think you should continue to call your tests "tests" (or "checks", if you prefer the framing of James Bach and Michael Bolton).
As best I can tell - there's no historicity to the idea that "unit test" was a reference to the isolation of a tests from its peers; it's just a ret-con.
Even worse, most people didnt realize there was a problem coz they always knew what they meant.
The only time I managed to work past it was by convincing everyone to never use that term again - burning it to the ground - and agreeing to replace it with two or more new, unambiguous terms.
Id love to burn "unit test" and "integration test" to the ground but nobody outside my team listens to me :)
Id probably replace them with:
* code coupled
* interface coupled
* high level
* low level
* xUnit
* faked infrastructural
* deployed infrastructural
* hermetic / non hermetic
* declarative / non declarative
That's not the only argument. The important result of this, is ensuring the "unit" of code is written to be testable. This happens to require it be simple and extensible. It does not enforce making the code or tests comprehensible.
When you don't trust someone's code, have them write detailed unit tests. They will find most of their problems on their own and learn better practices, along the way.
I am, in no way, implying that unit tests are a replacement for integration or behavioral or E2E testing et al...depending on how you want to define those.
I auto test the API of the server/system/library/module I am responsible for. Nothing else. No auto testing of internal details.
It lets me completely rewrite internals without breaking the tests.
The API tests needs to be so good that another developer could implement the same server/system/library/module using the tests only.
And the API tests needs to try as hard as possible to break the code being tested.
Using this method I have had zero bugs in production for the last 5+ years.