The article makes some fair points for performing end-to-end tests. There is however a benefit of unit tests over end-to-end tests that is not mentioned: code locality. The cost of fixing a discovered bug seems to grow exponentially with the amount of code in which the bug may appear. Unit tests typically cover a small part of the code. If a unit test fails, isolating and fixing the bug is typically cheap. On the other hand, if an end-to-end test fails (due to the same bug) then isolating and fixing the bug can be quite costly because there are many different parts of the code that need to be checked. To me, code locality is the main advantage of writing unit tests.
At the end of the day, there is a spectrum of tests going from unit tests to end-to-end tests. The spectrum represents several trade-offs such code locality vs. coverage. In my experience, the most economical approach is to write a balanced mix of tests the lie along this spectrum.
You are right!
I work on test automation for end-to-end testing. I dare say my work is very useful and has prevented a lot of bugs from hitting our prod. But at my workplace we also all agree on the fact that it is often quite painful to locate the cause of the bugs that I report, precisely because we don't have enough unit tests on our old code.
Since we live in a world of limited budget and time to spend, I agree with the conclusion of the article: "use unit tests where it makes sense". It is what we try to implement on our new code (and when refactoring legacy code).
I hear this argument a lot, but I do not think it is valid. What happens is that you simply put in the time before a bug occurs with every unit test that you write. Which isn't very efficient, seeing as not all of your units will lead to bugs. So yes, once you have all of your Units covered 100% it will be easier to find bugs, but you have invested a lot of time in order to get here and in order to keep the unit-test-suite maintained.
It's a trade off of when you want to spend the time.
Sure, it's less efficient to write unit tests ahead of time for every case, but in a lot of cases fixing a bug faster when it's discovered is much more important than even double the hours spent during less "pressing" times.
But I think the argument that the article is making is that it's not only the case that you're moving the time spent fixing bugs, (i.e. a greedy vs lazy strategy), the issue is also that the baseline maintenance cost of the code base is increased by orders of magnitude when unit tests are considered.
I tend to agree - in my personal experience in organizations with strict TDD culture, a perverse incentive often emerges to preserve existing flawed architecture over obviously better solutions just because it's so painful to deal with all the tests.
One of software development's most powerful properties is the ability to iterate quickly: it's foolish to prioritize dogmatic beliefs about testing over that quality.
Yeah I absolutely agree that in many cases tests become more of burden than a help, but I think that isn't a problem with testing but with it's application.
It's just an extension of your code, if you are going to throw some code away to change an interface, then throw the tests away too. If you are afraid to do that because of the time spent, then you probably spent too much time on writing tests.
While I am dogmatic about tests, I also believe that around 50% code coverage is normally enough in most application codebases. Cover the important parts, the parts that are hard to test manually, the "core" pieces that lots of other areas rely on, and some tests for bugs that you want to prevent happening again.
If you want to quickly iterate, go for it! You shouldn't have all that many tests in the parts you are changing frequently. But to change a core aspect of the codebase, or a really complicated aspect of it, then the extra work of rewriting the tests shouldn't be all that bad.
I'd shorten your last statement. It's foolish to prioritize dogmatic beliefs. Everything has a time and a place, and moderation is key.
But the goal of tests isn't just to find bugs in newly-written code. It's to defend against regressions, and a regression is even harder to localize, given the team may not even be well-familiar with the failing code.
Correct that the goal is to prevent regressions. I claim (I don't know how to study this) 80% of your tests will never fail and so they could safely be deleted - but I have no insight into which tests will fail so I say keep them all.
Incorrect because in fact it isn't hard to localize failures: it is something in the code you just touched!
I don't understand how you could live the experience that the problem is always with the just-edited code. The closest I can come is supposing that you've always worked with thoroughly unit-tested code (and correct, well-documented libraries, etc.).
I'm lucky enough that the code base I work on was a "big rewrite" not long enough with the goal of having everything tested. Also we are an embedded system where we know exactly which version of each library to support and upgrading any library is in itself a big deal done as a separate exercise.
Yes, but you also need to localize the effect of the bug to know why is it that the code you changed broke the program (and remember that we're talking about a case where the rest of the program is not familiar to you enough). Good unit tests can help you find the immediate effect of the failure, rather than the ultimate one.
This is a spot where the potential for test-induced design damage comes into play.
With bite-size integration tests, I find it's generally not too hard to isolate the cause of a failing test, because the code it's testing tends to be straightforward, and fairly easy to step through, if necessary.
I frequently have a harder time with unit tests. The code ends up involving a lot of extraneous abstractions that I need to think through. The test code is often so heavily mocked that it's hard to distinguish the behavior under test from stuff that's just being mocked or stubbed to get the SUT to run cleanly, meaning I've got to start with trying to figure out whether the bug is in the test or the code being tested.
It gets worse in long-lived code bases, where the unit tests are often subject to significant bit rot on account of how brittle they are. I've definitely had some code archaeology excursions reveal that the reason an entire suite of tests were tautological is because someone was doing a nominally unrelated refactor, and just put in the minimum effort necessary to get the tests to go green again.
You can argue that developers need to be more diligent. Me, though, I figure it's sort of like those lines of bare dirt you see criscrossing the lawns of university campuses: when things get to that point, it's a sign that the official way of getting around isn't appropriate to most people's real needs.
I should say, I was complaining there of code that is pervasively unit tested, not unit tests in general.
I do think it's important to have unit tests when the unit's behavior is complicated. Where I start to get worried is when there are unit tests being written against classes that have very little behavior that doesn't involve interaction with some other module.
> I do think it's important to have unit tests when the unit's behavior is complicated. Where I start to get worried is when there are unit tests being written against classes that have very little behavior that doesn't involve interaction with some other module.
I think this is precisely where unit test suits start to have problems. Good, flexible unit testing requires a lot of judgement about what will be useful to test and what will be too much of a burden in the future. Unfortunately judgement is hard to acquire and even more difficult to teach, and a lot of teams want to create and enforce over-dogmatic testing "standards." When unit testing, you have to balance:
1. What testing do I need to have confidence my code is working?
2. What testing do I need to catch likely regressions?
3. What kinds of tests will just get in my way in the future or are literally useless [1]?
[1] E.g. unit tests that essentially only test core language functionality, once you take out all the mocks.
This is a really important aspect of good testing that doesn't seem to get as much attention as it deserves - the tests that have brought the most value for me are the ones that assert the business requirements, not the implementation details.
So for a little passthrough/orchestration class, it probably doesn't make sense to do much testing. For something that actually performs business logic, that's a prime candidate for testing. I've seen plenty of tests that just seem to aim to increase coverage, heck, I've written plenty of those myself - but at the end of the day, the benefit they serve after being written is probably minimal.
I agree that regression scenarios are difficult to identify, this is why its good to have unit tests to begin with.
You're only unit testing the code how you 'intended' for it to work at that time. Even though the tests are written, it probably wouldn't be uncommon for a bug to slip through when running your code, what you then can do is write another test to account for that scenario, then repeat and your code becomes more robust as a result.
Maybe I'm a worse than average coder, but I find it nearly impossible to write a "unit" of code without also writing one or two bugs along the way. So whether you do your testing ad hoc as you develop, or write unit tests, you do have to spend time in testing each unit of code you write. It doesn't seem to me that formalizing this into a repeatable unit test is adding a lot of real extra work.
I would suggest trying to write correct code without unit tests. They fill up mental capacity making it harder to think about then write correct code in the first place.
Keeping below 1 bug per 100 lines of code is viable simply by being careful and thinking things through. That's a long way from perfection, but it really helps.
THen why not think "in-code"? I mean thinking through all aspects of edge cases and the like would likely result in some written documentation- why not do it in code then?
You can only focus on a relatively small number of different things at the exact same time.
1 3 2 5 9 2 8 7
vs.
1 3 2 5
9 2 8 7
Memorizing the first sequence is trivial, but dealing with each half independently is much less mental effort. Now each idea can be more complex than just a number, but even still you make fewer mistakes by removing mental overhead.
PS: Now, their are a lot of tricks on how you can get better at this stuff. But if the parent poster tends to write bugs in most functions then simplifying things may help.
I find that writing unit tests helps me think about the code so that I write better code. Plus it generally forces me to make the code more modular than I might otherwise. YMMV
Advice that boils down to “try harder” typically provides no value. If you want an outcome to change, you have to change an input. Better tooling or better processes may achieve that. Trying harder rarely will, because most people are already trying quite hard to do a good job.
I think people may be reading this backwards. I am specifically saying don't try as hard. Don't think about unit tests or anything else when writing correct code for a function.
Once you can generally write mostly correct code then you can work on improving your process. But, until most of your functions are working when you right them just work on improving that.
Edit: And yea quality may drop as part of this transition, but you need to get a feel for how much you can pay attention to.
I don't necessarily think he has a problem with code correctness after QA. I think he has a problem with code correctness before QA.
You can build a factory that has a Great QA process that finds all faults and corrects them. But, inside the factory you still want to minimize the amount of defects for QA to find. Either way you still need QA, but it's easy to get into the habit of improving QA vs improving the initial process.
PS: One exercise I did for a while was every time I found a non syntax bug in a function I just wrote (aka bug at run-time) I would start over and rewrite it. It's painful, but 'Build' wait let's look at this again if I hit run and it does not work that's going to be painful. So is this actually correct? Anyway, doing that showed me how important it was to just focus on the code (and what it means) to exclusion of everything else.
If you drive improved quality into the initial product phases in a factory, that’s still QA. QA is not an independent process unless management is incompetent.
But you will not drive significantly higher quality into earlier phases by telling your factory workers to try to do a better job. You drive higher quality by analyzing where your processes are failing you and changing them so that they no longer do.
If your workers are consistently failing to tighten bolts sufficiently, you can add a QA step to check the bolts every time, or require your workers use a torque wrench, or use a robot to make this more consistent. But you have to do something other than telling them to try harder. You undoubtedly told them to try harder already and it didn’t work. Now you need better tools or processes.
QA let's you build a robot to tighten a bolt, then build a robot to test the bolt is tightened and repeat as needed. It's inherently output focused.
However, the lady designing the factory in the first place should know if tightening bolts or using an arc wielder has more issues. So sure, QA should include checking the oil level inside a robot in the factory to avoid problems in the first place. But QA is not all inclusive it's not for example part of the collage education that's training your workforce other than perhaps a requirement that they have said degree.
I'm not sure what you're trying to get at. Production is output focused.
QA is the system of trying to produce at the desired quality. Anything done to improve quality is QA, including whether you hire better employees or train the ones you have. You can define QA more narrowly if you like, but then we're arguing semantics and we're already off in the weeds.
> until most of your functions are working when you right them
Heh...irony alert. You're advocating focusing on just getting it right the first time and you didn't get write right.
But seriously, the people you're responding to are correct. Any process that relies on humans being less error proneis bound to fail. You need to either create a process that makes humans less error prone (e.g. checklists) or embrace our propensity for making errors and plan for that eventuality.
I have found unit tests are useful for the original developer to very explicitly describe what test cases the code was written for (including parameter data types). The lack of unit tests for specific types of use cases can also signal to the inheriting developer what the code wasn't originally expected to do (which can signal performance limitations).
That's exactly what the article is saying. Also, your point about unit tests assumes that tests cost the same to write. Sure, a unit test will have ten times better locality than an integration test, but you have to write ten of them to catch a bug. Also, unit tests don't catch bugs where they're very likely to crop up: the boundaries.
You have to write 10 of them if they are not part of your workflow. If you TDD (which is applicable in a lot of cases in my experience with web development), then instead of testing with Curl or a web browser, you write tests. This takes the same amount of time upfront, so the additional cost of writing the tests is actually 0.
Not sure about other types of development, but in web development, TDD can be a good way to have automated tests without the additional cost.
What I mean is that compared to testing in browser, writing unit tests and using those tests instead of the browser takes about the same amount of time. So doing tests in browser or in TDD ends up taking the same amount of time upfront. So testing in browser is as expensive as writing unit tests with TDD.
Oh, I guess you're right. I may have interchangeably used unit-tests and integration tests because in PHP/Symfony, they look pretty much the same: some setup code and "assert()" calls.
Sorry for any confusion for anyone reading my previous comments.
The whole tests don't take much time to write is bullshit. There are always things to configure in the test framework and something to go wrong. Plus you actually have to write the tests.
What kinds of things do you configure ? For us, the test environment is the same as the development environment, only we don't send out emails. Then there are some test scripts, but those are provided by Symfony and remain unchanged. So there is very little to setup.
And indeed, tests don't take much time to write once you get used to writing them. It's like anything. The more you do it, the better you get at it.
The test environment on a local machine, as the tests are pretty useless if you can't run them. There can be database issues, as you aren't going to connect to the production database for running tests locally.
We have a full continuous integration environment at work, and the tests run there fine, but trying to reproduce that on a local machine is often a fairly difficult experience.
We have maybe 30 components in our system, so often I haven't touched the component before and I am asked to fix a bug in it. Sometimes they are using standard testing libraries, other times there are a lot of extra libraries that I haven't used before. Getting everything to play nicely isn't always trivial.
This implies your team needs to invest more in getting your local environment to parity. My team invested heavily in making the local setup identical to the pull request environment so that the testing is identical. We’ve also invested heavily to ensure that nearly[1] the exact same tests run in our continuous integration pipeline as our pull requests, with the net effect being that a break in CI is by definition an environmental difference.
This drastically scopes down the surface area for CI breaks.
[1] There are a few exceptions for tests that specifically cover environmental config/behavior that cannot be fully tested locally.
collyw is right. “Tests don’t take much time” is bullshit. Testing a feature typically takes as long as writing the feature, whether you pay that cost with unit tests, end-to-end tests, or ad hoc testing. (As hoc actually costs less up front but eventually many times more over the life of the product.)
If you’re very familiar with testing and/or doing TDD, you might include your testing costs in your estimate, but you still have the cost to pay. And if you write good testing up front, it will cost more up front.
You can isolate your server code behind an interface that lets you test it in your IDE, using the same requests that would normally originate in the browser. You just translate each browser request into an API call, and test that API call. No browser needed.
From my experience trusting a unit test instead of looking at things in the browser is a certain recipe for introducing bugs. Not sure if this is what you are advocating or if I misunderstood.
That is what I do for most back-end tasks (not so much for front-end stuff yet although there is some interesting work happening in the open source world for that too). And I've actually had far fewer bugs testing with TDD than testing in browser.
In the browser, testing edge cases sometimes requires custom headers, encoding of data to create authorization tokens, etc. This means you have to either have amazing browser tools (which I've yet to find) or use a combination of browser and shell to achieve what I need.
In TDD, most setup can be automated with simple function calls. In addition, well made frameworks, such as PHP's Symfony, have utilities that even avoid making real HTTP requests, so tests run faster than using a browser, but the result is the same.
I'm not saying everyone should do TDD, but from my experience, it can lead to increased productivity and fewer bugs in some cases.
By the way, if you know of browser tools that make testing easier, I'd be glad to learn more. I use TDD because I lack the browser tools. If I had the right tools, maybe I'd consider going back to in browser testing.
I have been made aware further down that I might have confused unit and integration tests a bit. Sorry about that. In Symfony code, the two co-exist and look somewhat similar. Maybe that's why I don't really differentiate them.
To be honest I don't actually differentiate them either.
When my function under test calls sort() I don't fake that call out, so technically I just wrote an integration test. (In fact I work with one group that will inject a fake sort())
If I write a library foo which has sub-module bar which has class baz and fuzz, is the test for bar that tests both baz and fuzz a unit test of an integration test? Of course if you are a user of my library tests for foo are unit tests to you...
I once spent an evening reading and watching wise people on the internet to find out where is difference between unit tests and integration tests. It's really fuzzy and depends on a source.
There's no agreement what is "unit". Few classes interacting with each other could still be a unit.
I didn't find any strict definition which would be useful in practice. I just write tests and guess they are mostly integration tests, some end-to-end tests and a few unit tests.
>At the end of the day, there is a spectrum of tests going from unit tests to end-to-end tests. The spectrum represents several trade-offs such code locality vs. coverage. In my experience, the most economical approach is to write a balanced mix of tests the lie along this spectrum.
IME unit tests work acceptably in one very specific scenario and fail pretty badly in all others. That scenario being:
1) You're surrounding a self contained block of code that interacts "with the outside world" via a code API.
2) That code API is a very stable and clean abstraction.
3) It has minimal interactions with modules outside of it and those interactions that it does have are tightly scoped (i.e. minimal to zero mock objects are required to write the test).
4) The logic of the code is relatively complex and most bugs that crop up are logical in nature (e.g. off by one, things getting swapped around, incorrect calculations, wrong behavior with negative numbers).
Meanwhile, integration tests (at varying levels) work well for pretty much every case apart from this and still work okay for this type of code. They make much more sense as a go-to default.
I've also worked on several projects where there was little to no code that it actually made sense to unit test. It's not uncommon that an entire codebase is predicated mainly on hooking systems together and doing some shallow calculations. IMHO, having zero unit tests in that environment is actually desirable.
The worst unit tests I've seen have been written when two or more of those preconditions have failed. They would fail constantly, require massive maintenance and, somewhat comically, almost never fail in the presence of an actual bug.
This is spot on IMO. I often see people touting the benefits of unit tests during a refactoring...but 95% of the time refactoring involves modifying class APIs since the hardest part of development is getting the object model right. Unit tests only assist refactoring when you don't modify the APIs - in other cases they are a burden.
Yes, if you change a unit's interface you will have to also change any code relying on said interface. That's a maintenance cost of unit tests.
But it doesn't follow that changing a unit's interface means unit tests suddenly become just a burden. Ideally unit tests are, well, testing a bunch of core functionality of the unit under test. You adapt them to the new interface. Then you're back to having a quick, automatic sanity check you can run against the unit whenever you have to make a change.
I don't understand people bemoaning this 'cost' of unit tests when the benefits they provide typically far outweigh the costs. It's possible broader functional/integration tests have a better ROI in certain situations, but they come with a maintenance cost as well.
>But it doesn't follow that changing a unit's interface means unit tests suddenly become just a burden.
If your refactoring is largely centered around changing unit interfaces (not uncommon) then it means that those unit tests are 100% overhead because most of the time they fail just because you changed the code.
>I don't understand people bemoaning this 'cost' of unit tests when the benefits they provide typically far outweigh the costs. It's possible broader functional/integration tests have a better ROI in certain situations, but they come with a maintenance cost as well.
I've certainly found that the ROI is a lot better. I find that integration tests have a higher up-front cost but maintenance-wise they're the same or cheaper. % of failures that are actually catching bugs is higher too.
I mostly agree, except that for parts of the code that meet these conditions, I aggressively unit test as the default. To the point that if an end-to-end test uncovers a potential bug in this component, my first action is to write a new unit test for this component inspired by the end-to-end test. If that unit test fails, then I can focus on just the unit test, and not worry about the end-to-end test until the unit test passes.
Another condition I would add to the list, which the component of mine I am thinking about meets is:
5) Failures in this component have cascading effects to multiple other components, causing seemingly "impossible" failures that are not obvious to others that they were caused by this component.
If your end to end test uncovers a bug in a lower level module that has a clean and stable abstraction, yes it makes sense to write a test that surrounds that instead.
That needn't be a unit test. It could simply be a lower level integration test.
Reading this makes me realize I had a bias against making lower level integration tests in this situation, definitely something I'll have to look for in the future.
In some cases maybe, but it's not effective in my case. In order to catch failures in this particular component, the most effective thing is to check that a large set of invariants is still true. An integration test of some kind only checks for behavior; does this component work when interacting with the rest of the system? That can show a failure exists, but is not useful for showing why. It's the unit tests which continually check the large set of invariants that can really get to the why: if a particular invariant is no longer true, the path to the problem is usually clear.
Maybe that should be a number 6): easily tested invariants.
> The cost of fixing a discovered bug seems to grow exponentially with the amount of code in which the bug may appear.
That's completely at odds with my experience.
I find that for local bugs, the cost of locating them grows with O(log n) of the amount of code. And for non-local bugs (interface bugs, system bugs, incompatible specs...) unit tests don't catch them anyway.
> The cost of fixing a discovered bug seems to grow exponentially with the amount of code in which the bug may appear.
I don't think this is true.
Fixing a bug is comprised of four parts.
1. Understanding and reproducing the bug.
2. Finding the code responsible for the bug.
3. Coming up with code that fixes the bug.
4. Verifying that your fixing code does not introduce any new bugs.
#2 is the only one that could even theoretically be exponential. The only bugs I've found where I've spent days on 2, were hard to reproduce, intermittent, race condition bugs, which units tests aren't very good at finding anyway.
At the practical level, I find that most end-to-end test tells you something failed, but not which part failed.
A unit test, or a component test, would tell you that something failed, and it's in this general area, which narrows down the search quite a bit.
They're both useful, but I've seen far more problems with people arguing that end-to-end test is more than what they need, while a bad conversion from seconds to nanoseconds would be caught quickly if a unit test was actually written.
I'm a big fan of "economic testing". Test that are not going to "be economically viable" to write during the dev't of new software, usually do not get my permission to be written in that phase.
Unit test thus get written for part that we absolutely want to be maximum certain of that it works correctly. Mostly that functionality can be extracted into a library. For instance a lib that deal with scheduling of events of arbitrary length: we do not want to have issues with those getting messed up, so we try to get 100% unit test coverage of the functionality in that lib.
Another thing with unit test is that I found they mean something different in dynamic languages than in strongly typed languages. In the first unit test are often something to provide confidence that the type system can give you in the latter. Sure both will also test the logic. But this understanding has pushed me to love strong types (as a rubyist), as I can go with less unit test and still be more confident about my code base when it is being heavily worked on. Refactoring becomes so much easier with strong types, as all is "in the code".
When it comes to integration tests I usually want to make a biz case for them. How much does it cost to have it all manually tested, and how much do the automatic integration tests cost to make. While including the benefit of being able to run the automated integration tests at near zero cost (thus several times per day if we need to).
Code robustness clearly has different priority levels though. A graphical error is much less serious than a part of your code that could nuke your server.
Except that's not a hard rule (not that you implied it was).
A graphical error that prevents users from being able to use the application at all can be basically the same as server downtime in terms of impact to the end user.
If the e.g. scheduling code is messed up, and we find out 5 weeks in, then there is a lot of damage. We need to call all parties to review their schedules with them.
On the other hand if the code responsible for the styling of some menu is not working correct, the functionality may still work, and once we discover the bug we can simply fix it.
> rarely do I hear of code that doesn't need to work correctly.
So there are different grades. That's why I used the word maximum. The cost for these unit test should pay bank, even during the dev't phase, IHMO. There are often areas in a large code base for which this hold true, and often they can be identified as such beforehand; then writing these tests should be a priority.
The difference is not between code that need or doesn't need to work.
The difference is having code for which having some bugs is acceptable, vs having code for which bugs are not acceptable.
There is definitely code for which having some bugs is acceptable: think about some small feature that the user can totally do without. I had to do one recently, and bugs were acceptable, as long as it was shipped in time, and working for the client specific devices. That code shipped with no automated test (testing is done manually), and a few known bugs not worth solving. On the other hand, anything that touch core features must be tested.
There is a test plan, and specs. Acceptance tests (manual) weren't even done by a dev.
Also, the code has been reviewed and is documented.
Bus factor really isn't a problem in the instance I was talking about. In fact, the original dev won't be around in a few months, and that's fine. The rest of the team will be able to modify the code.
> Unit test thus get written for part that we absolutely want to be maximum certain of that it works correctly.
I think this is a partial answer.
Yes, I write unit tests for such pieces of software. However, what I also write a lot of unit tests for a pieces of software which are hard to write good integration test for because of combinatorial explosion. If I have a part for integration method which consists of a chain of 4 methods, each of which can have 5 different results, that'd mean I'd have to write 4^5 integration tests, which is clearly not possible.
If I were to write unit + integration tests, however, I could do with 4x5 unit tests plus a couple of integration test which prove that the overall combination of those methods work well together.
Combinatorial explosions are best mitigated by doing property based testing, design by contract and creating clean, low level abstractions which you test against.
In practice I've also run into a lot of combinatorial explosions for which unit testing is worse than useless. E.g. say you have a library that has two dependencies and interacts with browsers you can have:
* 30 different versions of library A
* 50 different versions of library B
* 25 different versions of the language runtime
* 30 different browsers
That's over a million different potential variations for every test. In practice though, there's a sharp power law. You don't have to test all of those variations, you just have to randomly throw variations at the code until bugs stop cropping up. 99% of the bugs you find probably be found in the first 1,000 iterations.
It's also worth looking at all-pairs testing.
The insight is that you can cover all pairs of a set of combinations in only O(N^2) tests rather than O(N^D) irrespective of the number of dimensions. The number of tests ends up being the product of the size of the two largest dimensions. So in your example you can cover all combinations of each pair:
* (lib A, lib B)
* (lib A, language runtime)
* (lib A, browser)
* (lib B, language runtime)
* (lib A, browser)
* (language runtime, browser)
in 50*30 = 1500 tests (or a little more to make the analysis tractable)
I think that article is a necessary read after the original posted article. The original posted article seems a bit too keen to convince you of its point of view that it seems a bit disingenuous.
My feeling about unit testing / integration test is you do as much of it as you can, trying to sincerely learn the lessons of how to do it it effectively, until you know why you don't need to do it. Or until you know what the right mix is. I started with Extreme Programming and testing around 18 years ago and my thinking has constantly evolved over that time. One of the biggest factors, I think, is project context and the nature of the software you are developing.
I find in embedded systems, unit tests done extensively are very very useful. As is integration testing.
In web systems, I find integration tests are often most useful as so much is all about gluing together various tech, and it's often the gluing the comes unstuck. However there is often chunks that have some interesting logic that lends itself to more extensive unit tests.
Sometimes testing is just a mess because of the design. Too many things that are not really needed, too many layers, too many levels of abstractions, too much "future proofing". Then you see people start trying to throw unit testing around it, and it is mostly just useless and explodes out the amount of work to do. Often I think articles like the original are reactions to this scenario.
The dogma around unit testing, TDD and the book selling test hype cycles have caused immeasurable damage to software development as technical and scientific disciplines.
Where to begin.. perhaps the projects with code metrics tied into the check in system - want to commit code changes? You're now on the hook to provide at least 85% unit test coverage or the source control tool will not accept it. So what did most people do? They wrote useless tests, assert true equals true, no-ops or pass statements, basically anything to just get past the arbitrary metric.
Or how about another project that had lots of real tests, that were old, unmaintained, did humongous amounts of work in each single unit test and took a weekend to run, and of course, fail.
Or perhaps the team that obsessed about the testing for each user story, spent all the time doing that, then running out of time to deliver the actual stories.
"Nobody got fired for writing unit tests", and this makes it hard to justify writing less of them, especially to people that are exposed only to development side of software business.
Meh. Thinking about ROI when writing tests is important, but my analysis of ROI is mostly at odds with this anti-unit-test screed: the greatest ROI comes from writing thorough unit tests against stable low-level sub-components.
Integration tests, in contrast, require attention more frequently -- because not all subcomponents are stable and when one changes, multiple integration tests need superficial readjustment.
I'm skeptical that my own ROI analysis is more "right" -- it probably varies by project and team. The conclusion I draw instead is that ROI analysis is important in general when writing tests, but that one particular ROI analysis is unlikely to apply universally.
This article makes lots of good claims as to why integration tests can be the most useful area to focus on. I do however think that unit tests can be really useful in encouraging junior developers to improve their plan vs. write approach to software development (if properly guided).
Somewhat interesting but calling an old idea something catchy isn't really revolutionary.
The article makes a lot of assumptions like "In practice, most agree as most projects set the lower bound for coverage to around 80%". Most projects where, which industry, who are these 80% that agree? This is just taken out of thin air in this case. It might be true but should we take the author's word for it?
There are also some psychological arguments. For
example, if you value unit-testability, you would prefer
a program design that is easier to test than a design
that is harder to test but is otherwise better, because
you know that you’ll spend a lot more time writing tests.
This is very subjective without examples. The opposite argument can also be made that code which is easier to test is sometimes better. It felt like reading a collection of quotes and articles by other people.
I think TDD, unit, integration and E2E testing works. How much of which to apply is entirely project and industry specific and it's up to teams to decide what testing strategy works best for them.
"The outlook for the effectiveness of testing used by itself is bleak. Jones points out that a combination of unit testing, functional testing, and system testing often results in a cumulative defect detection of less than 60 percent, which is usually inadequate for production software."
personally I like write some cursory unit test because it ensures the code logic is correctly decoupled from data retrieval, but most of my unit tests are written from user reports, as part of a large no regression suite.
There's two things that humble me on a daily basis, in no particular order: collaboration and testing.
While I use all sorts of testing, I still write a good amount of unit tests. For whatever kind of tests I write: not a day goes by without tests or the act of writing them exposing bugs that would otherwise have moved on to the "next level". Doesn't mean they'd have ended up in production, but they'd still have been a lot more expensive to fix.
> Integration tests lie somewhere between unit tests and end-to-end tests so they provide the best balance. Therefore, they have the highest ROI.
Being in between does not mean it gives the highest ROI or the best balance. They can easily share the worst of both.
> Lean Testing takes an economic point of view to reconsider the Return on Investment of unit tests.
But does not in any way actually try and work this out, which means that you can't make the conclusions that are in this.
> The Return on investment (ROI) of an end-to-end test is higher than that of a unit test. This is because an end-to-end test covers a much greater area of the code base. Even taking into account higher costs, it provides disproportionally more confidence.
And yet no figures or evidence.
Many e2e tests can cover the same path segments, meaning adding a new test may not increase your confidence by much at all. But they're still taking longer to run, and despite the articles insistence that integration and unit tests are brittle, I've found E2E tests are also brittle, just in different ways.
I shouldn't have to change my string normalisation tests just because the website changed moving the resulting output field to a new place on the page. They shouldn't break because the login flow changes.
I have definitely seen (and written) E2E tests that provide almost no real value, and unit tests that provide huge amounts.
> Plus, end-to-end tests test the critical paths that your users actually take. Whereas unit tests may test corner cases that are never or very seldomly encountered in practice
Unfortunately low proportions of errors still can account for extremely large actual numbers of problems. 1/1000 bugs will happen all the time if you've got a reasonable number of users. Those bugs also may happen to most of your users if they're somewhat random, or worse may heavily impact one group.
> For many products, it is acceptable to have the common cases work but not the exotic ones (‘Unit Test Fetish’). If you miss a corner case bug due to low code coverage that affects 0.1% of your users you might survive. If your time to market increases due to high code coverage demands you might not survive.
Or if roughly one in 1000 customers keep taking your production server down due to a bug you might not, and you may have been just fine waiting another week to go live.
I agree with the idea, you should consider what it is you're trying to achieve and what the best ways of doing that are. What you absolutely shouldn't do is use terms like ROI and economic without any analysis, purely to justify not doing something you don't really want to do.
Edsger Dijkstra: "The first moral of the story is that program testing can be used very effectively to show the presence of bugs but never to show their absence." [1]
"I suggest that the programmer should continue to understand what he is doing, that his growing product remains firmly within his intellectual grip." [1]
The trouble is that advice isn't really actionable. In the real world of commercial software often the only way to satisfy customer requirements is to build a product more complex than the programmer can really understand. Most customers would rather have buggy software (which eventually gets fixed) rather than no software at all.
But it is actionable. Each and every time you chose a certain technology (FP over OOP, immutable vs. mutable state, static typing vs. dynamic typing, shared vs. specific terminology), we make a choice. These choices together drive our community towards a certain mode of programming.
So, your argument is true for individuals, but for the community it is void. Together we should focus on identifying correct, effective, simple and reliable building blocks on which we can build our larger systems.
Perhaps you haven't seen "Simple made easy" by Rich Hickey, which illustrates this concept rather well.
Those technology choices all carry trade-offs. Optimizing for correctness doesn't always maximize business value.
And in my experience most of the defects which aren't caught by our quality control process wouldn't have been prevented by switching from OOP to FP, or from mutable to immutable state, or anything like that. So I'm not convinced.
I'm working on a ticket at the moment that involves a lot of refactoring work to support a new feature.
In the initial run on this code base we went a bit unit test crazy, which was fine at the time, but I'm seeing the drawbacks now where I'm doing a lot of surgery on the components.
I've ended up just beefing up the current integration tests with some extra edge case testing and, after the refactoring work, they still pass, which has given me a lot more confidence.
I think a lot of this comes to head when you unit test the glue logic that orchestrates the entire program, unit testing those becomes painful with mocks etc because they become fragile whenever you do any refactoring work.
Integration testing and unit testing aren't direct substitutes. You would never ever say "I have unit tests, I don't need to run the product before shipping it". Furthermore, automated integration testing is strictly less capable than a human doing integration testing.
Unit tests also have the advantage of being cheap to set up and cheap to run, so their limitations in whole-system testing are offset by the fact that you can have many, many more of them. You can easily blow through a couple thousand unit tests in a matter of 10 minutes or less, whereas even a modest battery of integration testing will be an overnight process at best. That means your OODA loop is a lot faster on unit testing.
(the ideal is of course structuring your application so you can deploy many tests in parallel, but that's not always possible.)
Fixing Selenium so WebDriver instances don't take 60+ seconds to launch would be a major help. Until then, integration tests that work at a controller level (minus the UI) are probably the best compromise between speed and integration. You can pass in a mocked request as input, and assert on the output. Runs much faster than a "true" integration test with a driven browser client, but hits the whole stack better than a unit test. They are also really easy to set up, since you can just have a tester or an integration test go through the motions and then dump request data to JSON, then feed it back as a test. The lack of state persisted between requests makes implementation a lot simpler.
"Unit tests also have the advantage of being cheap to set up and cheap to run".
They are not cheap to develop and maintain however. And given enough developers a certain percentage of them will completely distort the design of the code to enable unit testing making the code itself much more complex.
Unit tests can be hard to maintain for several reasons. When you refactor your code (renaming methods, changing parameters, splitting methods up, etc), you will need to update those tests. Also some types of unit testing can be more brittle then others. For example if you use mocks to invocation counts, refactorings can be even more costly.
This comes up because there's a secret disagreement about the definition of the term "unit test". Some people see units tests as per method tests. If you have 20 methods in a class, all 20 methods get unit tested.
Others see units as the smallest bits of irreducible business or tech knowledge. So if you have 20 methods in a class, you test the 5 public ones, or the methods that compose a useful "unit". This is the definition used by the original test driven development book.
Many of the unit test advocates I've read use the second definition because testing private functionality incurs all the problems the article points out.
Using the second definition leads to the testing pyramid, which is a thing of beauty. Unit tests have a few mocks to the unit's partners. The unit tests test for the correctness of the unit. The integration tests are one level up in terms of code coverage in a single test, and they ensure that the mocks used in the unit tests are correct. The e2e tests are then used to test that the app starts correctly and is configured correctly. The obvious happy and sad path scenarios are also tested to ensure that dependencies are functioning correctly (in a complex application there can be hundred of "obvious" e2e tests).
In my career, every single instance I've seen about someone complaining about writing unit tests, they're either using the first definition I mentioned above, or their code is not composed correctly. I'm sure exceptions exist somewhere, but I've yet to see them.
Using integration tests like the article describes leads to a host of problems. For starters, it's extremely difficult to get complete code coverage without huge or redundant test suites. These large test suites are harder to read and take longer to write because more complex testing requires more complex mocking. Regression is also more likely during large refactors since you haven't clearly enumerated expected behavior.
Fun fact: in the glossary at the end of the original xp book, it describes unit tests as 'tests from the perspective of the programmer', and that's that - a woolly cop-out.
Thats a great point about about the 2 definitions. Many devs dont seem to get encapsulation at all and definition 1 might be related to that. If you expose everything from a module I guess both your calling code an your unit test are unclear of where the boundaries are
Unit tests should only change if you also change public apis (I have one exception to this rule, which is very complex private methods which could easily be their own unit).
Then if you're changing the tests, you're also refactoring everything else.
>For example if you use mocks to invocation counts, refactorings can be even more costly.
These should only be costly if you're over-testing. You shouldn't be mocking and asserting calls on every call within a system under test, you should be checking the ones that matter.
Correctly written unit tests are a refactoring aid.
I have seen statements like 'even bad tests are better than no tests'. I feel that for unit tests, 'no tests are better than bad tests'. Failure in unit tests often break CI and I have seen many tests which tests the structure of the code and not it's intent. For example, I have seen some tests where the product code is putting some element to a list and the test code checks if the element is present in the list. I can understand why people do it ( they are trying to improve "code coverage" ), however if tomorrow you refactor the code to not put that element in the list, but still accomplish the same thing, the test would break. Often, this breaks CI also and now you are sitting late at office trying to figure out what went wrong so you can push your change into the release. This is valuable time that could be spent on new features, fixing bugs or just going home and relaxing. End to end tests ( or integration tests ) are better since they are real user interactions.
If a test fails and it breaks CI, there should be a problem in the product. Move everything else to some "additional testing" bucket and not block developers.
I think that's not right way to test units. You should test the interface resp. the right execution of the specification of the unit and not testing implementation details.
That means you're either testing the return value or testing the inputs which were modified by the unit.
If the behavior of a unit changes in such a way that it does something else with its inputs, it's a specification change which requires other test cases.
This is why you need coding standards for ensuring good quality unit tests, and enforce those standards through rigorous code reviews. Developers shouldn't be able to check in crappy unit tests on the release branch.
In an ideal world developers never push something with broken tests to CI. When developers don't run the tests locally it seems like a sign that there are too many bloated integration tests and not enough fast unit tests.
I find this article misleading; with a biased PoV. The author says "empirical evidence is missing", yet advocates a testing methodology, banking largely on articles and PoVs of other individuals. Is there any software project the author would like to quote as a case-study? Is there any evidence to prove that the Lean method is better OR, the definition of what is better?
It is OK to be economical in what one would see as a cost of testing a product end-to-end. Call it whatever term you like, but functionally testing a product end-to-end has always followed an economical approach, as all such testing is a factor of time & resources. You'd perhaps be less economical if you were testing life-safety systems, but you could choose to be more economical if you considered your software to be not-that-critical, or, you needn't have to set exceptionally high standards.
I find it extremely odd that unittests are even considered into this equation of 'testing-costs', despite decades of improvements in software development processes. Unittests should be part of 'core' engineering & a part of development. Not a task that's added to testing costs. If you aren't writing unittests (irrespective of whether its TDD or not), you are not developing/engineering your product right. You are only building a stack of cards _hoping_ that it wouldn't collapse at some point in the future.
It's a sad state of affairs if one gives code 1st level treatment, but treats unittests, documentation, build scripts and other support infra as something less important; worth economizing. This is really what _matters_ when it comes to overall quality of a software.
One must remember that software is always improved, refactored, expanded, ported or worked on in some way or another. When unittests are missing, then the very boundaries that were meant to dictate the rules of the software don't apply anymore. This leads to human errors causing portions of software to break.
Critical pieces of software that exist today, exist strongly because they were engineered right (Linux kernel for example). Not because their developers followed an _economical_ approach to testing. If an OS disto claimed to perform economical end-to-end testing (of a potential user's most commonly performed paths), would the author of this article want to use that OS over one that has had strong ground-up unittests and testing of interfaces where they matter?
His claim of lacking empirical evidence shows his lack of research on the subject. There's tons of research that consistently shows comprehensive testing during development makes your product better and cheaper over its lifetime.
I'd love to see some research around time cost in this area. My personal experience is that writing unit, integration and end-to-end testing ends up either being a wash or reducing the total time it takes to build a product[0].
The rule of thumb I always come back to is "the minute you decide it's time to attach a debugger, be prepared to burn an hour." I used to not mind firing up the debugger -- heck, I used to run my code with a debugger attached every time I ran it. These days, I write tests. I don't aim for 100% code coverage, though I aim for a majority and 100% coverage of critical paths. When I encounter a problem that isn't covered by a test (rare), I write a test centered around the problem. This usually surfaces the bug before I've even ran the actual test -- and once that test is there, any refactoring that affects that code will avoid that bug if the test passes.
This rule has been so effective for me that I've now gone through four different teams, encouraging each to do "the next project" with automated testing. The result has been a commitment to working that way from that point forward[1].
[0] Assuming some baseline quality requirements -- automated build, and a general requirement to publish a mostly functional product. :)
[1] I can't take full credit for the idea -- it was a blog post that convinced me to try it on myself and I advocated that idea after it convinced me.
For time cost, my rule of thumb is that unit tests roughly take the same amount of test code than the code being tested and about the same (mental) effort. Therefore I don't strive for 100% code coverage but I like all critical (or difficult to understand) code being (unit) tested.
Try building a tennis game, just in code with no gui, and take note of how much you would either write tests for things like the score, set points etc., or how much time you spend debugging to justify if your code works.
Unit tests are a great documentation tool as well, both for developers but also for the business needs of the software — the value.
Another rule of thumb I like to apply when mix-and-matching different levels of tests: how visible would a bug be in higher level tests? If things will fail in obvious ways, a unit test will give no benefit besides narrowing down the bug location in case of a failure.
But if symptoms would likely be too subtle to stand out, e.g. a typical bug would manifest itself in results that seem plausible despite being wrong, then it's a strong indication for unit tests.
Time-to-market is the only valid argument for writing crap code and not unit tests. I'm not sure if you can seriously argue with economic reasons when talking about testing. The most costs in software development arise in the operations/maintenance phase and are probably (there are much papers on this topic) multiple times higher than the costs of the implementation phase.
I agree that every software development team has to make their own decision to balance development time and maintenance cost but I don't agree with this low entitlement mentality of writing correct code in this article.
Of course there are also costs when implementing tests and your unit tests are rendered useless on design changes. But so what? You're also decreasing code assurance and it's better to increase it with tests so that you won't encounter bugs in production which is much more expensive than writing tests.
Software development is not only about adding functionality but also about ensuring its correctness.
It reminds me of a project I worked on. It was for critical embedded software in aircraft ("make an emergency landing" critical, not "we are all dead"). Obviously, the code is certified, with specific requirements (DO-178B level B) and third party audits.
And we made the choice not to have unit tests. Everything was done through integration tests. What made up for it is that we had very good traceability. So for we knew which lines of code were related to which requirement and which test. So if we changed some code, we knew which test to run, and if a test fails, we knew where to look at.
For the edge cases, we had the ability to inject data directly into memory, triggering the otherwise unreachable error conditions, and the very last cases where done through code analysis (we needed 100% coverage).
The highezt cost of unit tests is when they are refs tired with code. The value of a unit test is greatest when written before the implemantation. When refactored it is usually just made to pass by any means necessary to stay green without much/any thought. They are also brittle in these cases where refactoring the test can take meant times longer than the code it tests.
My advice is usually to do tdd for new dev have good coverage of full function tests and delete unit tests if the get in the way/lose value on refactoring. No one likes deleting tests but they must add value or its just more debt.
There are two kinds of end-to-end tests: 1) the software works, 2) the software works in production. The difficulty usually lies in implementing the latter.
For me this depends on what I'm working on, and these days my unit tests are generally for API regression. When I'm writing my APIs, generally my "integration" test would simply involve an external request and assertions of the response. This is fantastic for testing what a client expects, not as good at specific units of business logic where regression can be rather expensive. I would agree many folks take unit testing too far, but in some cases it's saved my neck at a pretty small cost of creation.
This is probably not the same story for something like a react or swift application. Good read, thanks for the solid points.
Great article and I'm somewhat surprised at how often I have to repeat the things mentioned in it -- heck, sometimes it's difficult to even convince developers of the value of unit testing, in general.
I'm a huge advocate of automated testing and with the available tools, like docker, it's relatively painless to get the pieces together to sort out automated testing. Often the tooling you use to enable automated testing is tooling you end up needing anyway -- it's dual purpose. Before the "first run" of a set of code, I'll create Dockerfiles that make up a complete, local, development instance of an application along with some boilerplate tooling that I include to make debugging easier on me. When setting up the production build, the final version is usually this same environment with fewer lines in the file. Because my environments tend to be similar, I have a zsh script that strips out lines in the Dockerfile to get me 90% of the way to a production container.
For me, it's always worth it. I came to this conclusion after spending a few months forcing myself to test rigorously[0], starting with unit tests written often and early and ending with a small number of integration tests and a much smaller number of end-to-end tests. I don't find any of these particularly difficult to write.
The benefits, however, are vast: (1) Avoiding the debugger time-sink: The #1 thing that I always come back to is that I generally end up never having to fire up a debugger. I noticed that every time I encountered a bug in code that was poorly covered, the first instinct was to attach a debugger and peek at locals to see what was going on. This rarely resulted in spending less than an hour troubleshooting. Sometimes you get lucky and you find more than one issue in that debugging session, but often it landed in at burning an hour on ever bug and way too often it was an hour spent debugging production code and the bug was customer impacting[1]. At the same time, it's rare for an automated test to have a time cost that high. (2) Refactoring - Since "premature optimization is the root of all evil", that necessarily means that a performance bug is going to involve injecting complexity into a running codebase and this often comes with high-impact refactoring. Unit tests, specifically, are incredibly helpful here. This is often an argument against integration/end-to-end test automation since refactoring regularly breaks these brittle tests, however, I've found in practice that this isn't the case at least half the time. Of the times that it does affect those tests, the practice of refactoring can surface subtle bugs (on a few occasions it surfaced a subtle race condition that might have been missed if a few of the integration tests covering a subset of the functionality hadn't broken). (3) Design - more for integration and unit tests, thinking about testing while writing code can result in a less brittle design[2]. On integration tests, it means writing SQL scripts and migrations to ensure that a fresh environment can be spun up on-demand instead of using GUI tooling (or, using the GUI tooling to generate said scripts/migrations). (4) Build automation - I'm somewhat surprised at how often I encounter a customer project where I have to follow a 20 step process to get things functional in a development environment. It seems like if CI isn't involved, people figure a README.md with a mess of shell commands and button clicking is OK. Scripting out environment configuration and build was already one of the first things I did when I began running the code I'd written, however, I find I no longer have to argue in favor of this when testing is involved -- everyone wants a single command to execute tests and once integration and end-to-end tests are involved, it just makes sense to add standing up the docker parts, too[3].
159 comments
[ 8.5 ms ] story [ 511 ms ] threadAt the end of the day, there is a spectrum of tests going from unit tests to end-to-end tests. The spectrum represents several trade-offs such code locality vs. coverage. In my experience, the most economical approach is to write a balanced mix of tests the lie along this spectrum.
Since we live in a world of limited budget and time to spend, I agree with the conclusion of the article: "use unit tests where it makes sense". It is what we try to implement on our new code (and when refactoring legacy code).
Sure, it's less efficient to write unit tests ahead of time for every case, but in a lot of cases fixing a bug faster when it's discovered is much more important than even double the hours spent during less "pressing" times.
I tend to agree - in my personal experience in organizations with strict TDD culture, a perverse incentive often emerges to preserve existing flawed architecture over obviously better solutions just because it's so painful to deal with all the tests.
One of software development's most powerful properties is the ability to iterate quickly: it's foolish to prioritize dogmatic beliefs about testing over that quality.
It's just an extension of your code, if you are going to throw some code away to change an interface, then throw the tests away too. If you are afraid to do that because of the time spent, then you probably spent too much time on writing tests.
While I am dogmatic about tests, I also believe that around 50% code coverage is normally enough in most application codebases. Cover the important parts, the parts that are hard to test manually, the "core" pieces that lots of other areas rely on, and some tests for bugs that you want to prevent happening again.
If you want to quickly iterate, go for it! You shouldn't have all that many tests in the parts you are changing frequently. But to change a core aspect of the codebase, or a really complicated aspect of it, then the extra work of rewriting the tests shouldn't be all that bad.
I'd shorten your last statement. It's foolish to prioritize dogmatic beliefs. Everything has a time and a place, and moderation is key.
Correct that the goal is to prevent regressions. I claim (I don't know how to study this) 80% of your tests will never fail and so they could safely be deleted - but I have no insight into which tests will fail so I say keep them all.
Incorrect because in fact it isn't hard to localize failures: it is something in the code you just touched!
Yes, but you also need to localize the effect of the bug to know why is it that the code you changed broke the program (and remember that we're talking about a case where the rest of the program is not familiar to you enough). Good unit tests can help you find the immediate effect of the failure, rather than the ultimate one.
With bite-size integration tests, I find it's generally not too hard to isolate the cause of a failing test, because the code it's testing tends to be straightforward, and fairly easy to step through, if necessary.
I frequently have a harder time with unit tests. The code ends up involving a lot of extraneous abstractions that I need to think through. The test code is often so heavily mocked that it's hard to distinguish the behavior under test from stuff that's just being mocked or stubbed to get the SUT to run cleanly, meaning I've got to start with trying to figure out whether the bug is in the test or the code being tested.
It gets worse in long-lived code bases, where the unit tests are often subject to significant bit rot on account of how brittle they are. I've definitely had some code archaeology excursions reveal that the reason an entire suite of tests were tautological is because someone was doing a nominally unrelated refactor, and just put in the minimum effort necessary to get the tests to go green again.
You can argue that developers need to be more diligent. Me, though, I figure it's sort of like those lines of bare dirt you see criscrossing the lawns of university campuses: when things get to that point, it's a sign that the official way of getting around isn't appropriate to most people's real needs.
I do think it's important to have unit tests when the unit's behavior is complicated. Where I start to get worried is when there are unit tests being written against classes that have very little behavior that doesn't involve interaction with some other module.
I think this is precisely where unit test suits start to have problems. Good, flexible unit testing requires a lot of judgement about what will be useful to test and what will be too much of a burden in the future. Unfortunately judgement is hard to acquire and even more difficult to teach, and a lot of teams want to create and enforce over-dogmatic testing "standards." When unit testing, you have to balance:
1. What testing do I need to have confidence my code is working?
2. What testing do I need to catch likely regressions?
3. What kinds of tests will just get in my way in the future or are literally useless [1]?
[1] E.g. unit tests that essentially only test core language functionality, once you take out all the mocks.
So for a little passthrough/orchestration class, it probably doesn't make sense to do much testing. For something that actually performs business logic, that's a prime candidate for testing. I've seen plenty of tests that just seem to aim to increase coverage, heck, I've written plenty of those myself - but at the end of the day, the benefit they serve after being written is probably minimal.
You're only unit testing the code how you 'intended' for it to work at that time. Even though the tests are written, it probably wouldn't be uncommon for a bug to slip through when running your code, what you then can do is write another test to account for that scenario, then repeat and your code becomes more robust as a result.
Keeping below 1 bug per 100 lines of code is viable simply by being careful and thinking things through. That's a long way from perfection, but it really helps.
PS: Now, their are a lot of tricks on how you can get better at this stuff. But if the parent poster tends to write bugs in most functions then simplifying things may help.
> I find it nearly impossible to write a "unit" of code without also writing one or two bugs along the way
Finding bugs early is great, but minimizing bug creation is even better.
Once you can generally write mostly correct code then you can work on improving your process. But, until most of your functions are working when you right them just work on improving that.
Edit: And yea quality may drop as part of this transition, but you need to get a feel for how much you can pay attention to.
The tests are there specifically to help find the errors that “trying harder” didn’t catch. You don’t get a higher quality result by cutting QA.
You can build a factory that has a Great QA process that finds all faults and corrects them. But, inside the factory you still want to minimize the amount of defects for QA to find. Either way you still need QA, but it's easy to get into the habit of improving QA vs improving the initial process.
PS: One exercise I did for a while was every time I found a non syntax bug in a function I just wrote (aka bug at run-time) I would start over and rewrite it. It's painful, but 'Build' wait let's look at this again if I hit run and it does not work that's going to be painful. So is this actually correct? Anyway, doing that showed me how important it was to just focus on the code (and what it means) to exclusion of everything else.
But you will not drive significantly higher quality into earlier phases by telling your factory workers to try to do a better job. You drive higher quality by analyzing where your processes are failing you and changing them so that they no longer do.
If your workers are consistently failing to tighten bolts sufficiently, you can add a QA step to check the bolts every time, or require your workers use a torque wrench, or use a robot to make this more consistent. But you have to do something other than telling them to try harder. You undoubtedly told them to try harder already and it didn’t work. Now you need better tools or processes.
However, the lady designing the factory in the first place should know if tightening bolts or using an arc wielder has more issues. So sure, QA should include checking the oil level inside a robot in the factory to avoid problems in the first place. But QA is not all inclusive it's not for example part of the collage education that's training your workforce other than perhaps a requirement that they have said degree.
QA is the system of trying to produce at the desired quality. Anything done to improve quality is QA, including whether you hire better employees or train the ones you have. You can define QA more narrowly if you like, but then we're arguing semantics and we're already off in the weeds.
Heh...irony alert. You're advocating focusing on just getting it right the first time and you didn't get write right.
But seriously, the people you're responding to are correct. Any process that relies on humans being less error proneis bound to fail. You need to either create a process that makes humans less error prone (e.g. checklists) or embrace our propensity for making errors and plan for that eventuality.
This metric means very little. It does not measure the extent of code path coverage and much more.
Code coverage can tell you what code is not tested at all.
Now, this is very useful. But:
Code coverage can't tell you what code _is tested_.
Code coverage can't tell you how _well_ code is tested.
Not sure about other types of development, but in web development, TDD can be a good way to have automated tests without the additional cost.
Sorry for any confusion for anyone reading my previous comments.
And indeed, tests don't take much time to write once you get used to writing them. It's like anything. The more you do it, the better you get at it.
We have a full continuous integration environment at work, and the tests run there fine, but trying to reproduce that on a local machine is often a fairly difficult experience.
We have maybe 30 components in our system, so often I haven't touched the component before and I am asked to fix a bug in it. Sometimes they are using standard testing libraries, other times there are a lot of extra libraries that I haven't used before. Getting everything to play nicely isn't always trivial.
This drastically scopes down the surface area for CI breaks.
[1] There are a few exceptions for tests that specifically cover environmental config/behavior that cannot be fully tested locally.
> invested heavily
pick one.
But also, testing in browser is more expensive long term that writing proper automated tests. Up front it’s cheaper, like any other technical debt.
If you’re very familiar with testing and/or doing TDD, you might include your testing costs in your estimate, but you still have the cost to pay. And if you write good testing up front, it will cost more up front.
Cake + Eat-it, too.
In the browser, testing edge cases sometimes requires custom headers, encoding of data to create authorization tokens, etc. This means you have to either have amazing browser tools (which I've yet to find) or use a combination of browser and shell to achieve what I need.
In TDD, most setup can be automated with simple function calls. In addition, well made frameworks, such as PHP's Symfony, have utilities that even avoid making real HTTP requests, so tests run faster than using a browser, but the result is the same.
I'm not saying everyone should do TDD, but from my experience, it can lead to increased productivity and fewer bugs in some cases.
By the way, if you know of browser tools that make testing easier, I'd be glad to learn more. I use TDD because I lack the browser tools. If I had the right tools, maybe I'd consider going back to in browser testing.
When my function under test calls sort() I don't fake that call out, so technically I just wrote an integration test. (In fact I work with one group that will inject a fake sort())
If I write a library foo which has sub-module bar which has class baz and fuzz, is the test for bar that tests both baz and fuzz a unit test of an integration test? Of course if you are a user of my library tests for foo are unit tests to you...
There's no agreement what is "unit". Few classes interacting with each other could still be a unit.
I didn't find any strict definition which would be useful in practice. I just write tests and guess they are mostly integration tests, some end-to-end tests and a few unit tests.
IME unit tests work acceptably in one very specific scenario and fail pretty badly in all others. That scenario being:
1) You're surrounding a self contained block of code that interacts "with the outside world" via a code API.
2) That code API is a very stable and clean abstraction.
3) It has minimal interactions with modules outside of it and those interactions that it does have are tightly scoped (i.e. minimal to zero mock objects are required to write the test).
4) The logic of the code is relatively complex and most bugs that crop up are logical in nature (e.g. off by one, things getting swapped around, incorrect calculations, wrong behavior with negative numbers).
Meanwhile, integration tests (at varying levels) work well for pretty much every case apart from this and still work okay for this type of code. They make much more sense as a go-to default.
I've also worked on several projects where there was little to no code that it actually made sense to unit test. It's not uncommon that an entire codebase is predicated mainly on hooking systems together and doing some shallow calculations. IMHO, having zero unit tests in that environment is actually desirable.
The worst unit tests I've seen have been written when two or more of those preconditions have failed. They would fail constantly, require massive maintenance and, somewhat comically, almost never fail in the presence of an actual bug.
But it doesn't follow that changing a unit's interface means unit tests suddenly become just a burden. Ideally unit tests are, well, testing a bunch of core functionality of the unit under test. You adapt them to the new interface. Then you're back to having a quick, automatic sanity check you can run against the unit whenever you have to make a change.
I don't understand people bemoaning this 'cost' of unit tests when the benefits they provide typically far outweigh the costs. It's possible broader functional/integration tests have a better ROI in certain situations, but they come with a maintenance cost as well.
If your refactoring is largely centered around changing unit interfaces (not uncommon) then it means that those unit tests are 100% overhead because most of the time they fail just because you changed the code.
>I don't understand people bemoaning this 'cost' of unit tests when the benefits they provide typically far outweigh the costs. It's possible broader functional/integration tests have a better ROI in certain situations, but they come with a maintenance cost as well.
I've certainly found that the ROI is a lot better. I find that integration tests have a higher up-front cost but maintenance-wise they're the same or cheaper. % of failures that are actually catching bugs is higher too.
Another condition I would add to the list, which the component of mine I am thinking about meets is:
5) Failures in this component have cascading effects to multiple other components, causing seemingly "impossible" failures that are not obvious to others that they were caused by this component.
That needn't be a unit test. It could simply be a lower level integration test.
Maybe that should be a number 6): easily tested invariants.
That's completely at odds with my experience.
I find that for local bugs, the cost of locating them grows with O(log n) of the amount of code. And for non-local bugs (interface bugs, system bugs, incompatible specs...) unit tests don't catch them anyway.
I don't think this is true. Fixing a bug is comprised of four parts. 1. Understanding and reproducing the bug.
2. Finding the code responsible for the bug.
3. Coming up with code that fixes the bug.
4. Verifying that your fixing code does not introduce any new bugs.
#2 is the only one that could even theoretically be exponential. The only bugs I've found where I've spent days on 2, were hard to reproduce, intermittent, race condition bugs, which units tests aren't very good at finding anyway.
A unit test, or a component test, would tell you that something failed, and it's in this general area, which narrows down the search quite a bit.
They're both useful, but I've seen far more problems with people arguing that end-to-end test is more than what they need, while a bad conversion from seconds to nanoseconds would be caught quickly if a unit test was actually written.
I'm a big fan of "economic testing". Test that are not going to "be economically viable" to write during the dev't of new software, usually do not get my permission to be written in that phase.
Unit test thus get written for part that we absolutely want to be maximum certain of that it works correctly. Mostly that functionality can be extracted into a library. For instance a lib that deal with scheduling of events of arbitrary length: we do not want to have issues with those getting messed up, so we try to get 100% unit test coverage of the functionality in that lib.
Another thing with unit test is that I found they mean something different in dynamic languages than in strongly typed languages. In the first unit test are often something to provide confidence that the type system can give you in the latter. Sure both will also test the logic. But this understanding has pushed me to love strong types (as a rubyist), as I can go with less unit test and still be more confident about my code base when it is being heavily worked on. Refactoring becomes so much easier with strong types, as all is "in the code".
When it comes to integration tests I usually want to make a biz case for them. How much does it cost to have it all manually tested, and how much do the automatic integration tests cost to make. While including the benefit of being able to run the automated integration tests at near zero cost (thus several times per day if we need to).
I hear this kind of rhetoric a lot, but rarely do I hear of code that doesn't need to work correctly.
A graphical error that prevents users from being able to use the application at all can be basically the same as server downtime in terms of impact to the end user.
On the other hand if the code responsible for the styling of some menu is not working correct, the functionality may still work, and once we discover the bug we can simply fix it.
> rarely do I hear of code that doesn't need to work correctly.
So there are different grades. That's why I used the word maximum. The cost for these unit test should pay bank, even during the dev't phase, IHMO. There are often areas in a large code base for which this hold true, and often they can be identified as such beforehand; then writing these tests should be a priority.
The difference is having code for which having some bugs is acceptable, vs having code for which bugs are not acceptable.
There is definitely code for which having some bugs is acceptable: think about some small feature that the user can totally do without. I had to do one recently, and bugs were acceptable, as long as it was shipped in time, and working for the client specific devices. That code shipped with no automated test (testing is done manually), and a few known bugs not worth solving. On the other hand, anything that touch core features must be tested.
An issue with that is bus factor when it needs to be changed in the future. What if the original dev isn't around any more?
Also, the code has been reviewed and is documented.
Bus factor really isn't a problem in the instance I was talking about. In fact, the original dev won't be around in a few months, and that's fine. The rest of the team will be able to modify the code.
I think this is a partial answer.
Yes, I write unit tests for such pieces of software. However, what I also write a lot of unit tests for a pieces of software which are hard to write good integration test for because of combinatorial explosion. If I have a part for integration method which consists of a chain of 4 methods, each of which can have 5 different results, that'd mean I'd have to write 4^5 integration tests, which is clearly not possible. If I were to write unit + integration tests, however, I could do with 4x5 unit tests plus a couple of integration test which prove that the overall combination of those methods work well together.
In practice I've also run into a lot of combinatorial explosions for which unit testing is worse than useless. E.g. say you have a library that has two dependencies and interacts with browsers you can have:
* 30 different versions of library A
* 50 different versions of library B
* 25 different versions of the language runtime
* 30 different browsers
That's over a million different potential variations for every test. In practice though, there's a sharp power law. You don't have to test all of those variations, you just have to randomly throw variations at the code until bugs stop cropping up. 99% of the bugs you find probably be found in the first 1,000 iterations.
My feeling about unit testing / integration test is you do as much of it as you can, trying to sincerely learn the lessons of how to do it it effectively, until you know why you don't need to do it. Or until you know what the right mix is. I started with Extreme Programming and testing around 18 years ago and my thinking has constantly evolved over that time. One of the biggest factors, I think, is project context and the nature of the software you are developing.
I find in embedded systems, unit tests done extensively are very very useful. As is integration testing.
In web systems, I find integration tests are often most useful as so much is all about gluing together various tech, and it's often the gluing the comes unstuck. However there is often chunks that have some interesting logic that lends itself to more extensive unit tests.
Sometimes testing is just a mess because of the design. Too many things that are not really needed, too many layers, too many levels of abstractions, too much "future proofing". Then you see people start trying to throw unit testing around it, and it is mostly just useless and explodes out the amount of work to do. Often I think articles like the original are reactions to this scenario.
Or how about another project that had lots of real tests, that were old, unmaintained, did humongous amounts of work in each single unit test and took a weekend to run, and of course, fail.
Or perhaps the team that obsessed about the testing for each user story, spent all the time doing that, then running out of time to deliver the actual stories.
And those were the less egregious examples!
Integration tests, in contrast, require attention more frequently -- because not all subcomponents are stable and when one changes, multiple integration tests need superficial readjustment.
I'm skeptical that my own ROI analysis is more "right" -- it probably varies by project and team. The conclusion I draw instead is that ROI analysis is important in general when writing tests, but that one particular ROI analysis is unlikely to apply universally.
The article makes a lot of assumptions like "In practice, most agree as most projects set the lower bound for coverage to around 80%". Most projects where, which industry, who are these 80% that agree? This is just taken out of thin air in this case. It might be true but should we take the author's word for it?
This is very subjective without examples. The opposite argument can also be made that code which is easier to test is sometimes better. It felt like reading a collection of quotes and articles by other people.I think TDD, unit, integration and E2E testing works. How much of which to apply is entirely project and industry specific and it's up to teams to decide what testing strategy works best for them.
https://medium.com/@TuckerConnelly/94-gems-from-code-complet...
personally I like write some cursory unit test because it ensures the code logic is correctly decoupled from data retrieval, but most of my unit tests are written from user reports, as part of a large no regression suite.
While I use all sorts of testing, I still write a good amount of unit tests. For whatever kind of tests I write: not a day goes by without tests or the act of writing them exposing bugs that would otherwise have moved on to the "next level". Doesn't mean they'd have ended up in production, but they'd still have been a lot more expensive to fix.
Being in between does not mean it gives the highest ROI or the best balance. They can easily share the worst of both.
> Lean Testing takes an economic point of view to reconsider the Return on Investment of unit tests.
But does not in any way actually try and work this out, which means that you can't make the conclusions that are in this.
> The Return on investment (ROI) of an end-to-end test is higher than that of a unit test. This is because an end-to-end test covers a much greater area of the code base. Even taking into account higher costs, it provides disproportionally more confidence.
And yet no figures or evidence.
Many e2e tests can cover the same path segments, meaning adding a new test may not increase your confidence by much at all. But they're still taking longer to run, and despite the articles insistence that integration and unit tests are brittle, I've found E2E tests are also brittle, just in different ways.
I shouldn't have to change my string normalisation tests just because the website changed moving the resulting output field to a new place on the page. They shouldn't break because the login flow changes.
I have definitely seen (and written) E2E tests that provide almost no real value, and unit tests that provide huge amounts.
> Plus, end-to-end tests test the critical paths that your users actually take. Whereas unit tests may test corner cases that are never or very seldomly encountered in practice
Unfortunately low proportions of errors still can account for extremely large actual numbers of problems. 1/1000 bugs will happen all the time if you've got a reasonable number of users. Those bugs also may happen to most of your users if they're somewhat random, or worse may heavily impact one group.
> For many products, it is acceptable to have the common cases work but not the exotic ones (‘Unit Test Fetish’). If you miss a corner case bug due to low code coverage that affects 0.1% of your users you might survive. If your time to market increases due to high code coverage demands you might not survive.
Or if roughly one in 1000 customers keep taking your production server down due to a bug you might not, and you may have been just fine waiting another week to go live.
I agree with the idea, you should consider what it is you're trying to achieve and what the best ways of doing that are. What you absolutely shouldn't do is use terms like ROI and economic without any analysis, purely to justify not doing something you don't really want to do.
"I suggest that the programmer should continue to understand what he is doing, that his growing product remains firmly within his intellectual grip." [1]
[1] E.W. Dijkstra: https://www.cs.utexas.edu/users/EWD/transcriptions/EWD03xx/E...
So, your argument is true for individuals, but for the community it is void. Together we should focus on identifying correct, effective, simple and reliable building blocks on which we can build our larger systems.
Perhaps you haven't seen "Simple made easy" by Rich Hickey, which illustrates this concept rather well.
And in my experience most of the defects which aren't caught by our quality control process wouldn't have been prevented by switching from OOP to FP, or from mutable to immutable state, or anything like that. So I'm not convinced.
In the initial run on this code base we went a bit unit test crazy, which was fine at the time, but I'm seeing the drawbacks now where I'm doing a lot of surgery on the components.
I've ended up just beefing up the current integration tests with some extra edge case testing and, after the refactoring work, they still pass, which has given me a lot more confidence.
I think a lot of this comes to head when you unit test the glue logic that orchestrates the entire program, unit testing those becomes painful with mocks etc because they become fragile whenever you do any refactoring work.
I'm not sure what to think really.
Unit tests also have the advantage of being cheap to set up and cheap to run, so their limitations in whole-system testing are offset by the fact that you can have many, many more of them. You can easily blow through a couple thousand unit tests in a matter of 10 minutes or less, whereas even a modest battery of integration testing will be an overnight process at best. That means your OODA loop is a lot faster on unit testing.
(the ideal is of course structuring your application so you can deploy many tests in parallel, but that's not always possible.)
Fixing Selenium so WebDriver instances don't take 60+ seconds to launch would be a major help. Until then, integration tests that work at a controller level (minus the UI) are probably the best compromise between speed and integration. You can pass in a mocked request as input, and assert on the output. Runs much faster than a "true" integration test with a driven browser client, but hits the whole stack better than a unit test. They are also really easy to set up, since you can just have a tester or an integration test go through the motions and then dump request data to JSON, then feed it back as a test. The lack of state persisted between requests makes implementation a lot simpler.
They are not cheap to develop and maintain however. And given enough developers a certain percentage of them will completely distort the design of the code to enable unit testing making the code itself much more complex.
If it's because you need to change them after a feature changes or breaks, isn't that the entire purpose of testing and not exclusive to unit tests?
Others see units as the smallest bits of irreducible business or tech knowledge. So if you have 20 methods in a class, you test the 5 public ones, or the methods that compose a useful "unit". This is the definition used by the original test driven development book.
Many of the unit test advocates I've read use the second definition because testing private functionality incurs all the problems the article points out.
Using the second definition leads to the testing pyramid, which is a thing of beauty. Unit tests have a few mocks to the unit's partners. The unit tests test for the correctness of the unit. The integration tests are one level up in terms of code coverage in a single test, and they ensure that the mocks used in the unit tests are correct. The e2e tests are then used to test that the app starts correctly and is configured correctly. The obvious happy and sad path scenarios are also tested to ensure that dependencies are functioning correctly (in a complex application there can be hundred of "obvious" e2e tests).
In my career, every single instance I've seen about someone complaining about writing unit tests, they're either using the first definition I mentioned above, or their code is not composed correctly. I'm sure exceptions exist somewhere, but I've yet to see them.
Using integration tests like the article describes leads to a host of problems. For starters, it's extremely difficult to get complete code coverage without huge or redundant test suites. These large test suites are harder to read and take longer to write because more complex testing requires more complex mocking. Regression is also more likely during large refactors since you haven't clearly enumerated expected behavior.
Unit tests should only change if you also change public apis (I have one exception to this rule, which is very complex private methods which could easily be their own unit).
Then if you're changing the tests, you're also refactoring everything else.
>For example if you use mocks to invocation counts, refactorings can be even more costly.
These should only be costly if you're over-testing. You shouldn't be mocking and asserting calls on every call within a system under test, you should be checking the ones that matter.
Correctly written unit tests are a refactoring aid.
If a test fails and it breaks CI, there should be a problem in the product. Move everything else to some "additional testing" bucket and not block developers.
That means you're either testing the return value or testing the inputs which were modified by the unit.
If the behavior of a unit changes in such a way that it does something else with its inputs, it's a specification change which requires other test cases.
It is OK to be economical in what one would see as a cost of testing a product end-to-end. Call it whatever term you like, but functionally testing a product end-to-end has always followed an economical approach, as all such testing is a factor of time & resources. You'd perhaps be less economical if you were testing life-safety systems, but you could choose to be more economical if you considered your software to be not-that-critical, or, you needn't have to set exceptionally high standards.
I find it extremely odd that unittests are even considered into this equation of 'testing-costs', despite decades of improvements in software development processes. Unittests should be part of 'core' engineering & a part of development. Not a task that's added to testing costs. If you aren't writing unittests (irrespective of whether its TDD or not), you are not developing/engineering your product right. You are only building a stack of cards _hoping_ that it wouldn't collapse at some point in the future.
It's a sad state of affairs if one gives code 1st level treatment, but treats unittests, documentation, build scripts and other support infra as something less important; worth economizing. This is really what _matters_ when it comes to overall quality of a software.
One must remember that software is always improved, refactored, expanded, ported or worked on in some way or another. When unittests are missing, then the very boundaries that were meant to dictate the rules of the software don't apply anymore. This leads to human errors causing portions of software to break.
Critical pieces of software that exist today, exist strongly because they were engineered right (Linux kernel for example). Not because their developers followed an _economical_ approach to testing. If an OS disto claimed to perform economical end-to-end testing (of a potential user's most commonly performed paths), would the author of this article want to use that OS over one that has had strong ground-up unittests and testing of interfaces where they matter?
Just a quick google and not even the paper I was thinking of: https://softwareengineering.stackexchange.com/questions/6050...
The rule of thumb I always come back to is "the minute you decide it's time to attach a debugger, be prepared to burn an hour." I used to not mind firing up the debugger -- heck, I used to run my code with a debugger attached every time I ran it. These days, I write tests. I don't aim for 100% code coverage, though I aim for a majority and 100% coverage of critical paths. When I encounter a problem that isn't covered by a test (rare), I write a test centered around the problem. This usually surfaces the bug before I've even ran the actual test -- and once that test is there, any refactoring that affects that code will avoid that bug if the test passes.
This rule has been so effective for me that I've now gone through four different teams, encouraging each to do "the next project" with automated testing. The result has been a commitment to working that way from that point forward[1].
[0] Assuming some baseline quality requirements -- automated build, and a general requirement to publish a mostly functional product. :)
[1] I can't take full credit for the idea -- it was a blog post that convinced me to try it on myself and I advocated that idea after it convinced me.
Try building a tennis game, just in code with no gui, and take note of how much you would either write tests for things like the score, set points etc., or how much time you spend debugging to justify if your code works.
Unit tests are a great documentation tool as well, both for developers but also for the business needs of the software — the value.
But if symptoms would likely be too subtle to stand out, e.g. a typical bug would manifest itself in results that seem plausible despite being wrong, then it's a strong indication for unit tests.
I agree that every software development team has to make their own decision to balance development time and maintenance cost but I don't agree with this low entitlement mentality of writing correct code in this article.
Of course there are also costs when implementing tests and your unit tests are rendered useless on design changes. But so what? You're also decreasing code assurance and it's better to increase it with tests so that you won't encounter bugs in production which is much more expensive than writing tests.
Software development is not only about adding functionality but also about ensuring its correctness.
And we made the choice not to have unit tests. Everything was done through integration tests. What made up for it is that we had very good traceability. So for we knew which lines of code were related to which requirement and which test. So if we changed some code, we knew which test to run, and if a test fails, we knew where to look at.
For the edge cases, we had the ability to inject data directly into memory, triggering the otherwise unreachable error conditions, and the very last cases where done through code analysis (we needed 100% coverage).
https://rbcs-us.com/documents/Why-Most-Unit-Testing-is-Waste...
My advice is usually to do tdd for new dev have good coverage of full function tests and delete unit tests if the get in the way/lose value on refactoring. No one likes deleting tests but they must add value or its just more debt.
This is probably not the same story for something like a react or swift application. Good read, thanks for the solid points.
I'm a huge advocate of automated testing and with the available tools, like docker, it's relatively painless to get the pieces together to sort out automated testing. Often the tooling you use to enable automated testing is tooling you end up needing anyway -- it's dual purpose. Before the "first run" of a set of code, I'll create Dockerfiles that make up a complete, local, development instance of an application along with some boilerplate tooling that I include to make debugging easier on me. When setting up the production build, the final version is usually this same environment with fewer lines in the file. Because my environments tend to be similar, I have a zsh script that strips out lines in the Dockerfile to get me 90% of the way to a production container.
For me, it's always worth it. I came to this conclusion after spending a few months forcing myself to test rigorously[0], starting with unit tests written often and early and ending with a small number of integration tests and a much smaller number of end-to-end tests. I don't find any of these particularly difficult to write.
The benefits, however, are vast: (1) Avoiding the debugger time-sink: The #1 thing that I always come back to is that I generally end up never having to fire up a debugger. I noticed that every time I encountered a bug in code that was poorly covered, the first instinct was to attach a debugger and peek at locals to see what was going on. This rarely resulted in spending less than an hour troubleshooting. Sometimes you get lucky and you find more than one issue in that debugging session, but often it landed in at burning an hour on ever bug and way too often it was an hour spent debugging production code and the bug was customer impacting[1]. At the same time, it's rare for an automated test to have a time cost that high. (2) Refactoring - Since "premature optimization is the root of all evil", that necessarily means that a performance bug is going to involve injecting complexity into a running codebase and this often comes with high-impact refactoring. Unit tests, specifically, are incredibly helpful here. This is often an argument against integration/end-to-end test automation since refactoring regularly breaks these brittle tests, however, I've found in practice that this isn't the case at least half the time. Of the times that it does affect those tests, the practice of refactoring can surface subtle bugs (on a few occasions it surfaced a subtle race condition that might have been missed if a few of the integration tests covering a subset of the functionality hadn't broken). (3) Design - more for integration and unit tests, thinking about testing while writing code can result in a less brittle design[2]. On integration tests, it means writing SQL scripts and migrations to ensure that a fresh environment can be spun up on-demand instead of using GUI tooling (or, using the GUI tooling to generate said scripts/migrations). (4) Build automation - I'm somewhat surprised at how often I encounter a customer project where I have to follow a 20 step process to get things functional in a development environment. It seems like if CI isn't involved, people figure a README.md with a mess of shell commands and button clicking is OK. Scripting out environment configuration and build was already one of the first things I did when I began running the code I'd written, however, I find I no longer have to argue in favor of this when testing is involved -- everyone wants a single command to execute tests and once integration and end-to-end tests are involved, it just makes sense to add standing up the docker parts, too[3].
I get why there's resistance to doin...